public inbox for nncp-devel@lists.cypherpunks.su
Atom feed

* nncp-bundle, nncp-caller, and .seen not working?
@ 2020-12-30  6:37 John Goerzen
  2020-12-30 14:08 ` Sergey Matveev
  0 siblings, 1 reply; 13+ messages in thread
From: John Goerzen @ 2020-12-30  6:37 UTC (permalink / raw)
  To: nncp-devel

Hi,

Today I had a slow wifi link between two computes passing a few GB 
of data, so I experimented with nncp-bundle.

I didn't delete from the source machine.  I transferred to USB, 
then used nncp-bundle -rx on the destination.

It appeared to work.  I used nncp-toss with -seen and it did its 
thing.

Strangely, however, nncp-caller would re-download the same 
packets.

I noticed that nncp-bundle put incoming data in a directory named 
for the local self's id, while nncp-caller put incoming data in a 
directory named for the neighbor's id.  Perhaps that's the issue? 
I'd wind up with two .seen files, one in each place, after all 
this was over.

I haven't tested but wonder of nncp-xfer would have the same 
issue.

Thanks!

- John

^ permalink raw reply	[flat|nested] 13+ messages in thread

• * Re: nncp-bundle, nncp-caller, and .seen not working?
    2020-12-30  6:37 nncp-bundle, nncp-caller, and .seen not working? John Goerzen
  @ 2020-12-30 14:08 ` Sergey Matveev
    2020-12-30 15:37   ` John Goerzen
    0 siblings, 1 reply; 13+ messages in thread
  From: Sergey Matveev @ 2020-12-30 14:08 UTC (permalink / raw)
    To: nncp-devel
  
  [-- Attachment #1: Type: text/plain, Size: 1291 bytes --]
  
  Greetings!
  
  *** John Goerzen [2020-12-30 00:37]:
  >I noticed that nncp-bundle put incoming data in a directory named for the
  >local self's id, while nncp-caller put incoming data in a directory named for
  >the neighbor's id.  Perhaps that's the issue?
  
  Of course that was a bug. I do not know (or remember?) why it is not
  placed in the sender's rx/ directory, but in the self's one. nncp-toss
  worked anyway, because sender/recipient information is taken from
  packet's header. I have fixed that in:
  http://www.git.cypherpunks.ru/?p=nncp.git;a=commit;h=87245f236b40415c6e30fbaee18cdaf46b7f5c57
  You can try it just by overwriting nncp-bundle/main.go file and
  rebuilding it.
  
  By the way, I plan to add ability to create uncompressed nncp-exec
  packets. Currently nncp-exec always compresses the data inside, because
  originally it was used only for email transmission. I will add another
  plain packet type and an additional command line option for nncp-exec.
  Everything will stay the same and backwards compatibility with existing
  nncp-exec-generated packets won't break. After that I will create new
  release with current nncp-bundle fix and documentation updates.
  
  -- 
  Sergey Matveev (http://www.stargrave.org/)
  OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF
  
  [-- Attachment #2: signature.asc --]
  [-- Type: application/pgp-signature, Size: 833 bytes --]
  
  ^ permalink raw reply	[flat|nested] 13+ messages in thread

• • * Re: nncp-bundle, nncp-caller, and .seen not working?
      2020-12-30 14:08 ` Sergey Matveev
    @ 2020-12-30 15:37   ` John Goerzen
      2020-12-30 15:44     ` John Goerzen
      2020-12-30 19:10     ` Sergey Matveev
      0 siblings, 2 replies; 13+ messages in thread
    From: John Goerzen @ 2020-12-30 15:37 UTC (permalink / raw)
      To: Sergey Matveev; +Cc: nncp-devel
    
    
    On Wed, Dec 30 2020, Sergey Matveev wrote:
    
    > packet's header. I have fixed that in:
    
    Thanks!
    
    > By the way, I plan to add ability to create uncompressed 
    > nncp-exec
    > packets. Currently nncp-exec always compresses the data inside, 
    > because
    > originally it was used only for email transmission. I will add 
    > another
    > plain packet type and an additional command line option for 
    > nncp-exec.
    > Everything will stay the same and backwards compatibility with 
    > existing
    > nncp-exec-generated packets won't break. After that I will 
    > create new
    > release with current nncp-bundle fix and documentation updates.
    
    Interesting!  I just happened across an apparent memory leak in 
    nncp-exec in which it was using over 1GB of RAM.  To be sure, I 
    may have been piping several GB of data to it.  But I would expect 
    it to stream out the data as it is read from stdin, hopefully?  I 
    wonder if this is related?
    
    Thanks again for NNCP.  I am really appreciating it!
    
    - John
    
    ^ permalink raw reply	[flat|nested] 13+ messages in thread

  • • * Re: nncp-bundle, nncp-caller, and .seen not working?
        2020-12-30 15:37   ` John Goerzen
      @ 2020-12-30 15:44     ` John Goerzen
        2020-12-30 19:10     ` Sergey Matveev
        1 sibling, 0 replies; 13+ messages in thread
      From: John Goerzen @ 2020-12-30 15:44 UTC (permalink / raw)
        To: Sergey Matveev; +Cc: nncp-devel
      
      On Wed, Dec 30 2020, John Goerzen wrote:
      
      > Interesting!  I just happened across an apparent memory leak in 
      > nncp-exec in
      > which it was using over 1GB of RAM.  To be sure, I may have been 
      > piping several
      > GB of data to it.  But I would expect it to stream out the data 
      > as it is read
      > from stdin, hopefully?  I wonder if this is related?
      
      Also related:
      
      On the receiving side, I'm getting "data exceeds max slice limit" 
      for these large ones.
      
      Is there a file size limit to exec and (un-chunked) file?
      
      - John
      
      ^ permalink raw reply	[flat|nested] 13+ messages in thread

    • * Re: nncp-bundle, nncp-caller, and .seen not working?
        2020-12-30 15:37   ` John Goerzen
        2020-12-30 15:44     ` John Goerzen
      @ 2020-12-30 19:10     ` Sergey Matveev
        2020-12-30 19:16       ` Sergey Matveev
                               ` (2 more replies)
        1 sibling, 3 replies; 13+ messages in thread
      From: Sergey Matveev @ 2020-12-30 19:10 UTC (permalink / raw)
        To: nncp-devel
      
      [-- Attachment #1: Type: text/plain, Size: 3677 bytes --]
      
      *** John Goerzen [2020-12-30 09:37]:
      >Interesting!  I just happened across an apparent memory leak in nncp-exec in
      >which it was using over 1GB of RAM.  To be sure, I may have been piping
      >several GB of data to it.  But I would expect it to stream out the data as it
      >is read from stdin, hopefully?  I wonder if this is related?
      
      My intention for uncompressed execs is not related. I just see that
      someone can (obviously) use nncp-exec for tasks other than sending
      (highly compressible) email/news and compression can harm and be useless
      in many cases.
      
      nncp-exec really stores all compressed data in the memory before writing
      to the disk. I can replace it with writing to the temporary file, as
      "nncp-file - dst:..." already does now.
      
      Problem with streaming is that you do not know exact file size in
      advance, that is written in the beginning of the first encrypted block.
      So currently, for nncp-file reading from stdin, I store all read data in
      the temporary file and then read it, knowing its size. Of course
      ephemeral symmetric encryption is done, so even if computer will
      suddenly shutdown and that file stays there -- noone can decrypt it.
      
      I am not sure, (it is too late to think clear :-)), but I think that
      temporary file can be replaced with the following scheme:
      
      * we create an ordinary encrypted packet in place, streaming all the
        data inside but. But leaving its first encrypted 128KiB block blank
        (filled with zeros)! Its payload is memorized in RAM
      * after all the data is read and written, we fill size field in that
        block in the RAM and encrypt it
      * then seek to the zero-filled first block and overwrite with its
        encrypted fully ready contents from the RAM
      
      All encrypted blocks are independent from each other
      (http://www.nncpgo.org/Encrypted.html), so we can safely do that. Zero
      filled block is also will skip real block allocation on ZFS with any
      kind of compression enabled. But that single 128KiB block won't be
      sequentially placed to other ones. Not a big deal I think.
      
      But here is another problem: the whole packet is hashed from beginning
      to the end. And changing that single block will require the entire
      hashing calculation to be done again. It is good that we get rid of
      temporary file at all, but we still need to sequentially write all the
      data from stdin, make seek/overwrite and then sequentially read all the
      data again for its checksumming.
      
      We can avoid it by using hash trees, Merkle trees. Changing of that
      single block will lead to recalculation of only one tree path, but not
      reading the data from the disk again. Moreover it gives ability to
      parallelize hash calculations (however I doubt it is bottleneck anywhere
      in practice, but who knows!).
      
      So... currently I do not see what can go wrong with that scheme. No
      temporary files (for "nncp-file -" and nncp-exec) and ability of
      parallelizable hashing. And of course it will "fix" (actually it is not
      a bug, just that is how it worked all the time :-)) the issue with
      memory consumption with nncp-exec.
      
      But it will make current packets backward incompatible with that scheme,
      because of different checksum algorithm.
      
      >On the receiving side, I'm getting "data exceeds max slice limit" for these
      >large ones.
      >Is there a file size limit to exec and (un-chunked) file?
      
      Should not be any. Currently do not know where it can raise that error.
      Will look at it later, probably when the "new scheme" will be implemented.
      
      >Thanks again for NNCP.  I am really appreciating it!
      
      Glad to hear that!
      
      -- 
      Sergey Matveev (http://www.stargrave.org/)
      OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF
      
      [-- Attachment #2: signature.asc --]
      [-- Type: application/pgp-signature, Size: 833 bytes --]
      
      ^ permalink raw reply	[flat|nested] 13+ messages in thread

    • • * Re: nncp-bundle, nncp-caller, and .seen not working?
          2020-12-30 19:10     ` Sergey Matveev
        @ 2020-12-30 19:16       ` Sergey Matveev
          2021-01-01  5:36       ` John Goerzen
          2021-01-01  5:39       ` John Goerzen
          2 siblings, 0 replies; 13+ messages in thread
        From: Sergey Matveev @ 2020-12-30 19:16 UTC (permalink / raw)
          To: nncp-devel
        
        [-- Attachment #1: Type: text/plain, Size: 500 bytes --]
        
        *** Sergey Matveev [2020-12-30 22:10]:
        >Problem with streaming is that you do not know exact file size in
        >advance, that is written in the beginning of the first encrypted block.
        
        Actually it is not stored in the first block, but just preceedes it. So
        "size" is just a 64-bit ciphertext + 128-bit MAC tag. So we do not have
        to store that 128KiB of data, just only the 64-bit size.
        
        -- 
        Sergey Matveev (http://www.stargrave.org/)
        OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF
        
        [-- Attachment #2: signature.asc --]
        [-- Type: application/pgp-signature, Size: 833 bytes --]
        
        ^ permalink raw reply	[flat|nested] 13+ messages in thread

      • * Re: nncp-bundle, nncp-caller, and .seen not working?
          2020-12-30 19:10     ` Sergey Matveev
          2020-12-30 19:16       ` Sergey Matveev
        @ 2021-01-01  5:36       ` John Goerzen
          2021-01-02 12:56         ` Sergey Matveev
          2021-01-01  5:39       ` John Goerzen
          2 siblings, 1 reply; 13+ messages in thread
        From: John Goerzen @ 2021-01-01  5:36 UTC (permalink / raw)
          To: Sergey Matveev; +Cc: nncp-devel
        
        Happy New Year!
        
        On Wed, Dec 30 2020, Sergey Matveev wrote:
        > Should not be any. Currently do not know where it can raise that 
        > error.
        > Will look at it later, probably when the "new scheme" will be 
        > implemented.
        >
        
        I'm getting things like this periodically:
        
        Dec 31 12:41:00 hostname nncp-caller[14357]: E 
        2020-12-31T18:41:00.090076849Z
        [sp-recv err="xdr:decodeArray: data exceeds max slice limit - 
        read: '415373007'"
        nice="255" node="L...Q"]
        
        Dec 31 15:48:00 hostname nncp-caller[14357]: E 
        2020-12-31T21:48:00.046258445Z
        [sp-recv err="xdr:decodeArray: data exceeds max slice limit - 
        read:
        '3814302942'" nice="255"
        node="L...Q"]
        
        It doesn't seem to result in any harm in the end, though I will 
        note that multi-GB files seem to cause either the caller or the 
        daemon to hang up.  It's not entirely clear which, but it seems 
        that the caller takes awhile to finish the write (a fsync?  or 
        hash verification?) and the daemon times out, perhaps killing 
        simultaneous transfers (it appears that's a thing?)
        
        The nice value is odd too; I've never explicitly set anything to 
        nice 255.  Maybe some form of corruption?
        
        Thanks again,
        
        John
        
        ^ permalink raw reply	[flat|nested] 13+ messages in thread

      • • * Re: nncp-bundle, nncp-caller, and .seen not working?
            2021-01-01  5:36       ` John Goerzen
          @ 2021-01-02 12:56         ` Sergey Matveev
            0 siblings, 0 replies; 13+ messages in thread
          From: Sergey Matveev @ 2021-01-02 12:56 UTC (permalink / raw)
            To: nncp-devel
          
          [-- Attachment #1: Type: text/plain, Size: 734 bytes --]
          
          Happy New Year everyone!
          
          *** John Goerzen [2020-12-31 23:36]:
          >I'm getting things like this periodically:
          
          Aah! Yes, I have also seen similar errors in my logs for a long time.
          Well, actually I do not remember why that issue is still not fixed.
          Either I could not find the reason, or I have never tried it, because,
          as you mentioned, actually no data is either lost or corrupted, so why
          bothering :-). When I will start working with NNCP this year, I will
          look at that issues closer again. My daemon/caller just restarts their
          connection and everything continues to flow smoothly again. So that just
          annoys.
          
          -- 
          Sergey Matveev (http://www.stargrave.org/)
          OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF
          
          [-- Attachment #2: signature.asc --]
          [-- Type: application/pgp-signature, Size: 833 bytes --]
          
          ^ permalink raw reply	[flat|nested] 13+ messages in thread

      • * Re: nncp-bundle, nncp-caller, and .seen not working?
          2020-12-30 19:10     ` Sergey Matveev
          2020-12-30 19:16       ` Sergey Matveev
          2021-01-01  5:36       ` John Goerzen
        @ 2021-01-01  5:39       ` John Goerzen
          2021-01-02 13:04         ` Sergey Matveev
          2021-01-06 17:15         ` Sergey Matveev
          2 siblings, 2 replies; 13+ messages in thread
        From: John Goerzen @ 2021-01-01  5:39 UTC (permalink / raw)
          To: Sergey Matveev; +Cc: nncp-devel
        
        
        On Wed, Dec 30 2020, Sergey Matveev wrote:
        
        > But here is another problem: the whole packet is hashed from 
        > beginning
        > to the end. And changing that single block will require the 
        > entire
        > hashing calculation to be done again. It is good that we get rid 
        > of
        > temporary file at all, but we still need to sequentially write 
        > all the
        > data from stdin, make seek/overwrite and then sequentially read 
        > all the
        > data again for its checksumming.
        >
        > We can avoid it by using hash trees, Merkle trees. Changing of 
        > that
        >
        
        I think I'm following here :-)
        
        I'm not familiar with Merkle trees, but one solution to this would 
        be to hash it in the order it's written at verification time: save 
        the first 128K in RAM, read the rest, and when you're at EOF, 
        supply that first 128K to the hasher and then you'd have your 
        calculation.
        
        - John
        
        ^ permalink raw reply	[flat|nested] 13+ messages in thread

      • • * Re: nncp-bundle, nncp-caller, and .seen not working?
            2021-01-01  5:39       ` John Goerzen
          @ 2021-01-02 13:04         ` Sergey Matveev
            2021-01-06 17:15         ` Sergey Matveev
            1 sibling, 0 replies; 13+ messages in thread
          From: Sergey Matveev @ 2021-01-02 13:04 UTC (permalink / raw)
            To: nncp-devel
          
          [-- Attachment #1: Type: text/plain, Size: 716 bytes --]
          
          *** John Goerzen [2020-12-31 23:39]:
          >I'm not familiar with Merkle trees, but one solution to this would be to hash
          >it in the order it's written at verification time: save the first 128K in
          >RAM, read the rest, and when you're at EOF, supply that first 128K to the
          >hasher and then you'd have your calculation.
          
          Of course that is also solution and very simple one indeed. But just no
          so beautiful and elegant like general tree hashing mode. Current hashing
          even compatible with ordinary b2sum commands, but tree hashing of course
          is won't be anymore too. Soon will look at all of this closer.
          
          -- 
          Sergey Matveev (http://www.stargrave.org/)
          OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF
          
          [-- Attachment #2: signature.asc --]
          [-- Type: application/pgp-signature, Size: 833 bytes --]
          
          ^ permalink raw reply	[flat|nested] 13+ messages in thread

        • * Re: nncp-bundle, nncp-caller, and .seen not working?
            2021-01-01  5:39       ` John Goerzen
            2021-01-02 13:04         ` Sergey Matveev
          @ 2021-01-06 17:15         ` Sergey Matveev
            2021-01-06 18:55           ` Sergey Matveev
            2021-01-07 17:56           ` John Goerzen
            1 sibling, 2 replies; 13+ messages in thread
          From: Sergey Matveev @ 2021-01-06 17:15 UTC (permalink / raw)
            To: nncp-devel
          
          [-- Attachment #1: Type: text/plain, Size: 4222 bytes --]
          
          *** John Goerzen [2020-12-31 23:39]:
          >> We can avoid it by using hash trees, Merkle trees. Changing of that
          >
          >I think I'm following here :-)
          
          Well, I tried to get rid of temporary files and make different checksum
          calculation. I tried own Merkle tree implementation, but did not finish
          with it. I decided to use just hash(hash(header) || hash(body)), where
          body is hashed when written on the disk, and at the end, when we
          overwrite the header, we just calculate two more hashes. I tried BLAKE3
          and it was twice as fast, without any parallelization, on my hardware --
          just was curious about it.
          
          But I failed. All of that can be done with single encrypted packet. But
          when you want to send via some other nodes, then all that wrapped
          encrypted packets are created *on the fly* simultaneously. Overwriting
          headers requires seeking ability inside that *unaligned* encrypted
          streams. So hardly can be done.
          
          Moreover, I have started to refactor nncp-exec (to be able to use
          temporary files at least). And I remembered why it was so simple and
          even kept everything in memory. If nncp-exec transfers very big volume
          of data, then it should be chunked (depending on configuration of
          course) as an ordinary nncp-file. That means creation of multiple NNCP
          packets, that can be reordered during the transfer. You can not run
          any command when not the whole data is ready. Moreover all the data is
          inside multiple encrypted packets.
          
          So there must be some complicated code that somehow must find the .meta
          packet containing information about all other chunks. Then it must find
          that chunks somehow. It differs from current nncp-file/toss operations,
          because they decrypt packets and store them on the filesystem. Ok, let's
          just decrypt nncp-exec's chunks too and store on the filesystem and
          do literally something like "nncp-reass -stdout nncp-exec's.meta |".
          
          So either we have very complex code trying to find chunks *inside*
          encrypted packets (obviously by parsing them again and again, or by
          having some additional state anywhere), or we just have an ordinary
          nncp-file-like transmission.
          
          But current nncp-reass has options for being able to keep fragments on
          the disk even after reassembling. It is able to feed them and delete at
          once, not requiring to have double disk space (for chunks and whole
          reassembled file). If it will be feeding the file to some external
          command -- what kind of behaviour should it have? Delete all files if
          that external command fails? Keep them? Possibly it is better to
          reassemble them on dedup-ed ZFS dataset and pass that reassembled file
          as an argument? I just simply can not tell any of those questions at all.
          
          So I remembered why nncp-exec is so simple. If you want to pass big
          volumes of data -- then pass them using nncp-file, and then nncp-exec
          commands passing them that filename. Remote side's exec configuration
          will work with incoming files anyhow user wants. nncp-exec packet can
          reach destination much earlier than nncp-file's data itself. It can just
          fail and fail every time nncp-toss met nncp-exec packet, until nncp-file
          will come and (probably) be reassembled). I will prefer to use some kind
          of cron-ed daemon that will check new files in some special directory
          with reassembled files, that are treated like tasks to some command.
          
          It is the most flexible way, without huge code complications and many
          hard-to-answer questions/decisions. As I remember, FidoNet had similar
          kind of things, where big binary files were transferred with "attached"
          additional packet, referencing that binary file.
          
          That is why nncp-exec packets can not be chunked, can not be big and no
          temporary files are made, because anyway those -exec packets are
          relatively small (well, actually there could be multimegabyte email
          messages, but I assume that NNCP is running on computers with multiple
          hundreds MB of RAM). Their "nature" is to be streamed inside some
          command, then tossed, repeated again if it fail. If they are big, then
          nncp-file must be used for data transfer itself (chunked, reassembled,
          and so on).
          
          -- 
          Sergey Matveev (http://www.stargrave.org/)
          OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF
          
          [-- Attachment #2: signature.asc --]
          [-- Type: application/pgp-signature, Size: 833 bytes --]
          
          ^ permalink raw reply	[flat|nested] 13+ messages in thread

        • • * Re: nncp-bundle, nncp-caller, and .seen not working?
              2021-01-06 17:15         ` Sergey Matveev
            @ 2021-01-06 18:55           ` Sergey Matveev
              2021-01-07 17:56           ` John Goerzen
              1 sibling, 0 replies; 13+ messages in thread
            From: Sergey Matveev @ 2021-01-06 18:55 UTC (permalink / raw)
              To: nncp-devel
            
            [-- Attachment #1: Type: text/plain, Size: 601 bytes --]
            
            *** Sergey Matveev [2021-01-06 20:15]:
            >That is why nncp-exec packets can not be chunked, can not be big and no
            >temporary files are made, because anyway those -exec packets are
            >relatively small
            
            Also that means that I see no profit in ability to create non-compressed
            nncp-exec packets. Those packets them are small and even if contain
            incompressible data, then Zstandard quickly fallbacks to incompressible
            mode of operation, taking unnoticeable CPU and space overhead amount.
            
            -- 
            Sergey Matveev (http://www.stargrave.org/)
            OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF
            
            [-- Attachment #2: signature.asc --]
            [-- Type: application/pgp-signature, Size: 833 bytes --]
            
            ^ permalink raw reply	[flat|nested] 13+ messages in thread

          • * Re: nncp-bundle, nncp-caller, and .seen not working?
              2021-01-06 17:15         ` Sergey Matveev
              2021-01-06 18:55           ` Sergey Matveev
            @ 2021-01-07 17:56           ` John Goerzen
              1 sibling, 0 replies; 13+ messages in thread
            From: John Goerzen @ 2021-01-07 17:56 UTC (permalink / raw)
              To: Sergey Matveev; +Cc: nncp-devel
            
            
            On Wed, Jan 06 2021, Sergey Matveev wrote:
            
            > That is why nncp-exec packets can not be chunked, can not be big 
            > and no
            > temporary files are made, because anyway those -exec packets are
            > relatively small (well, actually there could be multimegabyte 
            > email
            > messages, but I assume that NNCP is running on computers with 
            > multiple
            > hundreds MB of RAM). Their "nature" is to be streamed inside 
            > some
            > command, then tossed, repeated again if it fail. If they are 
            > big, then
            > nncp-file must be used for data transfer itself (chunked, 
            > reassembled,
            > and so on).
            
            Thank you for the new update and all your work on NNCP.  This will 
            be a very nice improvement!
            
            The option to use an encrypted temporary file for nncp-exec will 
            be perfect for me and the temp file solution is fine.  chunked for 
            NNCP isn't important for my use case; although the files are 
            sometimes many GB in size, that is still smaller than the 
            transport (whether network or airgapped) and thus the resumable 
            transfers are sufficient for my needs.
            
            Thanks again!
            
            - John
            
            ^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2021-01-07 17:57 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-30  6:37 nncp-bundle, nncp-caller, and .seen not working? John Goerzen
2020-12-30 14:08 ` Sergey Matveev
2020-12-30 15:37   ` John Goerzen
2020-12-30 15:44     ` John Goerzen
2020-12-30 19:10     ` Sergey Matveev
2020-12-30 19:16       ` Sergey Matveev
2021-01-01  5:36       ` John Goerzen
2021-01-02 12:56         ` Sergey Matveev
2021-01-01  5:39       ` John Goerzen
2021-01-02 13:04         ` Sergey Matveev
2021-01-06 17:15         ` Sergey Matveev
2021-01-06 18:55           ` Sergey Matveev
2021-01-07 17:56           ` John Goerzen