Greetings, again!

*** Leah Neukirchen <leah@vuxu.org> [2017-04-25 16:27]:
>1) AFAICS, nncp-xfer will transfer the whole packet at once, even if a
>partial nncp-call succeeded, is that how it's supposed to be?  I'm not
>sure nncp-xfer can do partial transmissions?
>2) nncp-xfer with a quota would be useful... (say, I partition 10G of
>my notebook disk space for nncp transfers, so transfering a 20G file
>will take two roundtrips.)

As I said in previous mail, I decided to implement automatic splitting
of files. Its draft initial version is already somehow working:
https://git.cypherpunks.ru/cgit.cgi/nncp.git/commit/?h=develop&id=66a1f121c12a73e49848e7946ceda8b75095b32e

If you specify -chunk XXX (where XXX is chunk size in KiBs) option to
nncp-file, then it will transmit not the whole file, but:
FILE.nncp.meta, FILE.nncp.part0, FILE.nncp.part1, .... Meta file
contains full FILE's size and all hashes of each of its part. nncp-toss
does NOT automatically do anything with them, except just saving as an
ordinary files. I thought about that and it seems very cumbersome to do
something at once. It is much more simpler to reassemble the whole file
in a separate process.

So there appeared nncp-reass utility that searches for those meta-files,
determines do we have enough chunks available, verifies their integrity
and reassembles the file (all those parts can be reassembled manually
just doing cat FILE.part0 FILE.part1 > FILE).

This is rather simple changes, but they allow to prepare huge quantity
of data for transferring through small storage devices. When all parts
are available -- you can reassemble the whole file.

Some notes:

* creating of outbound packets (partXXX, .meta) is done streamingly
* reassembling process, after all integrity tests are passed, simply
  copies all chunks in the destination file, simultaneously removing
  them at once (some kind of: cat partXXX >> file ; rm partXXX ; ...),
  so you have to have free space only for single chunk additionally. If
  you have got 100 GB filesystem and want to transmit 99 GB file with
  1GB chunks, then you must only have 99+1 GB of free space
* if you have got filesystem with deduplication capabilities, then all
  those cat * >> file should be done very efficiently, without any
  additional space requirements

I will write more documentation about that feature soon.

So, with your usecase when you have got relatively small transferable
media: you just

* nncp-file -chunk XXX HUGEFILE
* nncp-xfer to transferable device until disk space allows to hold at
  least some of packets (you can always use -nice for prioritizing
  packets)
* carry device to remote side
* nncp-xfer on remote side to copy that packets from device
* nncp-toss to process encrypted packets and save some of HUGEFILE's chunks
* repeat that process with nncp-xfer-s until all remaining are done
* nncp-reass on remote side to reassemble HUGEFILE

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF