*** John Goerzen [2020-12-31 23:39]:
>> We can avoid it by using hash trees, Merkle trees. Changing of that
>
>I think I'm following here :-)

Well, I tried to get rid of temporary files and make different checksum
calculation. I tried own Merkle tree implementation, but did not finish
with it. I decided to use just hash(hash(header) || hash(body)), where
body is hashed when written on the disk, and at the end, when we
overwrite the header, we just calculate two more hashes. I tried BLAKE3
and it was twice as fast, without any parallelization, on my hardware --
just was curious about it.

But I failed. All of that can be done with single encrypted packet. But
when you want to send via some other nodes, then all that wrapped
encrypted packets are created *on the fly* simultaneously. Overwriting
headers requires seeking ability inside that *unaligned* encrypted
streams. So hardly can be done.

Moreover, I have started to refactor nncp-exec (to be able to use
temporary files at least). And I remembered why it was so simple and
even kept everything in memory. If nncp-exec transfers very big volume
of data, then it should be chunked (depending on configuration of
course) as an ordinary nncp-file. That means creation of multiple NNCP
packets, that can be reordered during the transfer. You can not run
any command when not the whole data is ready. Moreover all the data is
inside multiple encrypted packets.

So there must be some complicated code that somehow must find the .meta
packet containing information about all other chunks. Then it must find
that chunks somehow. It differs from current nncp-file/toss operations,
because they decrypt packets and store them on the filesystem. Ok, let's
just decrypt nncp-exec's chunks too and store on the filesystem and
do literally something like "nncp-reass -stdout nncp-exec's.meta |".

So either we have very complex code trying to find chunks *inside*
encrypted packets (obviously by parsing them again and again, or by
having some additional state anywhere), or we just have an ordinary
nncp-file-like transmission.

But current nncp-reass has options for being able to keep fragments on
the disk even after reassembling. It is able to feed them and delete at
once, not requiring to have double disk space (for chunks and whole
reassembled file). If it will be feeding the file to some external
command -- what kind of behaviour should it have? Delete all files if
that external command fails? Keep them? Possibly it is better to
reassemble them on dedup-ed ZFS dataset and pass that reassembled file
as an argument? I just simply can not tell any of those questions at all.

So I remembered why nncp-exec is so simple. If you want to pass big
volumes of data -- then pass them using nncp-file, and then nncp-exec
commands passing them that filename. Remote side's exec configuration
will work with incoming files anyhow user wants. nncp-exec packet can
reach destination much earlier than nncp-file's data itself. It can just
fail and fail every time nncp-toss met nncp-exec packet, until nncp-file
will come and (probably) be reassembled). I will prefer to use some kind
of cron-ed daemon that will check new files in some special directory
with reassembled files, that are treated like tasks to some command.

It is the most flexible way, without huge code complications and many
hard-to-answer questions/decisions. As I remember, FidoNet had similar
kind of things, where big binary files were transferred with "attached"
additional packet, referencing that binary file.

That is why nncp-exec packets can not be chunked, can not be big and no
temporary files are made, because anyway those -exec packets are
relatively small (well, actually there could be multimegabyte email
messages, but I assume that NNCP is running on computers with multiple
hundreds MB of RAM). Their "nature" is to be streamed inside some
command, then tossed, repeated again if it fail. If they are big, then
nncp-file must be used for data transfer itself (chunked, reassembled,
and so on).

-- 
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263  6422 AE1A 8109 E498 57EF