rfe: multithread splitstream creation

[allisonkarlitskaya]

cfsctl oci pull is kinda slow right now.

When converting the container image .tar.gz into splitstreams it does the gzip decompression, sha256 verification (of the underlying .tar), fs-verity calculations (of the split objects), fdatasync() (of the split objects), zstd compression of the splitstream, and finally fs-verity and fdatasync() of the splitstream, all in a single thread.

mkcomposefs is smarter here: it spawns a bunch of worker threads to do the fs-verity and fdatasync() on the split objects.

This would be pretty easy for us to do, and Rust would help us make sure we're doing it right, except the way that splitstreams are currently constructed is pretty linear: you need to get the object ID of each object to write into the stream before you can proceed to the next one. But: we're just writing to a Vec<u8> anyway, so we can easily go back in fill those parts in, or come up with some kind of a "chunks" model where we merge all the chunks into the linear stream at the end, or any other design. We could then make some of the chunks be promises or something — I'm really not sure how this works in Rust...

[allisonkarlitskaya]

This also needs to work for cfsctl create-image which scans the filesystem. We might also try to opportunistically use reflinking in that case.

[allisonkarlitskaya]

For the splitstream case: I think some sort of model like this:

• we have two queues:
  • some kind of a "work queue" which is handled by workers
  • some kind of an "output" queue which the splitstream segments get written to
• one thread does the download/decompress/tar decoding
• for each inline chunk, write it directly to the output queue
• for each external chunk:
  • create a new ephemeral queue (as a sort of promise mechanism)
  • add an item to the worker queue containing:
    • the data
    • a promise queue Sender side
  • write the promise queue Receiver side to the "output" queue
• the workers receive (data, Sender) pairs and write the data to the repo, sending the result (the fs-verity digest) to the Sender.
• finally, have another thread responsible for composing the output. It reads inline chunks and promises and writes the splitstream.

[allisonkarlitskaya]

For the cfsctl create-image case it's going to be harder. We could imagine a filesystem tree where external files contain promises for fs-verity digests instead of the fs-verity digests but this is a serious complication of the existing structure. We could model it, instead, as a queue of (filename, fd) and move the read/checksum/write of the fd to a worker, producing another queue of (filename, checksum). Ensuring ordering would be annoying in this case, though...

edit: although maybe we don't have to worry about this so much. When creating from the filesystem we aren't doing multiple layers or anything, so the order of operations is actually not very important...

[alexlarsson]

This will make splitstream digests potentially different for each host. Is that not a problem? I guess its already like though, as we don't hardcode a specific zstd compressor version anyway.

[allisonkarlitskaya]

the title of this is a bit misleading: it's not really about multithreading the creation of the splitstream itself so much as the storing of the externally-referred objects (ie: the file content). The process of building and compressing the splitstream would still be serial...

But ya, in general, with splitstreams:

• we never take the content hash of the splitstream file itself
• we take the fs-verity hash, but we don't use it for anything other than to refer to the splitstream from the current host. ztstd stability is only one reason for that. another is that maybe we're on a host that wants sha512, or maybe we change to 64k block size, or someone decides to use a salt, or ....

[cgwalters]

When converting the container image .tar.gz into splitstreams it does the gzip decompression, sha256 verification (of the underlying .tar), fs-verity calculations (of the split objects), fdatasync() (of the split objects), zstd compression of the splitstream, and finally fs-verity and fdatasync() of the splitstream, all in a single thread.

Another inefficiency here is each file from the tar stream is copied to memory first too.

Anyways yeah, right now doing a composefs-based install is really noticeably slow.