Problem

Because of the sequence number in the AddRoots operation, we must add them sequentially and have them land on chain in separate tipsets. The sequence number comes from the chain and we fetch it when we need it with an eth_call. This causes two problems:

1. Slows down the ability to add pieces for an individual proofset; in practice it's at least 1 minute between successful adds.
2. Introduces race conditions such that you must wait for confirmation for another piece before adding another. This is fine if you're doing this all from a single instance of the SDK because after feat(pdp): add transaction tracking for root additions with server verification #98 and feat(pdp): add transaction tracking for root additions (synapse branch) filecoin-project/curio#530 roll out we'll have that in place, but you couldn't do multiple SDK instances, for example, without some kind of coordination.

Solution

While not solving completely for this problem (separate instances of the SDK), we can unblock parallelised piece pipelining. We have a few things we can use:

1. Ee can know when a new root has added on chain (now, after 98 and curio 530)
2. We can optionally use the AddRoots operation to add multiple roots
3. We are (currently) not limited in our ability to upload pieces to a Curio server, it'll just take them and park them for later use in AddRoots

This kind of use should be workable:

const results = await Promise.all([
  storage.upload(blob1),
  storage.upload(blob2),
  storage.upload(blob3),
  storage.upload(blob4)
])

Because these are async tasks, they'll all start doing CommP and pushing bytes to Curio. But currently they'll start to fail fairly quickly after that unless the blob sizes make for very different timings. They'll all get to AddRoots and pick the same sequence number, one of them will pass but the other 3 will fail.

Currently, you'd have to do this:

await storage.upload(blob1)
await storage.upload(blob2)
await storage.upload(blob3)
await storage.upload(blob4)

With opportunity for parallelism, waiting 60 seconds or more between each one.

Instead, what we should do, inside a SynapseStorage instance, is keep track of the last AddRoots operation and be able to determine the status (done, or waiting). In the case of "done", we just continue with a fresh AddRoots operation and assume that the last one landed on chain and our looked-up sequence number will be correct for the next one. In the case of "waiting", we push our root to a batch, to be processed later (and also insert a Promise that can be resolved so we can return that to the user). When an AddRoots operation has finished, it should look at the batch to see if it's nonempty. If there are new roots to be added, it should bundle them all in a single AddRoots call and proceed to wait for success. Once successful, all the Promises can be resolved for that batch (continue to loop if there are new ones in the batch).