[FEAT] - Snapshot at predictable points

[jasagredo]

Internal

Describe the feature you'd like
In order to make sure that multiple nodes take the same snapshots, it would be useful to implement an alternative snapshotting policy that decides based on the block number. I propose something like:

// currently
{ 
  "SnapshotInterval": 5000       // seconds
}

// proposal
{ 
  "SnapshotInterval": "5000s"       // seconds
}
{ 
  "SnapshotInterval": "100b"        // every @blockno `mod` 100 == 0@
}

We would have to pass to onDiskShouldTakeSnapshot also the current block number.

[amesgen]

We talked about this today with @jpraynaud. It will most likely make sense to create snapshots based on the slot number, namely near the end of chunk files (encompassing 10*k = 21600 slots each), such that one can simply take the (complete) ImmutableDB chunks up till this point.

Additionally, it makes sense to give these snapshots a special suffix such that they are not deleted by the node istelf. Rather, it will be the job of the Mithril signer to delete them.

[amesgen]

Just had another chat with @jpraynaud, where @jpraynaud raised the fact that it would be convenient for Mithril if the node would write these predictable snapshots directly in the format that Mithril would sign. This is mostly to simplify Mithril deployment, as the Mithril signer would otherwise need to invoke some CLI tool (think snapshot-converter) or depend on this functionality via FFI.

In the future, the (WIP) canonical ledger state format would be the right tool for use here. Before that, the Haskell in-memory format would be the closest approximation (as the LMDB/lsm-tree formats are even less portable).

It seemed plausible to us that the node could directly create snapshots in that format, ie a node using the LMDB or lsm-tree backend could write an in-memory¹ snapshot (either in the current format, or in "the" canonical format in the future, potentially with a version specifier). The only concern at this point are the performance problems of writing out ledger snapshots as tracked by #868.

Note that the idea is that if the ledger snapshot format chosen for Mithril is different from what the node uses "natively" (eg in the future when everyone uses the lsm-tree backend), one would write the snapshot in the in-memory format by converting to it "on the fly" in constant space, without fully materializing an entire ledger state in memory.

Footnotes

1. In particular, SPOs are currently still being told to use the in-memory backend, and they are the ones signing in Mithril. ↩

[amesgen]

FTR: I created separate tickets for the two related refinements that we also discussed just now, such that this ticket purely tracks the aspect of predictable snapshots:

• Optional random delay when creating snapshots #1573
• Allow to create snapshot in non-native format #1574