core/state/snapshot: write snapshot generator in batch

[rjl493456442]

This PR fixes a flaw in the snapshot.

Scenario Description

0. Have a good snapshot
1. Start geth without --snapshot, import a few blocks
2. Start geth with --snapshot, it starts deleting it
3. Crash
4. Restart Geth

WARN [01-12|20:47:02.323] Loaded snapshot journal                  diskroot="fc6ca8…0440f3" diffs=unmatched
[...]
BAD BLOCK

Analysis

In the snapshot, we have three different components:

• snapshot root: indicates the state hash of the disk layer
• snapshot generator: indicates the status of the snapshot disk layer(generating, wiping, done)
• snapshot diff layer journal

In step 0, we have a complete snapshot, so the snapshot generator is marked as "DONE"
In step 1, we introduce the gap between the snapshot and chain, the snapshot becomes useless
In step 2, we try to wipe the entire stale snapshot

The critical code of step 2

    // Wipe any previously existing snapshot from the database if no wiper is
    // currently in progress.
    if wiper == nil {
        wiper = wipeSnapshot(diskdb, true)
    }
    // Create a new disk layer with an initialized state marker at zero
    rawdb.WriteSnapshotRoot(diskdb, root)

    base := &diskLayer{
        diskdb:     diskdb,
        triedb:     triedb,
        root:       root,
        cache:      fastcache.New(cache * 1024 * 1024),
        genMarker:  []byte{}, // Initialized but empty!
        genPending: make(chan struct{}),
        genAbort:   make(chan chan *generatorStats),
    }
    go base.generate(&generatorStats{wiping: wiper, start: time.Now()})
    log.Debug("Start snapshot generation", "root", root)
    return base

Here the NEW ROOT is written as the snapshot root maker without updating the snapshot generator.

What's more, during this long wiping procedure, maybe there is no new diff layer created, which means the snapshot generator is not updated. So the generator is still the old one, which is marked as DONE.

Crash happens

Then restart, we have an invalid snapshot, with the snapshot root aligned with the chain but the status of snapshot generator
is DONE.

[holiman]

LGTM, and looks like it's backed by quite rigorous testing aswell 👍