DO NOT MERGE: synthetic parallel execution test framework

[graydon]

This adds a special mode you can expose to live traffic from mainnet or testnet (online using run or, more commonly, offline using catchup) to test out the new p23 parallel-execution code path for soroban phases.

The way it works is that just before running a sequential soroban phase, it:

• Synthesizes a fake parallel phase using the same phase-building path used in p23 txset nomination
• Runs that phase on a captured throwaway copy of the pre-state of the phase
• Captures the results of that (txresults and txmetas) into some buffers

It then proceeds to run the normal sequential phase as usual, and compares the captured parallel results with the sequential results, logging any differences as errors.

Its behaviour is controlled by two environment variables:

• STELLAR_TEST_PARALLEL_EXECUTION must be set to a nonzero number, which will be used as the parallelism factor for the synthesized parallel phase. So setting STELLAR_TEST_PARALLEL_EXECUTION=4 will make and run a 4-way parallel phase on 4 threads.
• STELLAR_COMPARISON_TOLERANCE is an optional but recommended comma-separated list of difference types to tolerate and not report as errors. Currently I recommend running with STELLAR_COMPARISON_TOLERANCE=event_topics,fees though other options are possible (browse the code). This is necessary because there are some small observable differences between p22 and p23 executions, both arising from minor protocol changes and also from the very fact of running in parallel (eg. fees go way down).

So overall, you probably want to run something like:

$ STELLAR_COMPARISON_TOLERANCE=event_topics,fees STELLAR_TEST_PARALLEL_EXECUTION=4 \
  ./src/stellar-core --conf ~/stellar-mainnet.cfg --console catchup current/1000

To help diagnose differences, it will also write them to some organized files in the filesystem, under the directory parallel-tx-diffs. For example my version just wrote these files:

parallel-tx-diffs/ledger-58007058
parallel-tx-diffs/ledger-58007058/tx-81
parallel-tx-diffs/ledger-58007058/tx-81/tx-envelope.json
parallel-tx-diffs/ledger-58007058/tx-81/meta-parallel.json
parallel-tx-diffs/ledger-58007058/tx-81/summary.txt
parallel-tx-diffs/ledger-58007058/tx-81/meta-sequential.json
parallel-tx-diffs/ledger-58007058/tx-61
parallel-tx-diffs/ledger-58007058/tx-61/tx-envelope.json
parallel-tx-diffs/ledger-58007058/tx-61/meta-parallel.json
parallel-tx-diffs/ledger-58007058/tx-61/summary.txt
parallel-tx-diffs/ledger-58007058/tx-61/meta-sequential.json
...

[dmkozh]

This looks sensible overall, I think the main issue is weird/incorrect fee bump handling.