simulators/eth2/engine: Fix Optimistic Sync Tests + Refactor Allow Client Restarts + No Viable Head Test Case

[marioevz]

Changes Included

• Fixes optimistic sync tests by properly waiting until the beacon node becomes optimistic, by querying execution_optimistic property from the block in the REST API
• Refactors/cleans up testnet code to allow startup/shutdown of a client later during the test
• Introduces test No viable head due to optimistic sync from https://github.com/txrx-research/TestingTheMerge/blob/main/tests/engine-api.md

Requires: #604 #636
Supersedes: #631

cc @zilm13 @mkalinin

[marioevz]

Hi @zilm13, this PR adds No viable head due to optimistic sync from this link: https://github.com/txrx-research/TestingTheMerge/blob/main/tests/engine-api.md

Teku is having an unexpected behavior on this test, during step Check that the node stays optimistic, i.e. execution_optimistic flag in Beacon API responses is true; We are querying block BeaconBlock(P10), and test expects execution_optimistic==true, however, the block is not available because the head of the node is further back in the chain:

• We have a chain that introduces bellatrix in slot 128
• At slot 130, the first beacon block with an execution payload is introduced, P1 (Test spec describes P0 as transition block, but we are sending 1 payload less)
• At slot 139 we have P10, which is the last valid payload from the importer's perspective
• At slot 140 we start spoofing the responses with SYNCING up to slot 161
• At slot 162 we spoof payload with INVALID and latestValidHash==P10
• We expect head to re-org to slot 139, but querying head returns slot 128, and test fails here

I have logs for this or if you prefer the test can be run in this branch using

./hive --client go-ethereum,teku-bn,teku-vc --sim eth2/engine --sim.limit "/no-viable-head-due-to-optimistic-sync"

[mkalinin]

• We expect head to re-org to slot 139, but querying head returns slot 128, and test fails here

We should remove this expectation from the test. When CL is in this state (there is no viable head) it isn't specified what the head it should return, it leaves a room for implementations to decide which is the most convenient response to getHead. The only thing that is required is to stay optimistic. As long as BN is optimistic (optimistic is a subset of syncing state) it's not serving VC API which is ultimately an expected behaviour when there is no viable head to vote for and to build a block atop of.

[marioevz]

• We expect head to re-org to slot 139, but querying head returns slot 128, and test fails here

We should remove this expectation from the test. When CL is in this state (there is no viable head) it isn't specified what the head it should return, it leaves a room for implementations to decide which is the most convenient response to getHead. The only thing that is required is to stay optimistic. As long as BN is optimistic (optimistic is a subset of syncing state) it's not serving VC API which is ultimately an expected behaviour when there is no viable head to vote for and to build a block atop of.

I updated the test to only check whether the head is optimistic, now Teku is passing this part but I am having trouble with the clients syncing with each other after this, but the test is still failing.

I think it's safe to merge this test and we could submit a follow up PR if there are any fixes required.