Kintsugi 🍵 #2768

paulhauner · 2021-11-03T04:56:49Z

Description

This is the canonical branch for the Kintsugi testnet.

For instructions on running the Kintsugi testnet, see #2830.

This PR contains the changes required for Lighthouse to run "the merge" specifications. It does not deliver a production-ready implementation of "the merge" for Lighthouse, instead it provides the necessary components for Lighthouse to run on testnets and progress towards a production implementation.

Where this PR touches code that will run on the pre-merge (i.e., current) Beacon Chain, any changes should be production ready. For post-merge components (e.g., execution_layer), some degree of unfinished-ness is acceptable. After all, "the merge" specification is not yet complete.

TODO

This section lists tasks required before this branch can be merged into unstable.

N/A

realbigsean · 2021-11-04T14:59:56Z

Upgrade to v1.1.4 spec tests, fixing any consensus changes.

I can work on this

paulhauner · 2021-11-11T05:21:00Z

I just rebased this branch onto unstable. It will fail CI until I push another fix (#2799).

beacon_node/execution_layer/src/engine_api/http.rs

paulhauner · 2021-11-28T22:53:07Z

I just rebased to fix a conflict with unstable and to fix a merge (instead of squash-merge) of #2796.

This will fail CI until we merge #2829.

michaelsproul

I have reviewed the ./consensus changes thoroughly and I'm happy to merge this to unstable.

Makefile

beacon_node/beacon_chain/src/beacon_chain.rs

consensus/proto_array/src/proto_array_fork_choice.rs

consensus/proto_array/src/ssz_container.rs

consensus/types/Cargo.toml

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs

beacon_node/src/config.rs

divagant-martian

I checked ./lighthouse_network and ./network. it looks good to me

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs

beacon_node/lighthouse_network/src/rpc/protocol.rs

common/slot_clock/src/lib.rs

Added Execution Payload from Rayonism Fork Updated new Containers to match Merge Spec Updated BeaconBlockBody for Merge Spec Completed updating BeaconState and BeaconBlockBody Modified ExecutionPayload<T> to use Transaction<T> Mostly Finished Changes for beacon-chain.md Added some things for fork-choice.md Update to match new fork-choice.md/fork.md changes ran cargo fmt Added Missing Pieces in eth2_libp2p for Merge fix ef test Various Changes to Conform Closer to Merge Spec

* Update to v1.1.0-beta.4 (squash of #2548) * SSZ, cached tree hash, EF tests

* Remove unchecked arith from ssz_derive * Address clippy lints in block_verfication * Use safe math for is_valid_gas_limit

* Add payload verification status to fork choice * Pass payload verification status to import_block * Add valid back-propagation * Add head safety status latch to API * Remove ExecutionLayerStatus * Add execution info to client notifier * Update notifier logs * Change use of "hash" to refer to beacon block * Shutdown on invalid finalized block * Tidy, add comments * Fix failing FC tests * Allow blocks with unsafe head * Fix forkchoiceUpdate call on startup

* update initializing from eth1 for merge genesis * read execution payload header from file lcli * add `create-payload-header` command to `lcli` * fix base fee parsing * Apply suggestions from code review * default `execution_payload_header` bool to false when deserializing `meta.yml` in EF tests Co-authored-by: Paul Hauner <[email protected]>

* Freshen Cargo.lock * Fix gossip worker * Update map_fork_name_with

* Fix arbitrary check kintsugi * Add merge chain spec fields, and a function to determine which constant to use based on the state variant * increment spec test version * Remove `Transaction` enum wrapper * Remove Transaction new-type * Remove gas validations * Add `--terminal-block-hash-epoch-override` flag * Increment spec tests version to 1.1.5 * Remove extraneous gossip verification ethereum/consensus-specs#2687 * - Remove unused Error variants - Require both "terminal-block-hash-epoch-override" and "terminal-block-hash-override" when either flag is used * - Remove a couple more unused Error variants Co-authored-by: Paul Hauner <[email protected]>

* Change base_fee_per_gas to Uint256 * Add custom (de)serialization to ExecutionPayload * Fix errors * Add a quoted_u256 module * Remove unused function * lint * Add test * Remove extra line Co-authored-by: Paul Hauner <[email protected]>

* Fix fork choice after rebase * Remove paulhauner warp dep * Fix fork choice test compile errors * Assume fork choice payloads are valid * Add comment * Ignore new tests * Fix error in test skipping

* Allocate less memory (3GB) in engine tests * Run cargo format * Remove tx too large test Co-authored-by: Michael Sproul <[email protected]>

* Added ForkchoiceUpdatedV1 & GetPayloadV1 * Added ExecutePayloadV1 * Added new geth test vectors * Separated Json Object/Serialization Code into file * Deleted code/tests for Requests Removed from spec * Finally fixed serialization of null '0x' * Made Naming of JSON Structs Consistent * Fix clippy lints * Remove u64 payload id * Remove unused serde impls * Swap to [u8; 8] for payload id * Tidy * Adjust some block gen return vals * Tidy * Add fallback when payload id is unknown * Remove comment Co-authored-by: Mark Mackey <[email protected]>

* Unify loading of eth2_network_config * Apply overrides at lighthouse binary level * Remove duplicate override values * Add merge values to existing net configs * Make override flags global * Add merge fields to testing config * Add one to TTD * Fix failing engine tests * Fix test compile error * Remove TTD flags * Move get_eth2_network_config * Fix warn * Address review comments

Fix max packet sizes Fix max_payload_size function Add merge block test Fix max size calculation; fix up test Clear comments Add a payload_size_function Use safe arith for payload calculation Return an error if block too big in block production Separate test to check if block is over limit

* Start v1.1.5 updates * Implement new payload creation logic * Tidy, add comments * Remove unused error enums * Add validate payload for gossip * Refactor validate_merge_block * Split payload verification in per block processing * Add execute_payload * Tidy * Tidy * Start working on new fork choice tests * Fix failing merge block test * Skip block_lookup_failed test * Fix failing terminal block test * Fixes from self-review * Address review comments

* Fix makefile * Return on invalid finalized block * Fix todo in gossip scoring * Require --merge for --fee-recipient * Bump eth2_serde_utils * Change schema versions * Swap hash/uint256 test_random impls * Use default for ExecutionPayload::empty * Check for DBs before removing * Remove kintsugi docker image * Fix CLI default value

* Remove TODOs * Fix typo

paulhauner · 2021-12-02T03:32:45Z

I've just rebased to fix a merge conflict.

paulhauner added the kintsugi 🍵 label Nov 3, 2021

paulhauner changed the base branch from stable to unstable November 3, 2021 04:57

paulhauner mentioned this pull request Nov 3, 2021

Merge F2F #2620

Closed

7 tasks

paulhauner force-pushed the kintsugi branch from 06bcbdd to 4b0ab58 Compare November 3, 2021 05:23

paulhauner force-pushed the kintsugi branch from f817784 to 954a9c2 Compare November 11, 2021 05:20

michaelsproul reviewed Nov 11, 2021

View reviewed changes

beacon_node/execution_layer/src/engine_api/http.rs Outdated Show resolved Hide resolved

michaelsproul mentioned this pull request Nov 11, 2021

Allocate less memory (3GB) in engine tests #2800

Closed

paulhauner mentioned this pull request Nov 15, 2021

Skip memory intensive engine test #2809

Merged

paulhauner force-pushed the kintsugi branch 2 times, most recently from 553ea8b to 6f2a3c6 Compare November 28, 2021 22:53

michaelsproul added the backwards-incompat Backwards-incompatible API change label Nov 28, 2021

paulhauner mentioned this pull request Nov 28, 2021

Kintsugi Tracking 🍵 #2830

Closed

12 tasks

michaelsproul approved these changes Nov 28, 2021

View reviewed changes

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs Outdated Show resolved Hide resolved

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs Outdated Show resolved Hide resolved

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/src/config.rs Outdated Show resolved Hide resolved

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/src/config.rs Outdated Show resolved Hide resolved

divagant-martian approved these changes Nov 29, 2021

View reviewed changes

paulhauner mentioned this pull request Nov 30, 2021

Kintsugi Diva comments #2836

Merged

michaelsproul reviewed Nov 30, 2021

View reviewed changes

common/slot_clock/src/lib.rs Outdated Show resolved Hide resolved

This was referenced Nov 30, 2021

[Merged by Bors] - Retrospective invalidation of exec. payloads for opt. sync #2837

Closed

Kintsugi slot clock fixes #2842

Merged

ethDreamer and others added 5 commits December 2, 2021 14:26

Fix consensus, SSZ, tree hash & run merge EF tests (#2622)

cce855f

* Update to v1.1.0-beta.4 (squash of #2548) * SSZ, cached tree hash, EF tests

Fix clippy lints on merge-f2f (#2626)

c10e8ce

* Remove unchecked arith from ssz_derive * Address clippy lints in block_verfication * Use safe math for is_valid_gas_limit

Update merge consensus to v1.1.0-beta.5 (#2630)

10b263f

Handle merge fork in web3signer (#2631)

18eee2d

pawanjay176 and others added 24 commits December 2, 2021 14:26

Fix db paths when datadir is relative (#2682)

aa1d57a

Accept TTD override as decimal (#2676)

6b4cc63

Kintsugi rebase patches (#2769)

86e0c56

* Freshen Cargo.lock * Fix gossip worker * Update map_fork_name_with

Move merge-f2f docker to kintsugi (#2774)

cee18ca

Fix arbitrary check kintsugi (#2777)

cdbe603

Fix Uint256 deserialization (#2786)

24966c0

* Change base_fee_per_gas to Uint256 * Add custom (de)serialization to ExecutionPayload * Fix errors * Add a quoted_u256 module * Remove unused function * lint * Add test * Remove extra line Co-authored-by: Paul Hauner <[email protected]>

Remove old uses of testnet

eb35c64

Fixes after rebasing Kintsugi onto unstable (#2799)

cbd2201

* Fix fork choice after rebase * Remove paulhauner warp dep * Fix fork choice test compile errors * Assume fork choice payloads are valid * Add comment * Ignore new tests * Fix error in test skipping

Skip memory intensive engine test (#2809)

cdfd130

* Allocate less memory (3GB) in engine tests * Run cargo format * Remove tx too large test Co-authored-by: Michael Sproul <[email protected]>

Removed PowBlock struct that never got used (#2813)

f674853

Bump crate versions (#2829)

82a8152

Cleanup Comments & Fix get_pow_block_hash_at_ttd() (#2835)

c2f2813

Kintsugi Diva comments (#2836)

ab86b42

* Remove TODOs * Fix typo

Support legacy data directories (#2846)

94385fe

Remove duplicate slot_clock method (#2842)

144978f

Restrict network limits based on merge fork epoch (#2839)

f3c237c

paulhauner force-pushed the kintsugi branch from 2d8650b to f3c237c Compare December 2, 2021 03:32

paulhauner merged commit f3c237c into unstable Dec 2, 2021

paulhauner deleted the kintsugi branch December 2, 2021 05:51

paulhauner restored the kintsugi branch December 2, 2021 05:52

AgeManning deleted the kintsugi branch January 20, 2022 21:39

Kintsugi 🍵 #2768

Kintsugi 🍵 #2768

Uh oh!

Conversation

paulhauner commented Nov 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

TODO

Uh oh!

realbigsean commented Nov 4, 2021

Uh oh!

paulhauner commented Nov 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

paulhauner commented Nov 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelsproul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

divagant-martian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulhauner commented Dec 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

paulhauner commented Nov 3, 2021 •

edited

Loading

paulhauner commented Nov 11, 2021 •

edited

Loading

paulhauner commented Nov 28, 2021 •

edited

Loading