[Merged by Bors] - Altair networking #2300
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pawanjay176
force-pushed
the
network-altair
branch
from
0afec84 to
1ad0c10
Compare
pawanjay176
force-pushed
the
network-altair
branch
3 times, most recently
from
fd7fadf to
6b06b22
Compare
pawanjay176
force-pushed
the
network-altair
branch
3 times, most recently
from
f54c9ec to
4573f9f
Compare
pawanjay176
force-pushed
the
network-altair
branch
2 times, most recently
from
6cef740 to
d251539
Compare
pawanjay176
force-pushed
the
network-altair
branch
from
d251539 to
4e83197
Compare
pawanjay176
force-pushed
the
network-altair
branch
from
4e83197 to
909529e
Compare
pawanjay176
force-pushed
the
network-altair
branch
2 times, most recently
from
2bffb56 to
bd27ae2
Compare
pawanjay176
force-pushed
the
network-altair
branch
from
bd27ae2 to
88e0227
Compare
* Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint
pawanjay176
force-pushed
the
network-altair
branch
from
87945c2 to
f37f267
Compare
* small refactors * Update attestation/deposit processing * Implement `upgrade_to_altair`, genesis init test * fix slashings and implement sync committee processing * refactor common epoch processing methods * delete unused imports * Fix typo in sync_committee_updates * Update to v1.1.0-alpha.3 * Fix remaining EF tests/handlers! * Fix beacon chain tests non-altair (#2305) * Use RefMut * Reset Cargo.lock to "stable" style * Tidy block tree hash * Temporarily hardcode base fork in op pool * Update block proposal & op pool * Fix bounds check on compute_sync_committee_indices * Tidy phase0 op pool * Remove global fork schedule (#2314) * Remove global fork schedule * Fix remaining EF tests for refactor * Get top-level compiling again * Bubble YAML config changes * Use superstruct partial getters This makes our Altair consensus logic compatible with future forks. * Lots of test refactors (#2315) * Fix beacon chain tests non-altair * fix op pool tests * fix merge * get block processing tests working * Refactor state transition vector tests, half are passing * cargo fmt * PR updates * Skip ahead to epoch 256 on beacon harness tests. Fix op pool tests. * clippy updates * More test cleanups and fixes * Use `upgrade_to_altair` in state transition * More test fixes * Begin adding pools related to sync committees * Add fork upgrade test, fix EF spec test * Sync committee pool refactors, attempt to get ssz tests working * Revert BitVector change * Update lcli and local testnet scripts * Fix `SyncCommitteeContribution` bit vector length, fix `SyncCommitteeSignature` field name * update docs and add tests for sync committee operation pool * More test updates, release-tests might pass now... * Nuke old store benchmark * Fix arbitrary instances * Participation bits fixes, clippy fixes * add BitVector intersection and union tests * Disable indexing in types & state processing * Delete tree hash and state proc benchmarks * Disable aggressive lints in tests * Appease udeps * Fix the RPC tests without hacks Thanks @pawanjay176 * Fix double update in github cargo fmt * Use published superstruct * Try splitting beacon_chain tests out of CI * Fix cargo-fmt --all * Abstract over fork in beacon_chain tests Also consolidate persistence_tests into store tests to reduce the # of integration test binaries slightly. * Fix clippy * Maintain backwards compatibility with std config * Sync committee signature verification work * Add an `Arc` to the `current_sync_committee` field, refactor `sync_committee_verification` code * Clean up imports * Move `sync_committee_base_epoch` to `Epoch` * Fmt and fix some lints * Add tests for new observation pools * Some op pool work * Provide clarity about BeaconState leaf count (#2339) * Remove sync committee indices cache We could add back a cache at some point, but it would likely look completely different due to having to compute the indices "backwards" from the public keys. Dealing with all the cases involving rebuilding the cache efficiently, or what to do if it isn't built will be quite involved so I think it's better we wait to see if we need it before optimising prematurely. * - Add beacon harness methods for sync signature create - separate sync signature `SubnetId` and `SelecitonProof` types from attestations - update op pool persistence to include sync aggregates * merge with `altair` * Add tests for decoding state/block with bad slot (#2341) * Tidy SyncCommittee types * More sync committee type cleanups/fallout * merge with `altair` * Some sync verification fixes and test progress * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Use graffiti len more consistently (#2348) * Replace a test deleted during Altair update (#2351) * Apply comment fixes from Paul Co-authored-by: Paul Hauner <[email protected]> * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Finish op pool sync committee logic, add a couple tests. * Avoid slicing in PartialBeaconState (#2361) * [Altair] Make anonymous var-length SSZ decoding explicit (#2362) * Add explicit function for anon var length items * Remove unused import * Update comment * Use macros for naive aggregation pool tests. * small updates * [Altair] Rewards tests (#2349) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * [Altair] Remove duplicate vars, panic on `u64::pow` overflow (#2350) * [Altair] Applying Base block to Altair chain (#2353) * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Fix test compile error * Fix test with inconsistent spec * Use fork_name_at_slot in genesis * [Altair] Add check for missed EF tests (#2346) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * Add file access logging * Tidy * Fix clippy lint * Fix typo * Tidy, remove commented-out test exceptions * Improve naming in python script Co-authored-by: Michael Sproul <[email protected]> * Fix beacon chain harness bug in `make_sync_contributions`. Key the `ObservedSyncAggregators` pool on slot + subcommittee index * Accidentally removed a couple constants while merging * fix a typo in a const, and a bunch of lints * Restore lost CI YAML * Address review comments * Update comment * Return an error if total bal is 0 (#2366) * Fix and test selection proof `verify` * Fix get_sync_aggregate * update op pool tests to correctly use `get_sync_aggregate` * Minor changes to justification code (#2367) * [Altair] Optimization for `get_unslashed_participating_indices` (#2369) * Lift calculation of unslashed indices * Use HashSet for unslashed participating indices * Make Operation Pool Compile Again (MOPCA) * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Add chainspec and genesis validators root to eth2_libp2p * Add altair gossip types * Subscribe to required gossip topics * Unsubscribe from pre fork topics * Compiles * Start adding sync committee attnets * Add Subnet enum to include SyncCommittee subnet_ids * update_enr_bitfield takes a Subnet instead of SubnetId * Fix network service * Add MetaData V2 * Don't read context bytes for MetaData V2 * Add sync committee topics to whitelist * Fix metadata encoding; fmt * Fix next_fork methods in chainspec * Minor fixes * Fix next_fork_update in network service; add next_unsubscribe delay * Accept peers which haven't updated next_fork_version/epoch * Add sync committee topic tests * Simplify required_gossip_digests; improve logging around fork boundary * Fix tests * Minor refactorings * Add sync contribution aggregation tests to the op pool and fix an op pool bug * Only advance state as necessary in sync committee verification tests. Fix compile error. * Delete `verify_sync_contributions.rs` and fix a comment. * Remove `Arc` from `current_sync_committee` * Add `max_capacity` to `ObservedAggregates` constructor * Update consensus code to v1.1.0-alpha.6 * Make the `SyncAggregateId` a fixed-length struct * Add metrics and update docs * Address review comments * Sync committee subnets * Ripple spec updates * Revert "Remove `Arc` from `current_sync_committee`" This reverts commit bf6324e. * fix some pool tests * Updates to some docs, general cleanup * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix more docs and metrics * Remove `is_valid` methods from `SignedAggregateAndProof` and `SignedContributionAndProof`. The logic is duplicated in the signature set verification methods. * remove outdated `FIXME` * Fix docs * Fix docs * Implement preset <> config distinction * Update confusing name `SyncCommitteeSubnetSize` -> `SyncSubcommitteeSize` * Updates for latest spec merge * update client for `lighthouse/spec` endpoint * Add back accidentally deleted import * Return ConfigAndPreset from VC API * Don't panic if there aren't any aggregators Rustfmt also returned with a vengeance to rearrange this block. * Check state root in transition tests * Fix clippy * Fix Pyrmont & Prater configs. Delete Toledo. * Add compatibility shim for /config/spec * Fix fork_digest in Status message * Revert "update client for `lighthouse/spec` endpoint" This reverts commit 7a9b2ac. # Conflicts: # validator_client/src/http_api/tests.rs * - Update `SYNC_COMMITTEE_SUBNET_COUNT` to alpha.6 - rename the `sync_aggregator_selection_data.rs` file - include sync committee ssz static tests in the `check_all_file_accessed.py` checks - Update `ObservedAggregates` consts so they are specific to the type of object being cached * - Update `NaiveAggregationPool`'s default capacity so it is specific to the type of object being inserted * fix `observed_aggregates` tests * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Use SyncSubnetId for sync committee subnets * Add worker tasks for sync committee gossip messages * Use correct type level constants * Decode altair gossip blocks * Add a SyncCommitteeSubscribe network event * PR updates * fix op pool tests * Minor cleanups * Calculate `SyncAggregate` on demand * Address review comments * Op pool migration + superstruct * Fix op pool test * Update to spec v1.1.0-alpha.7 * Fix sync committee subscriptions * Add sync committee subscription test * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * - Fix sync committee verification on slot prior to sync committee period boundary - Add `Arc` to `next_sync_committee` - Other PR updates * Refactor how we retrieve `PublicKey` for sync signature verification * Key `observed_contributors` on `SlotSubcommitteeIndex` * Update mainnet preset YAML * Gossipsub message_id_fn changes * Add tests for sync committee verification at slot before sync committee period boundary * fix op pool tests * - Make `SyncSubnetId` a required argument for `verify_sync_signature_for_gossip` - Drop locks as soon as possible during metrics scrape * fix sync committee verification test macro * Fix beacon_chain test * rust 1.53.0 updates * add recursion limit to `simulator` crate * cargo fmt * `SyncCommitteSignature` -> `SyncCommitteeMessage` * Updates related to merge with `unstable` * More updates related to `SyncCommitteMessage` refactor * Couple more `unstable` merge fixes * Fix `SyncAggregate` field name * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Change libp2p dependency for message_id_fn * appease clippy * Add sync committee related metrics * Expand and refactor fork functions * Simplify compute_subnets_for_sync_committee * Fix subnet service test * Add v2 protocols for OutboundRequests missed in the merge * Handle sync committee gossip errors * Remove delay for updating fork * fix the expected error in the sync contribution verification test at the sync period boundary * Updates related to merge with `unstable` * Couple more `unstable` merge fixes * Split rpc encode/decode into smaller functions * Import some relevant funcs from the VC Based on "Expand and refactor fork functions" * Fix issues with skip slots at fork boundaries * Fix clippy again * alpha.8 updates * Pop sync committee tasks from beacon processor * Fix some gossip scoring severity issue * Fix bug in peer_manager; Improve logging around subnet discoveries Co-authored-by: realbigsean <[email protected]> Co-authored-by: Michael Sproul <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: Paul Hauner <[email protected]>
pawanjay176
force-pushed
the
network-altair
branch
from
4972204 to
f54a086
Compare
pawanjay176
force-pushed
the
network-altair
branch
from
f54a086 to
9f29788
Compare
pawanjay176
force-pushed
the
network-altair
branch
from
525d01f to
6c76853
Compare
pawanjay176
commented
Jul 23, 2021
Member
Author
pawanjay176
left a comment
pawanjay176
left a comment
There was a problem hiding this comment.
@paulhauner @michaelsproul any feedback on the constants here?
pawanjay176
added
ready-for-review
Member
michaelsproul
left a comment
michaelsproul
left a comment
There was a problem hiding this comment.
I've reviewed about 38/51 files, and will come back for the rest next week.
The Rust 1.54.0 changes in unstable should also be brought in so we can ripple them up into altair-vc
michaelsproul
requested changes
Aug 3, 2021
Member
michaelsproul
left a comment
michaelsproul
left a comment
There was a problem hiding this comment.
I've finished reviewing the remaining files, and found just a few minor things.
I think we should proceed with merging and widespread testing via a release candidate 🎉
michaelsproul
added
waiting-on-author
michaelsproul
added
ready-for-merge
Member
|
This is ready to merge! Let's do it!! 🎉 bors r+ |
paulhauner
reviewed
Aug 4, 2021
| #version = "0.39.1" | ||
| #default-features = false | ||
| # TODO: Update once https://github.com/libp2p/rust-libp2p/pull/2103 and | ||
| # https://github.com/libp2p/rust-libp2p/pull/2137 are merged upstream. |
Member
There was a problem hiding this comment.
FYI, these have both been merged.
Member
There was a problem hiding this comment.
Yeah, I figured we handle the update as part of #2493
bors bot
pushed a commit
that referenced
this pull request
Aug 4, 2021
## Issue Addressed Resolves #2278 ## Proposed Changes Implements the networking components for the Altair hard fork https://github.com/ethereum/eth2.0-specs/blob/dev/specs/altair/p2p-interface.md ## Additional Info This PR acts as the base branch for networking changes and tracks #2279 . Changes to gossip, rpc and discovery can be separate PRs to be merged here for ease of review. Co-authored-by: realbigsean <[email protected]>
|
Pull request successfully merged into unstable. Build succeeded: |
Member
|
Woo hoo! Well done @pawanjay176 for months of hard work on this 🎉 ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue Addressed
Resolves #2278
Proposed Changes
Implements the networking components for the Altair hard fork https://github.com/ethereum/eth2.0-specs/blob/dev/specs/altair/p2p-interface.md
Additional Info
This PR acts as the base branch for networking changes and tracks #2279 . Changes to gossip, rpc and discovery can be separate PRs to be merged here for ease of review.