core/state/snapshot: increase batch size during diffToDisk #309
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
darioush
approved these changes
Aug 25, 2023
joshua-kim
added a commit
that referenced
this pull request
Aug 29, 2023
commit 4ef26a0 Author: Darioush Jalali <[email protected]> Date: Mon Aug 28 06:38:44 2023 -0700 bump go version in dockerfile (#311) commit 505a6ab Author: aaronbuchwald <[email protected]> Date: Fri Aug 25 15:43:09 2023 -0400 core/state/snapshot: increase batch size during diffToDisk (#309) commit cca3e00 Author: marun <[email protected]> Date: Fri Aug 25 09:44:42 2023 -0700 Bump golang version to 1.19.12 for CI (#306) * Bump golang version to 1.19.12 for CI Also bump targeted version of avalanchego to the v1.10.8 (which also uses golang 1.19.12). * fixup: Update go-version specification for consistency with other repos commit 42ba830 Author: Darioush Jalali <[email protected]> Date: Fri Aug 25 09:44:12 2023 -0700 add upstream test: TestClientBatchRequest_len (#307) commit 08e2b6e Author: Stephen Buttolph <[email protected]> Date: Wed Aug 23 18:09:51 2023 -0400 Update avalanchego dependency to v1.10.9-rc.4 (#308) commit 5cd24c3 Author: Darioush Jalali <[email protected]> Date: Wed Aug 23 08:48:01 2023 -0700 node/config.go: remove unused options (#304) * node/config.go: remove unused graphql options * remove more unused config commit 8ea8b18 Author: aaronbuchwald <[email protected]> Date: Tue Aug 15 11:18:29 2023 -0400 Add security file based on AvalancheGo (#303) commit 00214b1 Author: Stephen Buttolph <[email protected]> Date: Mon Aug 14 14:58:07 2023 -0400 Update AVAX client implementation and interface to align types (#301) commit 5666553 Author: Ceyhun Onur <[email protected]> Date: Mon Aug 14 21:38:04 2023 +0300 renamed flags (#298) * renamed flags * rename admin api flags * add deprecation commit e7645fa Author: Ikko Eltociear Ashimine <[email protected]> Date: Thu Aug 10 01:21:44 2023 +0900 Fix typo in sync/README.md (#296)
joshua-kim
added a commit
that referenced
this pull request
Aug 30, 2023
commit 4ef26a0 Author: Darioush Jalali <[email protected]> Date: Mon Aug 28 06:38:44 2023 -0700 bump go version in dockerfile (#311) commit 505a6ab Author: aaronbuchwald <[email protected]> Date: Fri Aug 25 15:43:09 2023 -0400 core/state/snapshot: increase batch size during diffToDisk (#309) commit cca3e00 Author: marun <[email protected]> Date: Fri Aug 25 09:44:42 2023 -0700 Bump golang version to 1.19.12 for CI (#306) * Bump golang version to 1.19.12 for CI Also bump targeted version of avalanchego to the v1.10.8 (which also uses golang 1.19.12). * fixup: Update go-version specification for consistency with other repos commit 42ba830 Author: Darioush Jalali <[email protected]> Date: Fri Aug 25 09:44:12 2023 -0700 add upstream test: TestClientBatchRequest_len (#307) commit 08e2b6e Author: Stephen Buttolph <[email protected]> Date: Wed Aug 23 18:09:51 2023 -0400 Update avalanchego dependency to v1.10.9-rc.4 (#308) commit 5cd24c3 Author: Darioush Jalali <[email protected]> Date: Wed Aug 23 08:48:01 2023 -0700 node/config.go: remove unused options (#304) * node/config.go: remove unused graphql options * remove more unused config commit 8ea8b18 Author: aaronbuchwald <[email protected]> Date: Tue Aug 15 11:18:29 2023 -0400 Add security file based on AvalancheGo (#303) commit 00214b1 Author: Stephen Buttolph <[email protected]> Date: Mon Aug 14 14:58:07 2023 -0400 Update AVAX client implementation and interface to align types (#301) commit 5666553 Author: Ceyhun Onur <[email protected]> Date: Mon Aug 14 21:38:04 2023 +0300 renamed flags (#298) * renamed flags * rename admin api flags * add deprecation commit e7645fa Author: Ikko Eltociear Ashimine <[email protected]> Date: Thu Aug 10 01:21:44 2023 +0900 Fix typo in sync/README.md (#296)
joshua-kim
added a commit
to joshua-kim/coreth
that referenced
this pull request
Aug 30, 2023
commit 3f5dc8a Author: marun <[email protected]> Date: Tue Aug 29 10:17:05 2023 -0700 e2e: Add avalanchego e2e job (ava-labs#305) * e2e: Add avalanchego e2e job * fixup: Use currently supported golang version * fixup: Enable configurable avalanchego clone path commit 4ef26a0 Author: Darioush Jalali <[email protected]> Date: Mon Aug 28 06:38:44 2023 -0700 bump go version in dockerfile (ava-labs#311) commit 505a6ab Author: aaronbuchwald <[email protected]> Date: Fri Aug 25 15:43:09 2023 -0400 core/state/snapshot: increase batch size during diffToDisk (ava-labs#309) commit cca3e00 Author: marun <[email protected]> Date: Fri Aug 25 09:44:42 2023 -0700 Bump golang version to 1.19.12 for CI (ava-labs#306) * Bump golang version to 1.19.12 for CI Also bump targeted version of avalanchego to the v1.10.8 (which also uses golang 1.19.12). * fixup: Update go-version specification for consistency with other repos commit 42ba830 Author: Darioush Jalali <[email protected]> Date: Fri Aug 25 09:44:12 2023 -0700 add upstream test: TestClientBatchRequest_len (ava-labs#307) commit 08e2b6e Author: Stephen Buttolph <[email protected]> Date: Wed Aug 23 18:09:51 2023 -0400 Update avalanchego dependency to v1.10.9-rc.4 (ava-labs#308) commit 5cd24c3 Author: Darioush Jalali <[email protected]> Date: Wed Aug 23 08:48:01 2023 -0700 node/config.go: remove unused options (ava-labs#304) * node/config.go: remove unused graphql options * remove more unused config commit 8ea8b18 Author: aaronbuchwald <[email protected]> Date: Tue Aug 15 11:18:29 2023 -0400 Add security file based on AvalancheGo (ava-labs#303) commit 00214b1 Author: Stephen Buttolph <[email protected]> Date: Mon Aug 14 14:58:07 2023 -0400 Update AVAX client implementation and interface to align types (ava-labs#301) commit 5666553 Author: Ceyhun Onur <[email protected]> Date: Mon Aug 14 21:38:04 2023 +0300 renamed flags (ava-labs#298) * renamed flags * rename admin api flags * add deprecation commit e7645fa Author: Ikko Eltociear Ashimine <[email protected]> Date: Thu Aug 10 01:21:44 2023 +0900 Fix typo in sync/README.md (ava-labs#296)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR migrates ethereum/go-ethereum#27977.
This increases the cap on the size of a batch when flattening a snapshot diff layer to disk. This is a non-atomic operation to ensure that flattening does not create such a large batch that it causes an OOM.
However, breaking this up may result in invalidating the snapshot since you may apply only part of the change, which results in needing to re-generate the snapshot.
Increasing the capped size to 64MB provides a reasonable cap to avoid an OOM and drastically increases the size of a diff that is needed to potentially corrupt a snapshot.