Zktrie may lead to a race condition and crash geth when preimage is enabled

[noel2004]

System information

Geth version: pre-alpha 2.2
OS & Version: Linux/OSX

Expected behaviour

Run l2geth with --gcmode=archive, which enforce preimage being enabled and start syncing to main network. App would crash while syncing at random time.

Actual behaviour

No crashing and successfully syncing

Steps to reproduce the behaviour

Following the instruction in running your own l2geth node, then wait for syncing finishing

Also notice that some 'addLeaf:GetNode err key not found in ZkTrieImpl' debug messages are being printed.

Backtrace

fatal error: concurrent map iteration and map write

goroutine 142 [running]:
runtime.throw({0x152168f?, 0x30?})
	runtime/panic.go:992 +0x71 fp=0xc001891420 sp=0xc0018913f0 pc=0x451c71
runtime.mapiternext(0xc01885ecd8?)
	runtime/map.go:871 +0x4eb fp=0xc001891490 sp=0xc001891420 pc=0x42b70b
github.com/scroll-tech/go-ethereum/core/rawdb.WritePreimages({0x7fdf90084058, 0xc01885ecd8}, 0xc017df59e0)
	github.com/scroll-tech/go-ethereum/core/rawdb/accessors_state.go:33 +0x99 fp=0xc0018915c8 sp=0xc001891490 pc=0x7b1ab9
github.com/scroll-tech/go-ethereum/trie.(*Database).Commit(0xc000447180, {0x1a, 0x4a, 0xef, 0xe7, 0x12, 0xea, 0xa, 0xcd, 0xec, ...}, ...)
	github.com/scroll-tech/go-ethereum/trie/database.go:730 +0x292 fp=0xc0018917f8 sp=0xc0018915c8 pc=0x7ef672
github.com/scroll-tech/go-ethereum/core.(*BlockChain).writeBlockWithState(0xc0004d0000, 0xc0068aa750, {0xc0051ad0c0, 0x1, 0x1}, {0xc0181d9ed8, 0x3, 0x3}, 0xc000d7ed00, 0x0)
	github.com/scroll-tech/go-ethereum/core/blockchain.go:1241 +0xd7c fp=0xc001891bb0 sp=0xc0018917f8 pc=0x9db1dc
github.com/scroll-tech/go-ethereum/core.(*BlockChain).insertChain(0xc0004d0000, {0xc01487c000?, 0x800, 0x800}, 0x1)
	github.com/scroll-tech/go-ethereum/core/blockchain.go:1658 +0x2037 fp=0xc0018938c8 sp=0xc001891bb0 pc=0x9debb7
github.com/scroll-tech/go-ethereum/core.(*BlockChain).InsertChain(0xc0004d0000, {0xc01487c000?, 0x800, 0x800})
	github.com/scroll-tech/go-ethereum/core/blockchain.go:1393 +0xb4e fp=0xc001893c10 sp=0xc0018938c8 pc=0x9dc66e
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).importBlockResults(0xc000c3cc40, {0xc014878000, 0x800, 0xc13940?})
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1723 +0x45d fp=0xc001893f28 sp=0xc001893c10 pc=0xaf71dd
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).processFullSyncContent(0xc000c3cc40)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1697 +0x85 fp=0xc001893f70 sp=0xc001893f28 pc=0xaf6d45
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).processFullSyncContent-fm()
	<autogenerated>:1 +0x26 fp=0xc001893f88 sp=0xc001893f70 pc=0xb0b9c6
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x70 fp=0xc001893fe0 sp=0xc001893f88 pc=0xaed550
runtime.goexit()
	runtime/asm_amd64.s:1571 +0x1 fp=0xc001893fe8 sp=0xc001893fe0 pc=0x484d01
created by github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x5f

goroutine 1 [chan receive]:
github.com/scroll-tech/go-ethereum/node.(*Node).Wait(...)
	github.com/scroll-tech/go-ethereum/node/node.go:423
main.geth(0xc00043a9a0)
	github.com/scroll-tech/go-ethereum/cmd/geth/main.go:323 +0x10a
gopkg.in/urfave/cli%2ev1.HandleAction({0x12c04e0?, 0x1755f58?}, 0xc000284000?)
	gopkg.in/urfave/cli.v1@v1.20.0/app.go:490 +0x5a
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc0003c16c0, {0xc00003c140, 0x13, 0x14})
	gopkg.in/urfave/cli.v1@v1.20.0/app.go:264 +0x71c
main.main()
	github.com/scroll-tech/go-ethereum/cmd/geth/main.go:262 +0x35

goroutine 18 [chan receive]:
github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc00007e840, 0x0?)
	github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:36 +0x45
created by github.com/rjeczalik/notify.newNonrecursiveTree
	github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:29 +0x16a

goroutine 19 [chan receive]:
github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc00007e840, 0xc00007e7e0)
	github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:81 +0x5a
created by github.com/rjeczalik/notify.newNonrecursiveTree
	github.com/rjeczalik/notify@v0.9.1/tree_nonrecursive.go:30 +0x1be

goroutine 20 [chan receive]:
github.com/scroll-tech/go-ethereum/core.(*txSenderCacher).cache(0x0?)
	github.com/scroll-tech/go-ethereum/core/tx_cacher.go:63 +0x3b
created by github.com/scroll-tech/go-ethereum/core.newTxSenderCacher
	github.com/scroll-tech/go-ethereum/core/tx_cacher.go:55 +0x75

goroutine 21 [chan receive]:
github.com/scroll-tech/go-ethereum/core.(*txSenderCacher).cache(0x0?)
	github.com/scroll-tech/go-ethereum/core/tx_cacher.go:63 +0x3b
created by github.com/scroll-tech/go-ethereum/core.newTxSenderCacher
	github.com/scroll-tech/go-ethereum/core/tx_cacher.go:55 +0x75

goroutine 22 [chan receive]:
github.com/scroll-tech/go-ethereum/metrics.(*meterArbiter).tick(0x23fa4a0)
	github.com/scroll-tech/go-ethereum/metrics/meter.go:290 +0x65
created by github.com/scroll-tech/go-ethereum/metrics.NewMeterForced
	github.com/scroll-tech/go-ethereum/metrics/meter.go:71 +0xc5

goroutine 23 [select]:
github.com/scroll-tech/go-ethereum/consensus/ethash.(*remoteSealer).loop(0xc00044a4b0)
	github.com/scroll-tech/go-ethereum/consensus/ethash/sealer.go:279 +0x205
created by github.com/scroll-tech/go-ethereum/consensus/ethash.startRemoteSealer
	github.com/scroll-tech/go-ethereum/consensus/ethash/sealer.go:263 +0x319

goroutine 12 [syscall]:
syscall.Syscall6(0xe8, 0x8, 0xc000063f84, 0x1, 0xffffffffffffffff, 0x0, 0x0)
	syscall/asm_linux_amd64.s:43 +0x5
golang.org/x/sys/unix.EpollWait(0x18db168?, {0xc000063f84?, 0x421b39?, 0x7?}, 0x8?)
	golang.org/x/sys@v0.0.0-20220722155257-8c9f86f7a55f/unix/zsyscall_linux_amd64.go:56 +0x58
github.com/rjeczalik/notify.(*inotify).loop(0xc0004fe000, 0xc0002ba900?)
	github.com/rjeczalik/notify@v0.9.1/watcher_inotify.go:194 +0x5a
created by github.com/rjeczalik/notify.(*inotify).lazyinit
	github.com/rjeczalik/notify@v0.9.1/watcher_inotify.go:134 +0x118

goroutine 13 [chan receive]:
github.com/rjeczalik/notify.(*inotify).send(0xc0004fe000, 0xc00040c240?)
	github.com/rjeczalik/notify@v0.9.1/watcher_inotify.go:254 +0x45
created by github.com/rjeczalik/notify.(*inotify).lazyinit
	github.com/rjeczalik/notify@v0.9.1/watcher_inotify.go:137 +0x1c8

goroutine 26 [select]:
github.com/scroll-tech/go-ethereum/accounts.(*Manager).update(0xc0004468c0)
	github.com/scroll-tech/go-ethereum/accounts/manager.go:134 +0x14d
created by github.com/scroll-tech/go-ethereum/accounts.NewManager
	github.com/scroll-tech/go-ethereum/accounts/manager.go:94 +0x2b5

goroutine 27 [select]:
github.com/scroll-tech/go-ethereum/accounts/keystore.(*watcher).loop(0xc0004322a0)
	github.com/scroll-tech/go-ethereum/accounts/keystore/watch.go:96 +0x43b
created by github.com/scroll-tech/go-ethereum/accounts/keystore.(*watcher).start
	github.com/scroll-tech/go-ethereum/accounts/keystore/watch.go:54 +0x76

goroutine 28 [select]:
github.com/scroll-tech/go-ethereum/accounts/keystore.(*KeyStore).updater(0xc00044a3c0)
	github.com/scroll-tech/go-ethereum/accounts/keystore/keystore.go:209 +0x85
created by github.com/scroll-tech/go-ethereum/accounts/keystore.(*KeyStore).Subscribe
	github.com/scroll-tech/go-ethereum/accounts/keystore/keystore.go:196 +0x110

goroutine 14 [chan receive]:
github.com/rjeczalik/notify.(*inotify).send(0xc0004fe000, 0xc0004322a0?)
	github.com/rjeczalik/notify@v0.9.1/watcher_inotify.go:254 +0x45
created by github.com/rjeczalik/notify.(*inotify).lazyinit
	github.com/rjeczalik/notify@v0.9.1/watcher_inotify.go:137 +0x1c8

goroutine 29 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00044a5a0)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x59b
created by github.com/syndtr/goleveldb/leveldb.newSession
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2d9

goroutine 30 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc000171340)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x158
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ea

goroutine 31 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc000171340)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xa8
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x52a

goroutine 32 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc000171340)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x657
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x598

goroutine 33 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc000171340)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x113
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5d6

goroutine 34 [select]:
github.com/scroll-tech/go-ethereum/ethdb/leveldb.(*Database).meter(0xc0000d4700, 0x0?)
	github.com/scroll-tech/go-ethereum/ethdb/leveldb/leveldb.go:433 +0x178a
created by github.com/scroll-tech/go-ethereum/ethdb/leveldb.NewCustom
	github.com/scroll-tech/go-ethereum/ethdb/leveldb/leveldb.go:149 +0x825

goroutine 35 [select]:
github.com/scroll-tech/go-ethereum/core/rawdb.(*freezer).freeze(0xc000472380, {0x18f1150?, 0xc0000d4700})
	github.com/scroll-tech/go-ethereum/core/rawdb/freezer.go:355 +0x19e
github.com/scroll-tech/go-ethereum/core/rawdb.NewDatabaseWithFreezer.func1()
	github.com/scroll-tech/go-ethereum/core/rawdb/database.go:222 +0x2a
created by github.com/scroll-tech/go-ethereum/core/rawdb.NewDatabaseWithFreezer
	github.com/scroll-tech/go-ethereum/core/rawdb/database.go:221 +0x870

goroutine 36 [select]:
github.com/scroll-tech/go-ethereum/core.(*ChainIndexer).updateLoop(0xc0004ef2c0)
	github.com/scroll-tech/go-ethereum/core/chain_indexer.go:311 +0xc7
created by github.com/scroll-tech/go-ethereum/core.NewChainIndexer
	github.com/scroll-tech/go-ethereum/core/chain_indexer.go:119 +0x31b

goroutine 37 [select]:
github.com/scroll-tech/go-ethereum/core.(*BlockChain).futureBlocksLoop(0xc0004d0000)
	github.com/scroll-tech/go-ethereum/core/blockchain.go:2020 +0x105
created by github.com/scroll-tech/go-ethereum/core.NewBlockChain
	github.com/scroll-tech/go-ethereum/core/blockchain.go:395 +0x1d6a

goroutine 38 [select]:
github.com/scroll-tech/go-ethereum/core.(*BlockChain).maintainTxIndex(0xc0004d0000, 0x0?)
	github.com/scroll-tech/go-ethereum/core/blockchain.go:2140 +0x233
created by github.com/scroll-tech/go-ethereum/core.NewBlockChain
	github.com/scroll-tech/go-ethereum/core/blockchain.go:402 +0x1de5

goroutine 39 [select]:
github.com/scroll-tech/go-ethereum/trie.(*Database).SaveCachePeriodically(0x0?, {0xc000474b40, 0x20}, 0x0?, 0xc000431080)
	github.com/scroll-tech/go-ethereum/trie/database.go:907 +0xf2
github.com/scroll-tech/go-ethereum/core.NewBlockChain.func1()
	github.com/scroll-tech/go-ethereum/core/blockchain.go:415 +0x7d
created by github.com/scroll-tech/go-ethereum/core.NewBlockChain
	github.com/scroll-tech/go-ethereum/core/blockchain.go:413 +0x1f93

goroutine 40 [select]:
github.com/scroll-tech/go-ethereum/core.(*ChainIndexer).eventLoop(0xc0004ef2c0, 0xc000422900, 0xc0001e7b00, {0x18e6bd8?, 0xc000c3e060?})
	github.com/scroll-tech/go-ethereum/core/chain_indexer.go:211 +0x25e
created by github.com/scroll-tech/go-ethereum/core.(*ChainIndexer).Start
	github.com/scroll-tech/go-ethereum/core/chain_indexer.go:152 +0x10a

goroutine 41 [select]:
github.com/scroll-tech/go-ethereum/core.(*TxPool).scheduleReorgLoop(0xc00044e400)
	github.com/scroll-tech/go-ethereum/core/tx_pool.go:1114 +0x2f4
created by github.com/scroll-tech/go-ethereum/core.NewTxPool
	github.com/scroll-tech/go-ethereum/core/tx_pool.go:328 +0x7f0

goroutine 42 [select]:
github.com/scroll-tech/go-ethereum/core.(*TxPool).loop(0xc00044e400)
	github.com/scroll-tech/go-ethereum/core/tx_pool.go:372 +0x2d7
created by github.com/scroll-tech/go-ethereum/core.NewTxPool
	github.com/scroll-tech/go-ethereum/core/tx_pool.go:345 +0xaac

goroutine 43 [select]:
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).stateFetcher(0xc000c3cc40)
	github.com/scroll-tech/go-ethereum/eth/downloader/statesync.go:84 +0x91
created by github.com/scroll-tech/go-ethereum/eth/downloader.New
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:236 +0x4fa

goroutine 44 [select]:
github.com/scroll-tech/go-ethereum/miner.(*worker).mainLoop(0xc000b31680)
	github.com/scroll-tech/go-ethereum/miner/worker.go:463 +0x439
created by github.com/scroll-tech/go-ethereum/miner.newWorker
	github.com/scroll-tech/go-ethereum/miner/worker.go:232 +0x725

goroutine 45 [select]:
github.com/scroll-tech/go-ethereum/miner.(*worker).newWorkLoop(0xc000b31680, 0xb2d05e00)
	github.com/scroll-tech/go-ethereum/miner/worker.go:391 +0x291
created by github.com/scroll-tech/go-ethereum/miner.newWorker
	github.com/scroll-tech/go-ethereum/miner/worker.go:233 +0x76b

goroutine 46 [select]:
github.com/scroll-tech/go-ethereum/miner.(*worker).resultLoop(0xc000b31680)
	github.com/scroll-tech/go-ethereum/miner/worker.go:614 +0xfd
created by github.com/scroll-tech/go-ethereum/miner.newWorker
	github.com/scroll-tech/go-ethereum/miner/worker.go:234 +0x7aa

goroutine 47 [select]:
github.com/scroll-tech/go-ethereum/miner.(*worker).taskLoop(0xc000b31680)
	github.com/scroll-tech/go-ethereum/miner/worker.go:575 +0x112
created by github.com/scroll-tech/go-ethereum/miner.newWorker
	github.com/scroll-tech/go-ethereum/miner/worker.go:235 +0x7ea

goroutine 48 [select]:
github.com/scroll-tech/go-ethereum/miner.(*Miner).update(0xc0003856c0)
	github.com/scroll-tech/go-ethereum/miner/miner.go:104 +0x21c
created by github.com/scroll-tech/go-ethereum/miner.New
	github.com/scroll-tech/go-ethereum/miner/miner.go:82 +0x238

goroutine 49 [chan receive]:
github.com/scroll-tech/go-ethereum/eth/gasprice.NewOracle.func1()
	github.com/scroll-tech/go-ethereum/eth/gasprice/gasprice.go:123 +0x8d
created by github.com/scroll-tech/go-ethereum/eth/gasprice.NewOracle
	github.com/scroll-tech/go-ethereum/eth/gasprice/gasprice.go:121 +0x96d

goroutine 50 [select]:
github.com/scroll-tech/go-ethereum/eth/downloader.(*PublicDownloaderAPI).eventLoop(0xc000bd5320)
	github.com/scroll-tech/go-ethereum/eth/downloader/api.go:63 +0x1f3
created by github.com/scroll-tech/go-ethereum/eth/downloader.NewPublicDownloaderAPI
	github.com/scroll-tech/go-ethereum/eth/downloader/api.go:49 +0x105

goroutine 51 [select]:
github.com/scroll-tech/go-ethereum/eth/filters.(*EventSystem).eventLoop(0xc00043fa20)
	github.com/scroll-tech/go-ethereum/eth/filters/filter_system.go:459 +0x40c
created by github.com/scroll-tech/go-ethereum/eth/filters.NewEventSystem
	github.com/scroll-tech/go-ethereum/eth/filters/filter_system.go:138 +0x36a

goroutine 52 [chan receive]:
github.com/scroll-tech/go-ethereum/eth/filters.(*PublicFilterAPI).timeoutLoop(0xc000b32500, 0x0?)
	github.com/scroll-tech/go-ethereum/eth/filters/api.go:82 +0x9b
created by github.com/scroll-tech/go-ethereum/eth/filters.NewPublicFilterAPI
	github.com/scroll-tech/go-ethereum/eth/filters/api.go:70 +0x149

goroutine 53 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc000d40000)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session_util.go:189 +0x59b
created by github.com/syndtr/goleveldb/leveldb.newSession
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/session.go:93 +0x2d9

goroutine 54 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc000c3d180)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:91 +0x158
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:148 +0x4ea

goroutine 55 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc000c3d180)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_state.go:101 +0xa8
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:149 +0x52a

goroutine 56 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc000c3d180)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:836 +0x657
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:155 +0x598

goroutine 57 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc000c3d180)
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db_compaction.go:773 +0x113
created by github.com/syndtr/goleveldb/leveldb.openDB
	github.com/syndtr/goleveldb@v1.0.1-0.20210819022825-2ae1ddf74ef7/leveldb/db.go:156 +0x5d6

goroutine 79 [chan receive]:
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync(0xc000c3cc40, {0xc000cbeb00, 0x5, 0x4?})
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:584 +0x148
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc000c3cc40, 0xc0016361a0, {0x42, 0x65, 0xa2, 0xbb, 0x7f, 0x41, 0x96, 0x9c, ...}, ...)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:563 +0xca5
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).synchronise(0x2411380?, {0xc0015bed00, 0x40}, {0x42, 0x65, 0xa2, 0xbb, 0x7f, 0x41, 0x96, ...}, ...)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:431 +0x498
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc000c3cc40, {0xc0015bed00, 0x40}, {0x42, 0x65, 0xa2, 0xbb, 0x7f, 0x41, 0x96, ...}, ...)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:326 +0x5c
github.com/scroll-tech/go-ethereum/eth.(*handler).doSync(0xc0000d5400, 0xc000cbe200)
	github.com/scroll-tech/go-ethereum/eth/sync.go:230 +0x213
github.com/scroll-tech/go-ethereum/eth.(*chainSyncer).startSync.func1()
	github.com/scroll-tech/go-ethereum/eth/sync.go:206 +0x29
created by github.com/scroll-tech/go-ethereum/eth.(*chainSyncer).startSync
	github.com/scroll-tech/go-ethereum/eth/sync.go:206 +0xaa

goroutine 138 [select]:
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).fetchHeaders(0xc000c3cc40, 0xc0016361a0, 0x2552)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1188 +0xca9
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func3()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:549 +0x29
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x70
created by github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x5f

goroutine 59 [select]:
github.com/scroll-tech/go-ethereum/p2p/nat.Map({0x18ec6a8?, 0xc000470e00}, 0xc000c7c9c0, {0x14ed6e9, 0x3}, 0x765f, 0x765f, {0x14f7894, 0xc})
	github.com/scroll-tech/go-ethereum/p2p/nat/nat.go:114 +0x3f8
github.com/scroll-tech/go-ethereum/p2p.(*Server).setupListening.func1()
	github.com/scroll-tech/go-ethereum/p2p/server.go:673 +0x5e
created by github.com/scroll-tech/go-ethereum/p2p.(*Server).setupListening
	github.com/scroll-tech/go-ethereum/p2p/server.go:672 +0x1b2

goroutine 60 [IO wait]:
internal/poll.runtime_pollWait(0x7fdf901ca958, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000473500?, 0xe0?, 0x0)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000473500)
	internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc000473500)
	net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000c3e828)
	net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc000c3e828)
	net/tcpsock.go:288 +0x3d
github.com/scroll-tech/go-ethereum/p2p.(*Server).listenLoop(0xc00028d180)
	github.com/scroll-tech/go-ethereum/p2p/server.go:862 +0x627
created by github.com/scroll-tech/go-ethereum/p2p.(*Server).setupListening
	github.com/scroll-tech/go-ethereum/p2p/server.go:680 +0x20a

goroutine 84 [IO wait]:
internal/poll.runtime_pollWait(0x7fdf901ca4a8, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0015e6600?, 0x47f75b3f48d57cca?, 0x0)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0015e6600)
	internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0015e6600)
	net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0xc000d246e8?)
	net/unixsock_posix.go:166 +0x1c
net.(*UnixListener).Accept(0xc0015d4f90)
	net/unixsock.go:260 +0x3d
github.com/scroll-tech/go-ethereum/rpc.(*Server).ServeListener(0xc000d55da0, {0x18eaa90, 0xc0015d4f90})
	github.com/scroll-tech/go-ethereum/rpc/ipc.go:30 +0x5b
created by github.com/scroll-tech/go-ethereum/rpc.StartIPCEndpoint
	github.com/scroll-tech/go-ethereum/rpc/endpoints.go:50 +0x20c

goroutine 83 [select]:
github.com/scroll-tech/go-ethereum/p2p.(*Server).run(0xc00028d180)
	github.com/scroll-tech/go-ethereum/p2p/server.go:714 +0x4cc
created by github.com/scroll-tech/go-ethereum/p2p.(*Server).Start
	github.com/scroll-tech/go-ethereum/p2p/server.go:487 +0x46a

goroutine 63 [chan receive]:
github.com/scroll-tech/go-ethereum/p2p/enode.(*FairMix).nextFromAny(...)
	github.com/scroll-tech/go-ethereum/p2p/enode/iter.go:241
github.com/scroll-tech/go-ethereum/p2p/enode.(*FairMix).Next(0xc0001ab980)
	github.com/scroll-tech/go-ethereum/p2p/enode/iter.go:215 +0x2c5
github.com/scroll-tech/go-ethereum/p2p.(*dialScheduler).readNodes(0xc00043adc0, {0x18ea580, 0xc0001ab980})
	github.com/scroll-tech/go-ethereum/p2p/dial.go:322 +0x9f
created by github.com/scroll-tech/go-ethereum/p2p.newDialScheduler
	github.com/scroll-tech/go-ethereum/p2p/dial.go:180 +0x425

goroutine 64 [select]:
github.com/scroll-tech/go-ethereum/p2p.(*dialScheduler).loop(0xc00043adc0, {0x18ea580, 0xc0001ab980})
	github.com/scroll-tech/go-ethereum/p2p/dial.go:243 +0x2ba
created by github.com/scroll-tech/go-ethereum/p2p.newDialScheduler
	github.com/scroll-tech/go-ethereum/p2p/dial.go:181 +0x49e

goroutine 66 [IO wait]:
internal/poll.runtime_pollWait(0x7fdf901ca868, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0002e4c00?, 0xc000ca02c0?, 0x0)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).ReadFromInet4(0xc0002e4c00, {0xc000ca02c0, 0x10, 0x10}, 0x4ec045?)
	internal/poll/fd_unix.go:250 +0x1e5
net.(*netFD).readFromInet4(0xc0002e4c00, {0xc000ca02c0?, 0xc0002e4c00?, 0xc000ca024c?}, 0x7fdf901ca800?)
	net/fd_posix.go:66 +0x29
net.(*UDPConn).readFrom(0xc000d43db8?, {0xc000ca02c0?, 0xc0002e4c00?, 0xc000ca024c?}, 0xc000d43e78)
	net/udpsock_posix.go:52 +0x1b8
net.(*UDPConn).readFromUDP(0xc00045c6a0, {0xc000ca02c0?, 0x2402d40?, 0x2402d40?}, 0x0?)
	net/udpsock.go:149 +0x31
net.(*UDPConn).ReadFromUDP(...)
	net/udpsock.go:141
github.com/jackpal/go-nat-pmp.(*network).call(0xc00000eea0, {0xc000ca024c, 0x2, 0x2}, 0x0)
	github.com/jackpal/go-nat-pmp@v1.0.2-0.20160603034137-1fa385a6f458/network.go:54 +0x38e
github.com/jackpal/go-nat-pmp.(*Client).rpc(0x18?, {0xc000ca024c, 0x2, 0x2}, 0x0?)
	github.com/jackpal/go-nat-pmp@v1.0.2-0.20160603034137-1fa385a6f458/natpmp.go:104 +0x46
github.com/jackpal/go-nat-pmp.(*Client).GetExternalAddress(0x0?)
	github.com/jackpal/go-nat-pmp@v1.0.2-0.20160603034137-1fa385a6f458/natpmp.go:56 +0x50
github.com/scroll-tech/go-ethereum/p2p/nat.discoverPMP.func1()
	github.com/scroll-tech/go-ethereum/p2p/nat/natpmp.go:73 +0xcf
created by github.com/scroll-tech/go-ethereum/p2p/nat.discoverPMP
	github.com/scroll-tech/go-ethereum/p2p/nat/natpmp.go:71 +0x6f

goroutine 139 [select]:
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).fetchParts(0xc000c3cc40, 0xc0001aa300, 0xc00169deb8, 0xc000af57a0, 0xc00169dea8, 0xc00169df08, 0xc00169def8, 0xc00169dee8, 0x0, 0x1754790, ...)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1352 +0x1cc
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).fetchBodies(0xc000c3cc40, 0xc000470e00?)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1279 +0x265
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func4()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:550 +0x25
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x70
created by github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x5f

goroutine 114 [select]:
github.com/scroll-tech/go-ethereum/p2p.(*Peer).run(0xc0015feba0)
	github.com/scroll-tech/go-ethereum/p2p/peer.go:255 +0x24d
github.com/scroll-tech/go-ethereum/p2p.(*Server).runPeer(0xc00028d180, 0xc0015feba0)
	github.com/scroll-tech/go-ethereum/p2p/server.go:1039 +0x1b3
created by github.com/scroll-tech/go-ethereum/p2p.(*Server).launchPeer
	github.com/scroll-tech/go-ethereum/p2p/server.go:1022 +0xd9

goroutine 141 [select]:
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc000c3cc40, 0xc00073f7d0?, 0xa74a27?)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1546 +0x174
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:552 +0x29
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x70
created by github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x5f

goroutine 140 [select]:
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).fetchParts(0xc000c3cc40, 0xc0001aa360, 0xc000cadeb8, 0xc000af5810, 0xc000cadea8, 0xc000cadf08, 0xc000cadef8, 0xc000cadee8, 0x0, 0x17547a0, ...)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1352 +0x1cc
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).fetchReceipts(0xc000c3cc40, 0xa74a27?)
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:1305 +0x265
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func5()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:551 +0x25
github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1()
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x70
created by github.com/scroll-tech/go-ethereum/eth/downloader.(*Downloader).spawnSync
	github.com/scroll-tech/go-ethereum/eth/downloader/downloader.go:573 +0x5f

goroutine 85 [IO wait]:
internal/poll.runtime_pollWait(0x7fdf901ca3b8, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0015f6800?, 0x0?, 0x0)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0015f6800)
	internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0015f6800)
	net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc0015c83a8)
	net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc0015c83a8)
	net/tcpsock.go:288 +0x3d
net/http.(*Server).Serve(0xc0015ea1c0, {0x18eaa60, 0xc0015c83a8})
	net/http/server.go:3039 +0x385
created by github.com/scroll-tech/go-ethereum/node.(*httpServer).start
	github.com/scroll-tech/go-ethereum/node/rpcstack.go:146 +0x290

goroutine 86 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 87 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 88 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 89 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 90 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 91 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 92 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 93 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 94 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 95 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 96 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 97 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 98 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 99 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 100 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 101 [select]:
github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers.func1()
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:50 +0xcc
created by github.com/scroll-tech/go-ethereum/eth.(*Ethereum).startBloomHandlers
	github.com/scroll-tech/go-ethereum/eth/bloombits.go:48 +0x34

goroutine 102 [select]:
github.com/scroll-tech/go-ethereum/eth.(*handler).txBroadcastLoop(0xc0000d5400)
	github.com/scroll-tech/go-ethereum/eth/handler.go:524 +0x118
created by github.com/scroll-tech/go-ethereum/eth.(*handler).Start
	github.com/scroll-tech/go-ethereum/eth/handler.go:401 +0xee

goroutine 103 [chan receive]:
github.com/scroll-tech/go-ethereum/eth.(*handler).minedBroadcastLoop(0xc0000d5400)
	github.com/scroll-tech/go-ethereum/eth/handler.go:512 +0x8a
created by github.com/scroll-tech/go-ethereum/eth.(*handler).Start
	github.com/scroll-tech/go-ethereum/eth/handler.go:406 +0x19b

goroutine 104 [select]:
github.com/scroll-tech/go-ethereum/eth.(*chainSyncer).loop(0xc000c2d1d0)
	github.com/scroll-tech/go-ethereum/eth/sync.go:119 +0x265
created by github.com/scroll-tech/go-ethereum/eth.(*handler).Start
	github.com/scroll-tech/go-ethereum/eth/handler.go:410 +0x1f8

goroutine 105 [chan receive]:
github.com/scroll-tech/go-ethereum/cmd/utils.StartNode.func1()
	github.com/scroll-tech/go-ethereum/cmd/utils/cmd.go:91 +0x2a7
created by github.com/scroll-tech/go-ethereum/cmd/utils.StartNode
	github.com/scroll-tech/go-ethereum/cmd/utils/cmd.go:76 +0xb9

goroutine 106 [chan receive]:
github.com/scroll-tech/go-ethereum/rpc.(*Server).ServeCodec(0xc00011f620, {0x18ed9d8?, 0xc000b33590}, 0x0?)
	github.com/scroll-tech/go-ethereum/rpc/server.go:88 +0x165
created by github.com/scroll-tech/go-ethereum/rpc.DialInProc.func1
	github.com/scroll-tech/go-ethereum/rpc/inproc.go:29 +0xc5

goroutine 107 [select]:
github.com/scroll-tech/go-ethereum/rpc.(*Client).dispatch(0xc00040d320, {0x18ed9d8?, 0xc000b33630})
	github.com/scroll-tech/go-ethereum/rpc/client.go:576 +0x2cc
created by github.com/scroll-tech/go-ethereum/rpc.initClient
	github.com/scroll-tech/go-ethereum/rpc/client.go:240 +0x3aa

goroutine 108 [chan receive]:
main.startNode.func1()
	github.com/scroll-tech/go-ethereum/cmd/geth/main.go:358 +0x89
created by main.startNode
	github.com/scroll-tech/go-ethereum/cmd/geth/main.go:350 +0x18a

goroutine 109 [select]:
github.com/scroll-tech/go-ethereum/eth/fetcher.(*BlockFetcher).loop(0xc0004477a0)
	github.com/scroll-tech/go-ethereum/eth/fetcher/block_fetcher.go:379 +0x367
created by github.com/scroll-tech/go-ethereum/eth/fetcher.(*BlockFetcher).Start
	github.com/scroll-tech/go-ethereum/eth/fetcher/block_fetcher.go:231 +0x56

goroutine 110 [select]:
github.com/scroll-tech/go-ethereum/eth/fetcher.(*TxFetcher).loop(0xc00043e630)
	github.com/scroll-tech/go-ethereum/eth/fetcher/tx_fetcher.go:355 +0x165
created by github.com/scroll-tech/go-ethereum/eth/fetcher.(*TxFetcher).Start
	github.com/scroll-tech/go-ethereum/eth/fetcher/tx_fetcher.go:337 +0x56

goroutine 75 [select]:
github.com/scroll-tech/go-ethereum/rpc.(*Client).dispatch(0xc0002bb200, {0x18ed9d8?, 0xc000b33590})
	github.com/scroll-tech/go-ethereum/rpc/client.go:576 +0x2cc
created by github.com/scroll-tech/go-ethereum/rpc.initClient
	github.com/scroll-tech/go-ethereum/rpc/client.go:240 +0x3aa

goroutine 76 [select]:
net.(*pipe).read(0xc0015f6980, {0xc000cca000, 0x200, 0x4693f2?})
	net/pipe.go:159 +0x14f
net.(*pipe).Read(0xc000d25fb0?, {0xc000cca000?, 0x2?, 0xc000d25f54?})
	net/pipe.go:142 +0x25
encoding/json.(*Decoder).refill(0xc0002903c0)
	encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc0002903c0)
	encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0002903c0, {0x1329ea0, 0xc00000f530})
	encoding/json/stream.go:63 +0x78
github.com/scroll-tech/go-ethereum/rpc.(*jsonCodec).readBatch(0xc000b33590)
	github.com/scroll-tech/go-ethereum/rpc/json.go:209 +0x4d
github.com/scroll-tech/go-ethereum/rpc.(*Client).read(0xc0002bb200, {0x18ed9d8, 0xc000b33590})
	github.com/scroll-tech/go-ethereum/rpc/client.go:649 +0xb7
created by github.com/scroll-tech/go-ethereum/rpc.(*Client).dispatch
	github.com/scroll-tech/go-ethereum/rpc/client.go:573 +0x155

goroutine 111 [select]:
net.(*pipe).read(0xc0015f6a00, {0xc00010cc00, 0x200, 0x4693f2?})
	net/pipe.go:159 +0x14f
net.(*pipe).Read(0x0?, {0xc00010cc00?, 0x0?, 0x0?})
	net/pipe.go:142 +0x25
encoding/json.(*Decoder).refill(0xc000290780)
	encoding/json/stream.go:165 +0x17f
encoding/json.(*Decoder).readValue(0xc000290780)
	encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc000290780, {0x1329ea0, 0xc0015c85b8})
	encoding/json/stream.go:63 +0x78
github.com/scroll-tech/go-ethereum/rpc.(*jsonCodec).readBatch(0xc000b33630)
	github.com/scroll-tech/go-ethereum/rpc/json.go:209 +0x4d
github.com/scroll-tech/go-ethereum/rpc.(*Client).read(0xc00040d320, {0x18ed9d8, 0xc000b33630})
	github.com/scroll-tech/go-ethereum/rpc/client.go:649 +0xb7
created by github.com/scroll-tech/go-ethereum/rpc.(*Client).dispatch
	github.com/scroll-tech/go-ethereum/rpc/client.go:573 +0x155

goroutine 112 [syscall]:
os/signal.signal_recv()
	runtime/sigqueue.go:151 +0x2f
os/signal.loop()
	os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
	os/signal/signal.go:151 +0x2a

goroutine 78 [sleep]:
time.Sleep(0xdf8475800)
	runtime/time.go:194 +0x12e
github.com/scroll-tech/go-ethereum/cmd/utils.monitorFreeDiskSpace(0x18db290?, {0xc0000f6588, 0x16}, 0x10000000)
	github.com/scroll-tech/go-ethereum/cmd/utils/cmd.go:119 +0x57
created by github.com/scroll-tech/go-ethereum/cmd/utils.StartNode.func1
	github.com/scroll-tech/go-ethereum/cmd/utils/cmd.go:88 +0x29b

goroutine 115 [IO wait]:
internal/poll.runtime_pollWait(0x7fdf901ca778, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000473600?, 0xc001744000?, 0x0)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000473600, {0xc001744000, 0x10000, 0x10000})
	internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000473600, {0xc001744000?, 0xc000cd3b58?, 0x464ff1?})
	net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0015f2198, {0xc001744000?, 0x421669?, 0x147c940?})
	net/net.go:183 +0x45
io.ReadAtLeast({0x18e2de0, 0xc0015f2198}, {0xc001744000, 0x10000, 0x10000}, 0x20)
	io/io.go:331 +0x9a
github.com/scroll-tech/go-ethereum/p2p/rlpx.(*readBuffer).read(0xc0002909c0, {0x18e2de0?, 0xc0015f2198?}, 0x20)
	github.com/scroll-tech/go-ethereum/p2p/rlpx/buffer.go:64 +0x1d8
github.com/scroll-tech/go-ethereum/p2p/rlpx.(*sessionState).readFrame(0xc0002908c0, {0x18e2de0, 0xc0015f2198})
	github.com/scroll-tech/go-ethereum/p2p/rlpx/rlpx.go:167 +0x107
github.com/scroll-tech/go-ethereum/p2p/rlpx.(*Conn).Read(0xc000b33770)
	github.com/scroll-tech/go-ethereum/p2p/rlpx/rlpx.go:137 +0x65
github.com/scroll-tech/go-ethereum/p2p.(*rlpxTransport).ReadMsg(0xc0015cb940)
	github.com/scroll-tech/go-ethereum/p2p/transport.go:64 +0x145
github.com/scroll-tech/go-ethereum/p2p.(*Peer).readLoop(0xc0015feba0, 0xc0015feba0?)
	github.com/scroll-tech/go-ethereum/p2p/peer.go:308 +0xd0
created by github.com/scroll-tech/go-ethereum/p2p.(*Peer).run
	github.com/scroll-tech/go-ethereum/p2p/peer.go:245 +0xf8

goroutine 116 [select]:
github.com/scroll-tech/go-ethereum/p2p.(*Peer).pingLoop(0xc0015feba0)
	github.com/scroll-tech/go-ethereum/p2p/peer.go:292 +0x11d
created by github.com/scroll-tech/go-ethereum/p2p.(*Peer).run
	github.com/scroll-tech/go-ethereum/p2p/peer.go:246 +0x13c

goroutine 117 [select]:
github.com/scroll-tech/go-ethereum/p2p.(*protoRW).ReadMsg(0xc0003ae460)
	github.com/scroll-tech/go-ethereum/p2p/peer.go:464 +0xbb
github.com/scroll-tech/go-ethereum/eth/protocols/eth.handleMessage({0x18f00a8, 0xc0000d5400}, 0xc000441a40)
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/handler.go:195 +0x88
github.com/scroll-tech/go-ethereum/eth/protocols/eth.Handle({0x18f00a8, 0xc0000d5400}, 0xc000441a40)
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/handler.go:161 +0x3d
github.com/scroll-tech/go-ethereum/eth/protocols/eth.MakeProtocols.func1.1(0xc000c35a40?)
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/handler.go:118 +0x27
github.com/scroll-tech/go-ethereum/eth.(*handler).runEthPeer(0xc0000d5400, 0xc000441a40, 0xc0015c8900)
	github.com/scroll-tech/go-ethereum/eth/handler.go:337 +0xffa
github.com/scroll-tech/go-ethereum/eth.(*ethHandler).RunPeer(0x42?, 0xc0015feba0?, 0x18e6f70?)
	github.com/scroll-tech/go-ethereum/eth/handler_eth.go:45 +0x19
github.com/scroll-tech/go-ethereum/eth/protocols/eth.MakeProtocols.func1(0x0?, {0x18e6f70, 0xc0003ae460})
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/handler.go:117 +0x122
github.com/scroll-tech/go-ethereum/p2p.(*Peer).startProtocols.func1()
	github.com/scroll-tech/go-ethereum/p2p/peer.go:407 +0x8c
created by github.com/scroll-tech/go-ethereum/p2p.(*Peer).startProtocols
	github.com/scroll-tech/go-ethereum/p2p/peer.go:405 +0xae

goroutine 118 [select]:
github.com/scroll-tech/go-ethereum/p2p.(*protoRW).ReadMsg(0xc0003ae960)
	github.com/scroll-tech/go-ethereum/p2p/peer.go:464 +0xbb
github.com/scroll-tech/go-ethereum/eth/protocols/snap.handleMessage({0x18ec0f8, 0xc0000d5400}, 0xc0015cbf00)
	github.com/scroll-tech/go-ethereum/eth/protocols/snap/handler.go:135 +0x84
github.com/scroll-tech/go-ethereum/eth/protocols/snap.handle({0x18ec0f8, 0xc0000d5400}, 0xc0015cbf00)
	github.com/scroll-tech/go-ethereum/eth/protocols/snap/handler.go:123 +0x3d
github.com/scroll-tech/go-ethereum/eth/protocols/snap.MakeProtocols.func2.1(0xc000470300?)
	github.com/scroll-tech/go-ethereum/eth/protocols/snap/handler.go:103 +0x27
github.com/scroll-tech/go-ethereum/eth.(*handler).runSnapExtension(0xc0000d5400, 0xc0015cbf00, 0xc0015c8888)
	github.com/scroll-tech/go-ethereum/eth/handler.go:352 +0x1c4
github.com/scroll-tech/go-ethereum/eth.(*snapHandler).RunPeer(0x1?, 0xc0015feba0?, 0x18e6f70?)
	github.com/scroll-tech/go-ethereum/eth/handler_snap.go:33 +0x19
github.com/scroll-tech/go-ethereum/eth/protocols/snap.MakeProtocols.func2(0x0?, {0x18e6f70?, 0xc0003ae960?})
	github.com/scroll-tech/go-ethereum/eth/protocols/snap/handler.go:102 +0xa2
github.com/scroll-tech/go-ethereum/p2p.(*Peer).startProtocols.func1()
	github.com/scroll-tech/go-ethereum/p2p/peer.go:407 +0x8c
created by github.com/scroll-tech/go-ethereum/p2p.(*Peer).startProtocols
	github.com/scroll-tech/go-ethereum/p2p/peer.go:405 +0xae

goroutine 119 [select]:
github.com/scroll-tech/go-ethereum/eth/protocols/eth.(*Peer).broadcastBlocks(0xc000441a40)
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/broadcast.go:44 +0xd9
created by github.com/scroll-tech/go-ethereum/eth/protocols/eth.NewPeer
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/peer.go:110 +0x45c

goroutine 120 [select]:
github.com/scroll-tech/go-ethereum/eth/protocols/eth.(*Peer).broadcastTransactions(0xc000441a40)
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/broadcast.go:105 +0x185
created by github.com/scroll-tech/go-ethereum/eth/protocols/eth.NewPeer
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/peer.go:111 +0x49f

goroutine 121 [select]:
github.com/scroll-tech/go-ethereum/eth/protocols/eth.(*Peer).announceTransactions(0xc000441a40)
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/broadcast.go:172 +0x17d
created by github.com/scroll-tech/go-ethereum/eth/protocols/eth.NewPeer
	github.com/scroll-tech/go-ethereum/eth/protocols/eth/peer.go:112 +0x4e5

goroutine 1238 [runnable]:
github.com/iden3/go-iden3-crypto/ff.NewElement(...)
	github.com/iden3/go-iden3-crypto@v0.0.12/ff/element.go:86
github.com/scroll-tech/go-ethereum/crypto/poseidon.zero(...)
	github.com/scroll-tech/go-ethereum/crypto/poseidon/poseidon.go:19
github.com/scroll-tech/go-ethereum/crypto/poseidon.permute({0xc01891d6c8?, 0x3, 0x3}, 0x3)
	github.com/scroll-tech/go-ethereum/crypto/poseidon/poseidon.go:91 +0x6cd
github.com/scroll-tech/go-ethereum/crypto/poseidon.HashFixed({0xc018205d80?, 0x2, 0x2})
	github.com/scroll-tech/go-ethereum/crypto/poseidon/poseidon.go:186 +0x228
github.com/scroll-tech/go-ethereum/core/types/zktrie.HashElems(0xc018205df0?, 0x2?, {0x0, 0x0, 0x0})
	github.com/scroll-tech/go-ethereum/core/types/zktrie/util.go:14 +0x52
github.com/scroll-tech/go-ethereum/core/types/zktrie.HashElems(0xc01894b200?, 0x1?, {0xc018205e58, 0x1, 0x1})
	github.com/scroll-tech/go-ethereum/core/types/zktrie/util.go:21 +0xc5
github.com/scroll-tech/go-ethereum/trie.LeafKey(0xc000000004?, 0xc017d16f80?)
	github.com/scroll-tech/go-ethereum/trie/zk_trie_node.go:115 +0x8f
github.com/scroll-tech/go-ethereum/trie.(*Node).Key(0xc018205f68)
	github.com/scroll-tech/go-ethereum/trie/zk_trie_node.go:138 +0xf4
github.com/scroll-tech/go-ethereum/trie.(*ZkTrieImpl).tryUpdate(0xc017df5dd0, 0xc018942e80, 0x4, {0xc017d16f80, 0x4, 0x4})
	github.com/scroll-tech/go-ethereum/trie/zk_trie_impl.go:116 +0x1e5
github.com/scroll-tech/go-ethereum/trie.(*ZkTrieImpl).TryUpdateAccount(0xc018743460?, {0xc018942e60?, 0x14?, 0x429067?}, 0x14?)
	github.com/scroll-tech/go-ethereum/trie/zk_trie_impl.go:536 +0x112
github.com/scroll-tech/go-ethereum/trie.(*ZkTrie).TryUpdateAccount(0xc0053a4c80, {0xc0185e7770, 0x14, 0x14}, 0x18?)
	github.com/scroll-tech/go-ethereum/trie/zk_trie.go:109 +0xa5
github.com/scroll-tech/go-ethereum/core/state.(*StateDB).updateStateObject(0xc000d7f040, 0xc018270d20)
	github.com/scroll-tech/go-ethereum/core/state/statedb.go:516 +0x16c
github.com/scroll-tech/go-ethereum/core/state.(*StateDB).IntermediateRoot(0xc000d7f040, 0x60?)
	github.com/scroll-tech/go-ethereum/core/state/statedb.go:923 +0x365
github.com/scroll-tech/go-ethereum/core.(*statePrefetcher).Prefetch(_, _, _, {0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, {0x0, ...}, ...}, ...)
	github.com/scroll-tech/go-ethereum/core/state_prefetcher.go:81 +0x6f8
github.com/scroll-tech/go-ethereum/core.(*BlockChain).insertChain.func3({0xc0d404d6bb393d15?, 0x54edd1139?, 0x2402d40?}, 0xc0068aa870?, 0xc000d7f040?, 0xc01808a4e0)
	github.com/scroll-tech/go-ethereum/core/blockchain.go:1608 +0xd5
created by github.com/scroll-tech/go-ethereum/core.(*BlockChain).insertChain
	github.com/scroll-tech/go-ethereum/core/blockchain.go:1607 +0x1bcf

When submitting logs: please submit them as text and not screenshots.

[hsyodyssey]

Several days ago, I found this function doesn't use the RWLock (https://github.com/scroll-tech/go-ethereum/blob/staging/trie/database.go#L365-L376). Wonder if this place is helpful for this issue.

[noel2004]

Several days ago, I found this function doesn't use the RWLock (https://github.com/scroll-tech/go-ethereum/blob/staging/trie/database.go#L365-L376). Wonder if this place is helpful for this issue.

Many thanks for your discovery. It is the caller's responsibility to lock the db outside of this function. (So we lock the db when calling it). I think the issue is infact caused by Commit function do not lock the preimage map

[hsyodyssey]

Several days ago, I found this function doesn't use the RWLock (https://github.com/scroll-tech/go-ethereum/blob/staging/trie/database.go#L365-L376). Wonder if this place is helpful for this issue.

Many thanks for your discovery. It is the caller's responsibility to lock the db outside of this function. (So we lock the db when calling it). I think the issue is infact caused by Commit function do not lock the preimage map

You are right Bro. In geth, they extracted this part into a new function with a RWMutex lock (https://github.com/ethereum/go-ethereum/blob/master/trie/preimages.go#L73-L87).

[noel2004]

Looking into the stacks and we found the issue should be cause by prefetching. By disabling it, the node can sync smoothly, under ~10mgasps, or ~100tps, everything works well and app can exit and restart normally. The memory usage is kept under 2GB

I am afraid we do not handle prefetch for skip any actions in trie's Commit, moreover I guess we would make no benefit from enabling prefetch so I advise disabling it permantly when zktrie is used.

Currently, prefetch can be disabled by --cache.noprefetch=true options