Fatal: Failed to register the Ethereum service: gap (#492149) in the chain between ancients and leveldb

[osizadmin]

System information

Geth version: 1.9.25-e7872729

Error : Fatal: Failed to register the Ethereum service: gap (#492149) in the chain between ancients and leveldb

We have faced the above error while start the geth in fast syncmode.

[TheEdgeOfRage]

I am seeing the same error after a SIGSEGV crash on the previous run. The only difference is that I'm doing a full sync.

Here is the log of the crash:

INFO [03-01|16:33:09.714] Starting Geth on Ethereum mainnet...                                                                                                                                                       
INFO [03-01|16:33:09.717] Maximum peer count                       ETH=50 LES=0 total=50
INFO [03-01|16:33:09.718] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"     
INFO [03-01|16:33:09.722] Set global gas cap                       cap=25000000                                                                                                                                      
INFO [03-01|16:33:09.723] Allocated trie memory caches             clean=1023.00MiB dirty=1024.00MiB                                                                                                                 
INFO [03-01|16:33:09.723] Allocated cache and file handles         database=/home/user/.ethereum/geth/chaindata cache=2.00GiB handles=262144
WARN [03-01|16:33:10.560] Truncating dangling indexes              database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes              database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes              database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes              database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.560] Truncating dangling indexes              database=/home/user/.ethereum/geth/chaindata/ancient table=bodies indexed=1.44GiB stored=1.44GiB
WARN [03-01|16:33:10.563] Truncating freezer table                 database=/home/user/.ethereum/geth/chaindata/ancient table=receipts items=2593457 limit=2593452
WARN [03-01|16:33:10.563] Truncating freezer table                 database=/home/user/.ethereum/geth/chaindata/ancient table=diffs    items=2593457 limit=2593452
WARN [03-01|16:33:10.563] Truncating freezer table                 database=/home/user/.ethereum/geth/chaindata/ancient table=headers  items=2593457 limit=2593452
WARN [03-01|16:33:10.563] Truncating freezer table                 database=/home/user/.ethereum/geth/chaindata/ancient table=hashes   items=2593457 limit=2593452
INFO [03-01|16:33:10.565] Opened ancient database                  database=/home/user/.ethereum/geth/chaindata/ancient       
INFO [03-01|16:33:10.576] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantin
ople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, YOLO v2: <nil>, Engine: ethash}"
INFO [03-01|16:33:10.576] Disk storage enabled for ethash caches   dir=/home/user/.ethereum/geth/ethash count=3
INFO [03-01|16:33:10.576] Disk storage enabled for ethash DAGs     dir=/home/user/.ethash               count=2                                                     
INFO [03-01|16:33:10.584] Initialising Ethereum protocol           versions="[65 64 63]" network=1 dbversion=8
ERROR[03-01|16:33:10.647] Canonical hash missing, can't freeze     number=2594277            
INFO [03-01|16:33:10.743] Deep froze chain segment                 blocks=825 elapsed=170.839ms number=2594276 hash="a41766…89ccd5"
INFO [03-01|16:33:12.419] Loaded most recent local header          number=2684683 hash="1bafa3…36aeb9" td=94044217750532710492 age=4y3mo4w
INFO [03-01|16:33:12.419] Loaded most recent local full block      number=2684683 hash="1bafa3…36aeb9" td=94044217750532710492 age=4y3mo4w
INFO [03-01|16:33:12.419] Loaded most recent local fast block      number=2684683 hash="1bafa3…36aeb9" td=94044217750532710492 age=4y3mo4w
WARN [03-01|16:33:12.425] Head state missing, repairing            number=2684683 hash="1bafa3…36aeb9"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x561131f1263f]
                                                                                                          
goroutine 1 [running]:                                                                                    
github.com/ethereum/go-ethereum/core/types.(*Block).Root(...)                      
        github.com/ethereum/go-ethereum/core/types/block.go:336
github.com/ethereum/go-ethereum/core.(*BlockChain).SetHeadBeyondRoot.func1(0x7fcbd876a1d8, 0xc0001f6240, 0xc0006566c0, 0x7fcbd876a1d8, 0xc0001f6240)
        github.com/ethereum/go-ethereum/core/blockchain.go:528 +0x2df
github.com/ethereum/go-ethereum/core.(*BlockChain).SetHeadBeyondRoot(0xc0000d7900, 0x28f70b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)                                                                                     
        github.com/ethereum/go-ethereum/core/blockchain.go:606 +0x295
github.com/ethereum/go-ethereum/core.(*BlockChain).SetHead(...)                         
        github.com/ethereum/go-ethereum/core/blockchain.go:484                                                                                                                                                       
github.com/ethereum/go-ethereum/core.NewBlockChain(0x561132a6e528, 0xc0001f6240, 0xc0005dc140, 0x5611330aba60, 0x561132a6c0f8, 0xc000c8a000, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/core/blockchain.go:310 +0x1af0                              
github.com/ethereum/go-ethereum/eth.New(0xc0000ba9c0, 0xc00015f200, 0x561133099010, 0x1, 0x1)                                                                                                                        
        github.com/ethereum/go-ethereum/eth/backend.go:175 +0xac6                                                                                                                                                    
github.com/ethereum/go-ethereum/cmd/utils.RegisterEthService(0xc0000ba9c0, 0xc00015f200, 0x0, 0x1)      
        github.com/ethereum/go-ethereum/cmd/utils/flags.go:1701 +0xec                                                                                                                                                
main.makeFullNode(0xc00011a580, 0x561132586ba6, 0x1, 0xc0000992c0)
        github.com/ethereum/go-ethereum/cmd/geth/config.go:165 +0xfb
main.geth(0xc00011a580, 0x0, 0x0)
        github.com/ethereum/go-ethereum/cmd/geth/main.go:354 +0xf6
gopkg.in/urfave/cli%2ev1.HandleAction(0x56113285bfa0, 0x561132a252e8, 0xc00011a580, 0xc000280e40, 0x0)
        gopkg.in/urfave/cli.v1@v1.20.0/app.go:490 +0x84
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc0000bc9c0, 0xc0000320a0, 0x5, 0x5, 0x0, 0x0)
        gopkg.in/urfave/cli.v1@v1.20.0/app.go:264 +0x5f5
main.main()
        github.com/ethereum/go-ethereum/cmd/geth/main.go:273 +0x57

And here is the log I get whenever I try to run geth now:

INFO [03-01|16:38:17.018] Starting Geth on Ethereum mainnet... 
INFO [03-01|16:38:17.019] Maximum peer count                       ETH=50 LES=0 total=50
INFO [03-01|16:38:17.020] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [03-01|16:38:17.021] Set global gas cap                       cap=25000000
INFO [03-01|16:38:17.021] Allocated trie memory caches             clean=1023.00MiB dirty=1024.00MiB
INFO [03-01|16:38:17.021] Allocated cache and file handles         database=/home/user/.ethereum/geth/chaindata cache=2.00GiB handles=262144
INFO [03-01|16:38:17.086] Opened ancient database                  database=/home/user/.ethereum/geth/chaindata/ancient
Fatal: Failed to register the Ethereum service: gap (#2594277) in the chain between ancients and leveldb

I run geth using the following command:

geth --syncmode full --cache 4096

My .ethereum directory is currently 15GB, but I could upload the whole thing somewhere if that makes it easier for you guys to debug this.

[ligi]

@osizadmin can you provide some logs? Did you have a crash before the problem you describe?

[ligi]

@TheEdgeOfRage the crash should be fixed in the recent release - but you will need to resync. Thanks for your log this was most helpful

[voronindev]

Hello there! We have something close to yours:

Command:
command: --gpo.blocks 10 --gpo.percentile 90 --ropsten --cache 2000 --syncmode "full" --nousb --http.api debug,eth,web3,personal,net --http --http.addr "0.0.0.0" --http.vhosts="*" --ws --ws.addr "0.0.0.0" --ws.origins="*" --port 30303 --ws.port 8546 --http.port 8545

Version:

OS: Ubuntu 20.04 LTS (GNU/Linux 5.4.0-28-generic x86_64)
Geth: v1.10.1

geth_1  | INFO [03-29|15:19:11.838] Maximum peer count                       ETH=50 LES=0 total=50
geth_1  | INFO [03-29|15:19:11.838] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
geth_1  | WARN [03-29|15:19:11.838] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable 
geth_1  | INFO [03-29|15:19:11.838] Set global gas cap                       cap=25000000
geth_1  | INFO [03-29|15:19:11.839] Allocated trie memory caches             clean=300.00MiB dirty=500.00MiB
geth_1  | INFO [03-29|15:19:11.839] Allocated cache and file handles         database=/root/.ethereum/ropsten/geth/chaindata cache=1000.00MiB handles=524288
geth_1  | INFO [03-29|15:19:12.841] Opened ancient database                  database=/root/.ethereum/ropsten/geth/chaindata/ancient
geth_1  | Fatal: Failed to register the Ethereum service: gap (#9719292) in the chain between ancients and leveldb
geth_1  | Fatal: Failed to register the Ethereum service: gap (#9719292) in the chain between ancients and leveldb```

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have more relevant information or answers to our questions so that we can investigate further.

[aquanow-pratik]

Hey, Can someone help on the same issue? I am having the same error. I am running geth with --syncmode=full
Fatal: Failed to register the Ethereum service: gap (#12254272) in the chain between ancients and leveldb INFO [05-09|05:17:31.432] Starting Geth on Ethereum mainnet... INFO [05-09|05:17:31.433] Maximum peer count ETH=50 LES=0 total=50 INFO [05-09|05:17:31.433] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" WARN [05-09|05:17:31.433] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable INFO [05-09|05:17:31.434] Set global gas cap cap=25,000,000 INFO [05-09|05:17:31.434] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB INFO [05-09|05:17:31.434] Allocated cache and file handles database=/home/ubuntu/data/geth/chaindata cache=2.00GiB handles=524,288 INFO [05-09|05:17:33.255] Opened ancient database database=/home/ubuntu/data/geth/chaindata/ancient readonly=false Fatal: Failed to register the Ethereum service: gap (#12254272) in the chain between ancients and leveldb

[bogdangainusa]

Same issue with geth version v.1.10.2

/home/cryptodaemon/bin/geth --datadir=/home/cryptodaemon/.ethereum --rpc --rpcapi personal,eth,web3 --rpcaddr localhost --rpcport 8545 --maxpeers=100 --cache 2048 --nousb --syncmode "fast" --verbosity 5
INFO [05-31|10:59:59.995] Starting Geth on Ethereum mainnet... 
INFO [05-31|10:59:59.997] Maximum peer count                       ETH=100 LES=0 total=100
WARN [05-31|10:59:59.997] The flag --rpc is deprecated and will be removed June 2021, please use --http 
WARN [05-31|10:59:59.997] The flag --rpcaddr is deprecated and will be removed June 2021, please use --http.addr 
WARN [05-31|10:59:59.997] The flag --rpcport is deprecated and will be removed June 2021, please use --http.port 
WARN [05-31|10:59:59.997] The flag --rpcapi is deprecated and will be removed June 2021, please use --http.api 
INFO [05-31|10:59:59.997] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [05-31|10:59:59.997] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable 
TRACE[05-31|10:59:59.997] Started watching keystore folder         path=/home/cryptodaemon/.ethereum/keystore
DEBUG[05-31|10:59:59.997] FS scan times                            list="338.446µs" set="97.653µs" diff="35.429µs"
TRACE[05-31|10:59:59.998] Handled keystore changes                 time=1.187307ms
DEBUG[05-31|10:59:59.999] Sanitizing Go's GC trigger               percent=50
INFO [05-31|10:59:59.999] Set global gas cap                       cap=25000000
INFO [05-31|10:59:59.999] Allocated trie memory caches             clean=307.00MiB dirty=512.00MiB
INFO [05-31|10:59:59.999] Allocated cache and file handles         database=/home/cryptodaemon/.ethereum/geth/chaindata cache=1024.00MiB handles=524288
DEBUG[05-31|11:00:02.309] Chain freezer table opened               database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=hashes items=12437071 size=379.55MiB
DEBUG[05-31|11:00:02.309] Chain freezer table opened               database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=bodies items=12437071 size=253.89MiB
DEBUG[05-31|11:00:02.310] Chain freezer table opened               database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=receipts items=12437071 size=1.61GiB
DEBUG[05-31|11:00:02.310] Chain freezer table opened               database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=diffs    items=12437071 size=123.48MiB
DEBUG[05-31|11:00:02.310] Chain freezer table opened               database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient table=headers  items=12437071 size=1.60GiB
INFO [05-31|11:00:02.310] Opened ancient database                  database=/home/cryptodaemon/.ethereum/geth/chaindata/ancient readonly=false
Fatal: Failed to register the Ethereum service: gap (#12437071) in the chain between ancients and leveldb

[MariusVanDerWijden]

Should be fixed by @fjl upcoming ancients refactor, will reopen and close once that's merged

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have more relevant information or answers to our questions so that we can investigate further.