Fatal: Error starting protocol stack: genesis mismatch

[sssubik]

Hi there,

System information

Geth version: Geth/v1.9.13-stable
OS & Version: ubuntu 18.04

Expected behaviour

My server turned off abruptly where my full node GETH was running. Now when I fix the server and open the node the node should start running.

Actual behaviour

The GETH node before the server crashed was running fully synced and was displaying importing chain segment.
When I fixed the server after server turned off abruptly, and executing geth I am getting log as:

INFO [07-09|18:08:46.552] Starting Geth on Ethereum mainnet... 
INFO [07-09|18:08:46.553] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [07-09|18:08:46.555] Maximum peer count                       ETH=50 LES=0 total=50
INFO [07-09|18:08:46.555] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[07-09|18:08:46.555] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-09|18:08:46.555] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-09|18:08:46.555] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-09|18:08:46.555] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-09|18:08:46.555] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-09|18:08:46.556] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [07-09|18:08:46.556] Starting peer-to-peer node               instance=Geth/v1.9.13-stable-cbc4ac26/linux-amd64/go1.14.2
INFO [07-09|18:08:46.556] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [07-09|18:08:46.556] Allocated cache and file handles         database=/home/user/.ethereum/geth/chaindata cache=2.00GiB handles=524288
ERROR[07-09|18:08:47.563] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-09|18:08:47.564] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-09|18:08:47.564] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=3 err="failed to initialize libusb: libusb: unknown error [code -99]"
DEBUG[07-13|14:21:29.599] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=headers items=10329515 size=0.00B
DEBUG[07-13|14:21:29.599] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=hashes  items=10329515 size=0.00B
DEBUG[07-13|14:21:29.599] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=bodies  items=10329515 size=0.00B
DEBUG[07-13|14:21:29.599] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=receipts items=10329515 size=0.00B
DEBUG[07-13|14:21:29.600] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=diffs    items=10329515 size=0.00B
INFO [07-13|14:21:29.600] Opened ancient database                  database=/home/user/.ethereum/geth/chaindata/ancient
Fatal: Error starting protocol stack: genesis mismatch: 0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3 (leveldb) !=  (ancients))

The server was off for around 8 hours is this the reason there is a mismatch problem? I dont know why the Genesis mismatch problem is there. Could you provide me with a solution to initialize the GETH node with a genesis block again or any other solution
Steps to reproduce the behaviour
Install GETH full node and if the server turns off due to some problem the above error shows up if you reboot the server.

[sssubik]

Hey @karalabe . Really need your help. Please could you tell me some solution or any help..

[rjl493456442]

Hi @sssubik sorry for the late reply. Is the corrupted database still available?

#21327

I added a debug log to expose the root error for the failure, hopefully it's not late

[sssubik]

Hey @rjl493456442 Yes the corrupted database is still available. What should I do? Do I need to resync?
Okay I will show you the log once the commit is done.. Will it be done today?

[sssubik]

Hey @rjl493456442 I tried the new version and now I am getting this:

 `DEBUG[07-14|08:29:09.522] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=receipts items=10329515 size=0.00B
DEBUG[07-14|08:29:09.522] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=diffs    items=10329515 size=0.00B
DEBUG[07-14|08:29:09.522] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=headers  items=10329515 size=0.00B
DEBUG[07-14|08:29:09.522] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=hashes   items=10329515 size=0.00B
DEBUG[07-14|08:29:09.522] Chain freezer table opened               database=/home/user/.ethereum/geth/chaindata/ancient table=bodies   items=10329515 size=0.00B
INFO [07-14|08:29:09.522] Opened ancient database                  database=/home/user/.ethereum/geth/chaindata/ancient
Fatal: Error starting protocol stack: failed to retrieve genesis from ancient EOF`

What would I need to do? @rjl493456442

[sssubik]

Hey @rjl493456442. I did some research and found no solution. I find that the GETH once shutdown abruptly prunes the states in freezer without pruning level db due to some roll back to correct state. Could you provide me more detailed explanation or solution?

[rjl493456442]

@sssubik Can you list your ancient directory?

Something I don't understand is items=10329515 size=0.00B there are 10M items in the ancient directory but the head file size is 0.

Now I have no clue why the ancient will return an empty genesis for you. It's not easy to debug without a corrupted database. Maybe you can help a bit by offering some information.

[sssubik]

`bodies.0000.cdat  bodies.0021.cdat  bodies.0042.cdat    receipts.0004.cdat
bodies.0001.cdat  bodies.0022.cdat  bodies.0043.cdat    receipts.0005.cdat
bodies.0002.cdat  bodies.0023.cdat  bodies.0044.cdat    receipts.0006.cdat
bodies.0003.cdat  bodies.0024.cdat  bodies.0045.cdat    receipts.0007.cdat
bodies.0004.cdat  bodies.0025.cdat  bodies.0046.cdat    receipts.0008.cdat
bodies.0005.cdat  bodies.0026.cdat  bodies.0047.cdat    receipts.0009.cdat
bodies.0006.cdat  bodies.0027.cdat  bodies.0048.cdat    receipts.0010.cdat
bodies.0007.cdat  bodies.0028.cdat  bodies.cidx         receipts.0011.cdat
bodies.0008.cdat  bodies.0029.cdat  diffs.0000.rdat     receipts.0012.cdat
bodies.0009.cdat  bodies.0030.cdat  diffs.ridx          receipts.0013.cdat
bodies.0010.cdat  bodies.0031.cdat  FLOCK               receipts.0014.cdat
bodies.0011.cdat  bodies.0032.cdat  hashes.0000.rdat    receipts.0015.cdat
bodies.0012.cdat  bodies.0033.cdat  hashes.ridx         receipts.0016.cdat
bodies.0013.cdat  bodies.0034.cdat  headers.0000.cdat   receipts.0017.cdat
bodies.0014.cdat  bodies.0035.cdat  headers.0001.cdat   receipts.0018.cdat
bodies.0015.cdat  bodies.0036.cdat  headers.0002.cdat   receipts.0019.cdat
bodies.0016.cdat  bodies.0037.cdat  headers.cidx        receipts.0020.cdat
bodies.0017.cdat  bodies.0038.cdat  receipts.0000.cdat  receipts.0021.cdat
bodies.0018.cdat  bodies.0039.cdat  receipts.0001.cdat  receipts.cidx
bodies.0019.cdat  bodies.0040.cdat  receipts.0002.cdat
bodies.0020.cdat  bodies.0041.cdat  receipts.0003.cdat
`

Hey I have listed my ancient directory. Also the space I calculate gives me this:
131G ancient
If anything more need I will provide. Would be really frustrating if have to resync..

[rjl493456442]

Can you also give me the size of hashes.ridx and hashes.0000.rdat? Thanks!

[sssubik]

60M hashes.ridx
0 hashes.0000.rdat

hashes.0000.rdat shows 0 MB

[rjl493456442]

@sssubik I have to say sorry. I think there is a bug in the codebase, but the ancient database is already broken(and useless unless we do some manual recovery).

[sssubik]

@rjl493456442 What does this mean. Do I need to resync? Or can I do some manual recovery? Also if I need to resync am I gonna be able to leverage the downloaded states I already have to complete resync in less time?

[rjl493456442]

Do you have any other synced Geth instance?

Manual recovery means:

• Drop another synced hashes.0000.rdat file there for replacement(Note, the new file should contain more elements then the broken one
• Write a script the iterate the Header table and then inject the header hash into the broken hashes.0000.rdat file. It needs some coding work, maybe not a good idea

Also if I need to resync am I gonna be able to leverage the downloaded states I already have to complete resync in less time

To be honest I don't think we can plugin the downloaded states without any modification. The hacky way is: iterating the leveldb, delete all chain data(headers, tds, receipts, bodies, chain marker) but keep the states and then re-sync.

Download the chain is fast, download the state is painful.

[rjl493456442]

Can you search the history log that whether Truncating dangling head ... or Truncating dangling indexes ... or Truncating freezer table even appear?
I am digging the root cause and the log might help.

[sssubik]

Hey I dont have any other synced GETH instance

Drop another synced hashes.0000.rdat file there for replacement(Note, the new file should contain more elements then the broken one

So, can I find a synced hashes.0000.rdat file from somewhere else.
Also

The hacky way is: iterating the leveldb, delete all chain data(headers, tds, receipts, bodies, chain marker) but keep the states and then re-sync.

Do you mean this geth removedb? But isnt the problem in the ancient folder? And to resync I just run the geth command again? Also how long could it take?

Can you search the history log that whether Truncating dangling head ... or Truncating dangling indexes ... or Truncating freezer table even appear?

As far as I know how to search log I get this:

sudo journalctl -u geth.service --no-page | grep [Truncating dangling head ]
grep: Unmatched [ or [^
Subik@h2883546:~$ sudo journalctl -u geth.service --no-page | grep [Truncating dangling indexes ]
grep: Unmatched [ or [^
Subik@h2883546:~$ sudo journalctl -u geth.service --no-page | grep [Truncating freezer table]
grep: Unmatched [ or [^

I cant find the log you just sent me

[sssubik]

Sorry the above search of logs was wrong. I do get this:

Jul 09 11:20:42  WARN [07-09|11:20:42.683] Truncating dangling head                 database=/home/user/.ethereum/geth/chaindata/ancient table=diffs indexed=0.00B stored=101.37MiB
Jul 09 11:20:42 WARN [07-09|11:20:42.687] Truncating dangling head                 database=/home/user/.ethereum/geth/chaindata/ancient table=headers indexed=0.00B stored=1.86GiB
Jul 09 11:20:42  WARN [07-09|11:20:42.695] Truncating dangling head                 database=/home/user/.ethereum/geth/chaindata/ancient table=hashes  indexed=0.00B stored=315.23MiB
Jul 09 11:20:42 WARN [07-09|11:20:42.701] Truncating dangling head                 database=/home/user/.ethereum/geth/chaindata/ancient table=bodies  indexed=0.00B stored=1.86GiB
Jul 09 11:20:42  WARN [07-09|11:20:42.712] Truncating dangling head                 database=/home/user/.ethereum/geth/chaindata/ancient table=receipts indexed=0.00B stored=1.86GiB