Error "Bad block"

[mikel2000]

System information

Geth version:
Version: 1.10.1-stable
Git Commit: c2d2f4e
Git Commit Date: 20210308
Architecture: amd64
Go Version: go1.16
Operating System: linux
GOPATH=
GOROOT=go

Yesterday I found in the log the following error. The same error messages repeats several times. Finally the server resumes working normally. This occurred on two different servers, so I assume, that it's not a server specific problem.

WARN [04-28|20:54:14.280] Failed to load old bad blocks            error="leveldb: not found"
ERROR[04-28|20:54:14.298]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, YOLO v3: <nil>, Engine: ethash}

Number: 12330686
Hash: 0xfcf34e5f4e403c24c1b70eb1a46c3bda99d45b10fa3d261f2ebf0d2c20867423
         0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc834252f73a4f0157fc41924080a9bf2782fd1e0f740cdd1b3011a15db51def1 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
...
         444: cumulative: 14952636 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x462eb6a9dbaf1f8706af8af7b5319707207ff3ed5eef3bbd32f6aa20842dde90 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:

Error: invalid merkle root (remote: 79933dd27171e0cc7459bba2c4879762e2e392e25db4560cb66b1c02c39b5545 local: 36e2428c5177cb706bd876d591b01889a86e02f6ab8d75ed0b74846a8fe3354b)
##############################

INFO [04-28|20:54:14.712] Skip duplicated bad block                number=12330686 hash="fcf34e…867423"

Starting here, the same error message from above repeats several times.

I just found, that the same error also occurred for block 12330852.

[yorickdowne]

Yup seeing the same, so not likely to be DB corruption caused by hardware failure: Not if it happens on the same block on entirely disparate servers.

In my case this is Geth/v1.10.2-stable-97d11b01/linux-amd64/go1.16.3 on Ubuntu 20.04

Could this be related to #22728 ?

eth1_1    | INFO [04-28|18:54:06.634] Imported new chain segment               blocks=1    txs=235    mgas=14.970  elapsed=265.300ms   mgasps=56.428  number=12330685 hash="e4772a…d7a535" dirty=1022.20MiB
eth1_1    | INFO [04-28|18:54:06.636] Unindexed transactions                   blocks=1    txs=64     tail=9980686 elapsed=1.723ms
eth1_1    | WARN [04-28|18:54:14.553] Failed to load old bad blocks            error="leveldb: not found"
eth1_1    | ERROR[04-28|18:54:14.585]
eth1_1    | ########## BAD BLOCK #########
eth1_1    | Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, YOLO v3: <nil>, Engine: ethash}
eth1_1    |
eth1_1    | Number: 12330686
eth1_1    | Hash: 0xfcf34e5f4e403c24c1b70eb1a46c3bda99d45b10fa3d261f2ebf0d2c20867423

and

eth1_1    | INFO [04-28|19:30:45.994] Imported new chain segment               blocks=1    txs=415    mgas=14.976  elapsed=288.364ms   mgasps=51.934  number=12330851 hash="fd8c80…c072c7" dirty=1022.26MiB
eth1_1    | INFO [04-28|19:30:45.997] Unindexed transactions                   blocks=1    txs=96     tail=9980852 elapsed=2.233ms
eth1_1    | INFO [04-28|19:30:50.184] Deep froze chain segment                 blocks=6    elapsed=17.647ms    number=12240851 hash="975911…5a49c5"
eth1_1    | ERROR[04-28|19:31:46.151]
eth1_1    | ########## BAD BLOCK #########
eth1_1    | Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, YOLO v3: <nil>, Engine: ethash}
eth1_1    |
eth1_1    | Number: 12330852
eth1_1    | Hash: 0xbc9e15b17e3a132773825bd463cb5cf6949a5de991fdb62e2940e85b0ccc0ff9

[holiman]

Those are actually bad, not a local problem. Besu reports them aswell. See https://forkmon.ethdevops.io/

[karalabe]

The question is why they exist in the first place. We need to find who mined them, maybe something's of there.

[karalabe]

"Flexpool/S1/SE - Rome" and "Flexpool/H0/SE - Seattle"

[karalabe]

Closing this issue as the bad blocks are indeed bad. We'll continue investigating where they come from.

[karalabe]

Not a bug.