Many Kusama nodes stuck with a large number of transactions in queue

[bjweaver]

Client ver: 0.9.3
Network: Kusama

One of our nodes, and many in telemetry are running very large transactions in queue backlogs

Nodes in this state will run very high CPU loads.

Restarting the polkadot service will usually clear the transaction backlog and lower the CPU usage.

Nodes should be able to clear this backlog on their own without a service restart.

[AlexZhenWang]

Same issue here. I checked the log, there are many Error importing block xxx: Err(Other(ClientImport("Invalid op │ │ eration in the pending changes tree: Tried to import or finalize node that is an ancestor of a previously finalized node"))) errors. Seems like the nodes were busy reverting blocks, which took lots of CPU resources.

I added --in-peers and --out-peers to use fewer peers, and found the CPU usage became lower.

[Swader]

This was due to an NFT promotion on RMRK and is over now. Restart your nodes and it should be fine now.

[bkchr]

This was due to an NFT promotion on RMRK and is over now. Restart your nodes and it should be fine now.

I want to have this again after paritytech/substrate#8963 :P

[bjweaver]

Restarting is a work around, but this is a bug that will need to be resolved as more and more projects increase the transaction load on Kusama and Polkadot.

[Swader]

Sure, it's being fixed in paritytech/substrate#8963