Already shutting down, interrupt more to panic

[MareG76]

Geth version: 1.10.25
OS & Version: Windows 10

Expected behaviour

Almost every version of get lately gives me this problem, after hitting ctrl+c it just won't stop, some times I wait half an hour and nothing

Actual behaviour

It gives me "Already shutting down, interrupt more to panic" and when i interrupt it more I get this:
https://pastebin.com/HpWBRxMp

Steps to reproduce the behaviour

Try to stop geth on Windows 10 with ctrl+c

[holiman]

The shutdown sequence has become stuck here:

goroutine 55628060 [semacquire, 5 minutes]:
sync.runtime_Semacquire(0x3?)
        runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc003207bc0?)
        sync/waitgroup.go:136 +0x52
github.com/ethereum/go-ethereum/eth.(*handler).Stop(0xc0003ea600)
        github.com/ethereum/go-ethereum/eth/handler.go:565 +0x89
github.com/ethereum/go-ethereum/eth.(*Ethereum).Stop(0xc0003ea500)
        github.com/ethereum/go-ethereum/eth/backend.go:548 +0x50
github.com/ethereum/go-ethereum/node.(*Node).stopServices(0xc0000f8460, {0xc0000b7690, 0x1, 0x28?})
        github.com/ethereum/go-ethereum/node/node.go:303 +0xbe
github.com/ethereum/go-ethereum/node.(*Node).Close(0xc0000f8460)
        github.com/ethereum/go-ethereum/node/node.go:223 +0x191
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1.1
        github.com/ethereum/go-ethereum/cmd/utils/cmd.go:92 +0x96

I.e :

        // Disconnect existing sessions.
        // This also closes the gate for any new registrations on the peer set.
        // sessions which are already established but not added to h.peers yet
        // will exit when they try to register.
        h.peers.close()
        h.peerWG.Wait() <-- HERE

The peerWG is added to once for runEthPeer and runSnapExtension. Here's an eth handler, stuck trying to deliver block headers:

goroutine 11159874 [select, 1276 minutes]:
github.com/ethereum/go-ethereum/eth/protocols/eth.(*Peer).dispatchResponse(0xc0a4863860, 0xc08cde7da0, 0xc0536ef678)
        github.com/ethereum/go-ethereum/eth/protocols/eth/dispatcher.go:172 +0x28f
github.com/ethereum/go-ethereum/eth/protocols/eth.handleBlockHeaders66({0x7ff77361d580?, 0xc0001dae40?}, {0x7ff7739b8800, 0xc08cde6240}, 0xc0c0aef5914f4320?)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handlers.go:367 +0x178
github.com/ethereum/go-ethereum/eth/protocols/eth.handleMessage({0x7ff7739be900, 0xc0003ea600}, 0xc0a4863860)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:230 +0x597
github.com/ethereum/go-ethereum/eth/protocols/eth.Handle({0x7ff7739be900, 0xc0003ea600}, 0xc0a4863860)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:154 +0x3d
github.com/ethereum/go-ethereum/eth/protocols/eth.MakeProtocols.func1.1(0xc0a4863860?)
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:111 +0x27
github.com/ethereum/go-ethereum/eth.(*handler).runEthPeer(0xc0003ea600, 0xc0a4863860, 0xc262f38e40)
        github.com/ethereum/go-ethereum/eth/handler.go:475 +0x117a
github.com/ethereum/go-ethereum/eth.(*ethHandler).RunPeer(0x42?, 0xc26b531500?, 0x7ff7739b7090?)
        github.com/ethereum/go-ethereum/eth/handler_eth.go:41 +0x19
github.com/ethereum/go-ethereum/eth/protocols/eth.MakeProtocols.func1(0xc0fa3eaf00?, {0x7ff7739b7090, 0xc07419c0a0})
        github.com/ethereum/go-ethereum/eth/protocols/eth/handler.go:110 +0x122
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1()
        github.com/ethereum/go-ethereum/p2p/peer.go:415 +0x8c
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        github.com/ethereum/go-ethereum/p2p/peer.go:413 +0xae

This is where it's been sitting for 21 hours:

// dispatchRequest fulfils a pending request and delivers it to the requested
// sink.
func (p *Peer) dispatchResponse(res *Response, metadata func() interface{}) error {
        resOp := &response{
                res:  res,
                fail: make(chan error),
        }
        res.recv = time.Now()
        res.Done = make(chan error)

        select {
        case p.resDispatch <- resOp:
                // Ensure the response is accepted by the dispatcher
                if err := <-resOp.fail; err != nil {
                        return nil
                }
                // Request was accepted, run any postprocessing step to generate metadata
                // on the receiver thread, not the sink thread
                if metadata != nil {
                        res.Meta = metadata()
                }
                // Deliver the filled out response and wait until it's handled. This
                // path is a bit funky as Go's select has no order, so if a response
                // arrives to an already cancelled request, there's a 50-50% changes
                // of picking on channel or the other. To avoid such cases delivering
                // the packet upstream, check for cancellation first and only after
                // block on delivery.
                select { 
                case <-res.Req.cancel:
                        return nil // Request cancelled, silently discard response
                default:
                        // Request not yet cancelled, attempt to deliver it, but do watch
                        // for fresh cancellations too
                        select { <--- STUCK ON THIS LINE
                        case res.Req.sink <- res:
                                return <-res.Done // Response delivered, return any errors
                        case <-res.Req.cancel:
                                return nil // Request cancelled, silently discard response
                        }
                }

[holiman]

I think this should fix it

diff --git a/eth/protocols/eth/dispatcher.go b/eth/protocols/eth/dispatcher.go
index 65a935d555..1486a70022 100644
--- a/eth/protocols/eth/dispatcher.go
+++ b/eth/protocols/eth/dispatcher.go
@@ -174,6 +174,8 @@ func (p *Peer) dispatchResponse(res *Response, metadata func() interface{}) erro
 				return <-res.Done // Response delivered, return any errors
 			case <-res.Req.cancel:
 				return nil // Request cancelled, silently discard response
+			case <-p.term:
+				return errDisconnected
 			}
 		}

That should fix the shutdown problem, but there's some other underlying error here which also needs to be fixed, namely -- why doesn't that request become cancelled earlier?