Fresh full node start results in spam of `🔍 Discovered new external address for our node`

[skunert]

When starting a new node like so:

polkadot-omni-node --chain asset-hub-kusama-genesis.json --tmp

Results in the logs getting flooded with new address discoveries for different ports:

<standard startup message>
2025-01-16 17:10:54 [Parachain] Running JSON-RPC server: addr=127.0.0.1:9944,[::1]:9944    
2025-01-16 17:10:54 [Parachain] 🏁 CPU single core score: 1016.38 MiBs, parallelism score: 882.51 MiBs with expected cores: 8    
2025-01-16 17:10:54 [Parachain] 🏁 Memory score: 12.27 GiBs    
2025-01-16 17:10:54 [Parachain] 🏁 Disk score (seq. writes): 1.01 GiBs    
2025-01-16 17:10:54 [Parachain] 🏁 Disk score (rand. writes): 431.27 MiBs    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/36438/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/52486/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/44950/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/42304/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/36166/ws/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/60362/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/47590/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/42168/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:54 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41842/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:55 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/30334/ws/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:55 [Parachain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/30333/ws/p2p/12D3KooWRKHfcHjFxkVMMjV89AF3cUovsRunxpKdbW6mCaPRB3Kz    
2025-01-16 17:10:55 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/57346/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/50818/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/36246/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/37326/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/54562/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/51544/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/59034/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/39896/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:56 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/30334/ws/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:10:57 [Relaychain] Report 12D3KooWMrmrKcBk5PNjzqz3e6QqbPxXumznxxBSekQrWkZhg1p1: -2147483648 to -2147483648. Reason: Unsupported protocol. Banned, disconnecting.    
2025-01-16 17:10:59 [Relaychain] ⚙️  Syncing, target=#26674705 (8 peers), best: #3679 (0x234b…bf17), finalized #3584 (0xcdc0…728c), ⬇ 360.1kiB/s ⬆ 25.1kiB/s    
2025-01-16 17:10:59 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 0.5kiB/s ⬆ 0.2kiB/s    
2025-01-16 17:11:04 [Relaychain] ⚙️  Syncing 831.8 bps, target=#26674706 (8 peers), best: #7838 (0x814e…7b64), finalized #7680 (0x8862…b1be), ⬇ 265.3kiB/s ⬆ 6.3kiB/s    
2025-01-16 17:11:04 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 5 B/s ⬆ 25 B/s    
2025-01-16 17:11:04 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/56598/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:04 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/53764/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:04 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/33274/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:06 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/44500/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:09 [Relaychain] ⚙️  Syncing 601.2 bps, target=#26674707 (8 peers), best: #10844 (0x80cf…1e23), finalized #10752 (0x0877…1abf), ⬇ 389.5kiB/s ⬆ 4.5kiB/s    
2025-01-16 17:11:09 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 5 B/s ⬆ 25 B/s    
2025-01-16 17:11:10 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41300/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:12 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/54882/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:14 [Relaychain] ⚙️  Syncing 785.2 bps, target=#26674708 (8 peers), best: #14770 (0x0be7…9533), finalized #13824 (0xb224…3695), ⬇ 216.4kiB/s ⬆ 5.6kiB/s    
2025-01-16 17:11:14 [Parachain] 💤 Idle (0 peers), best: #0 (0x9256…0469), finalized #0 (0x9256…0469), ⬇ 18 B/s ⬆ 38 B/s    
2025-01-16 17:11:14 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/59388/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/46642/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/56654/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/60186/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/34190/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/45398/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41162/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/50954/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/48586/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/40156/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41818/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/44914/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/60434/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/39046/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/48714/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/35934/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip6/2001:bc8:1201:a0d:3eec:efff:fefe:eea8/tcp/48054/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/45414/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/46450/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/50894/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:15 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/41426/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    
2025-01-16 17:11:16 [Relaychain] 🔍 Discovered new external address for our node: /ip4/195.154.91.169/tcp/56786/p2p/12D3KooWMAksX9CguoyeMwQH3Pkv495ahwmJ6XVtbnVZqxwkspeZ    

This continues for minutes/hours. I encountered this issue on a scaleway dev machine, can easily be reproduced there.

cc @paritytech/networking

[lexnv]

I suspect this is coming from libp2p implementation on substrate side:

• we confirm all addresses
  

  polkadot-sdk/substrate/client/network/src/peer_info.rs

  Lines 408 to 419 in be2404c

  	FromSwarm::NewExternalAddrCandidate(e @ NewExternalAddrCandidate { addr }) => {
  	self.ping.on_swarm_event(FromSwarm::NewExternalAddrCandidate(e));
  	self.identify.on_swarm_event(FromSwarm::NewExternalAddrCandidate(e));
  	// Manually confirm all external address candidates.
  	// TODO: consider adding [AutoNAT protocol](https://docs.rs/libp2p/0.52.3/libp2p/autonat/index.html)
  	// (must go through the polkadot protocol spec) or implemeting heuristics for
  	// approving external address candidates. This can be done, for example, by
  	// approving only addresses reported by multiple peers.
  	// See also https://github.com/libp2p/rust-libp2p/pull/4721 introduced
  	// in libp2p v0.53 for heuristics approach.
  	self.pending_actions.push_back(ToSwarm::ExternalAddrConfirmed(addr.clone()));

• then we propagate them to discovery
  https://github.com/paritytech/polkadot-sdk/blob/be2404cccd9923c41e2f16bfe655f19574f1ae0e/substrate/client/network/src/discovery.rs#L778-L800C1

In litep2p we wait for 2 confirmations on the same address from different peers before propagating the address

polkadot-sdk/substrate/client/network/src/litep2p/discovery.rs

Line 79 in be2404c

const MIN_ADDRESS_CONFIRMATIONS: usize = 2;

[skunert]

Can confirm that it does not happen for litep2p

[skunert]

In litep2p we wait for 2 confirmations on the same address from different peers before propagating the address

This sounds reasonable to me.

One question is why we are seeing this just now, you think this broke with the most recent libp2p upgrade?

[lexnv]

There has been quite a few refactors / changes to the port reuse, I expect they are coming from:

Deprecate port_reuse setting, as this is now decided by the behaviour, not the transport.

• refactor(*): Transport redesign libp2p/rust-libp2p#4568

Other issue on origin/master

On the side, our origin/master libp2p does not contain the following fix for TCP:

• Fix the disabling of Nagle's algorithm, which requires setting TCP_NODELAY to true.

• fix(tcp): make TCP_NODELAY actually the default libp2p/rust-libp2p#5764

This is a bit concerning, as the changelog suggest the TCP_NODELAY issue was unnoticed for ~7months (cc libp2p/rust-libp2p#4916)

Will need to double check this, maybe there was a point release from another branch

[dmitry-markin]

Other issue on origin/master

On the side, our origin/master libp2p does not contain the following fix for TCP:

• Fix the disabling of Nagle's algorithm, which requires setting TCP_NODELAY to true.

This is a bit concerning, as the changelog suggest the TCP_NODELAY issue was unnoticed for ~7months (cc libp2p/rust-libp2p#4916)

Will need to double check this, maybe there was a point release from another branch

TCP_NODELAY is explicitly set to true in substrate, so this issue does not affect us:

polkadot-sdk/substrate/client/network/src/transport.rs

Line 51 in f5673cf

let tcp_config = tcp::Config::new().nodelay(true);

[dmitry-markin]

This issue happens due to a bug with address translation in libp2p-0.54.1: libp2p/rust-libp2p#5820

Due to this bug, external address candidates are returned as is with ephemeral ports. Even though libp2p-0.54.1 enforces port reuse, ephemeral ports are still used in two cases:

1. There is no listen address for specific transport. E.g., we listen on /ip4/.../tcp/30333/ws but not on bare TCP — in this case ephemeral ports will be used when connecting to TCP addresses like /ip4/.../tcp/30333/p2p/....
2. Multiple addresses of a remote node resolve to the same IP:port pair — in this case socket address can't be reused for a second connection during concurrent dial. For example polkadot-bootnode-0.polkadot.io has even four addresses that resolve to the same IP:port:
   • /dns/polkadot-bootnode-0.polkadot.io/tcp/30333
   • /dns4/cc1-0.parity.tech/tcp/30333
   • /ip4/51.75.144.133/tcp/30333
   • /dns/51.75.144.133/tcp/30333

When concurrent dial is disabled and listen addresses are provided for all transport (on both IPv4 & IPv6), ephemeral addresses are not discovered.

I tried fixing the libp2p bug by passing the correct PortUse from transport's dial() implementation all the way back to Swarm and, finally, to Identify behavior, but the required refactoring is too involved. Also, we would need to wait for the fix to be released in libp2p and upgrade the version of libp2p used by substrate to the next major release.

More feasible solution now would be to discard ExternalAddrCandidate messages from libp2p completely with all the peculiarities of address translation and instead use raw Identify messages to get observed addresses + apply litep2p heuristic of confirming addresses as external if two peers report them. While losing the information of whether ephemeral port was used or not and slightly reducing the reliability of the external address discovery (but not worse than litep2p), such solution would be self contained and under our control.

I am going to create a PR employing the latter approach.

[jxs]

Hi all, and thanks for the bug report upstream!

I tried fixing the libp2p bug by passing the correct PortUse from transport's dial() implementation all the way back to Swarm and, finally, to Identify behavior, but the required refactoring is too involved. Also, we would need to wait for the fix to be released in libp2p and upgrade the version of libp2p used by substrate to the next major release.

Would love for this path to be followed, and can promise to take a look into the PR and release a new patch version with the fix as soon as it is merged. If you prefer following what you described as option 2 can you share what you already did so I or anyone can take over?
Thanks