QUIC Connection local TLS error - Peer closed with transport code 306

[gherkins]

Describe the bug

I'm trying to connect to Solana TPU leaders via the quic string from the concactInfo,

tpuQuic: '18.132.XXX.XX:8009',

but consistently get:

QUIC Connection local TLS error - Peer closed with transport code 306

This happens to everyone of the Adresses from the cluster.
I just found error code 306 here:

js-quic/src/native/types.ts

Line 239 in 94f3839

DecodeError = 306,

which made me think I did the randomBytes part wrong, but I don't really see how...
I also tried using peculiarWebcrypto as in the benchmarks...

To Reproduce

import {QUICClient} from "@matrixai/quic";
import {getRandomValues} from "node:crypto";

const client = await QUICClient.createQUICClient({
        host: HOST_IP
        port: parseInt(PORT),
        config: {
            verifyPeer: false,
        },
        crypto: {
            ops: {
                randomBytes: async (data: ArrayBuffer) => {
                    getRandomValues(new Uint8Array(data));
                },
            },
        },
    }
)

// -> ERROR BEING THROWN HERE

const clientStream = client.connection.newStream();
const writer = clientStream.writable.getWriter()
await writer.write(BUFFER);
await writer.close();

Expected behavior

established connection w/o errors

Platform (please complete the following information)

macOS 14.4.1
node v20.11.1
ts-node v10.9.2

[linear]

ENG-287 QUIC Connection local TLS error - Peer closed with transport code 306

[CMCDragonkai]

What QUIC stack is Solana TPU using? This library is based on Cloudflare's quic and it follows a particular bootstrapping process. Best way to debug this it also build and run the Solana TPU side, so you can see what why it is closing with 306 error.

On the otherhand, we have tests/utils.ts. You can see randomBytes utility function there, we use that for our testing, as prefer less nodeisms.

[gherkins]

Thank you, I tried the randomBytes implementation as in test/utils.ts,
but that seems not to change much.

import {QUICClient} from "@matrixai/quic";
import * as peculiarWebcrypto from '@peculiar/webcrypto';

const webcrypto = new peculiarWebcrypto.Crypto();

async function randomBytes(data: ArrayBuffer) {
    webcrypto.getRandomValues(new Uint8Array(data));
}

const client = await QUICClient.createQUICClient({
  host: HOST_IP,
  port: parseInt(PORT),
  config: {
      verifyPeer: false,
  },
  crypto: {
      ops: {
          randomBytes
      },
  },
})

const clientStream = client.connection.newStream();
const writer = clientStream.writable.getWriter()
await writer.write(BUFFER);
await writer.close();

What QUIC stack is Solana TPU using? This library is based on Cloudflare's quic and it follows a particular bootstrapping process. Best way to debug this it also build and run the Solana TPU side, so you can see what why it is closing with 306 error.

I don't have enough in-depth understanding of the TPU server, yet.
So I just assumed that quic communication would be rather universally?

Running TPU the server unfortunately is above my possibilities for the moment, but I that wrapped in try/catch block which truncated the full error message, which is:

ErrorQUICConnectionPeerTLS: Peer closed with transport code 306
    at constructor_.send [...]/node_modules/@matrixai/quic/src/QUICConnection.ts:947:23)
    at constructor_.send ([...]/node_modules/@matrixai/async-init/src/StartStop.ts:174:20)
    at [...]/node_modules/@matrixai/quic/src/QUICConnection.ts:833:18
    at [...]/node_modules/@matrixai/async-locks/src/Lock.ts:57:63
    at withF ([...]/node_modules/@matrixai/resources/src/utils.ts:24:18)
    at async constructor_.recv ([...]/node_modules/@matrixai/quic/src/QUICConnection.ts:749:5)
    at async Socket.handleSocketMessage ([...]/node_modules/@matrixai/quic/src/QUICSocket.ts:119:7) {
  data: {
    isApp: false,
    errorCode: 306,
    reason: Uint8Array(50) [
      114, 101,  99, 101, 105, 118, 101, 100,  32,
       99, 111, 114, 114, 117, 112, 116,  32, 109,
      101, 115, 115,  97, 103, 101,  32, 111, 102,
       32, 116, 121, 112, 101,  32,  73, 110, 118,
       97, 108, 105, 100,  83, 101, 114, 118, 101,
      114,  78,  97, 109, 101
    ]
  },
  cause: undefined,
  timestamp: 2024-04-09T07:03:42.048Z
}

Cheers

[tegefaulkes]

The reason message is received corrupt message of type InvalidServerName. Maybe the server is expecting a client certificate?

You can provide a key and certificate as part of the QUICConfig when starting the client.

/**
   * Private key as a PEM string or Uint8Array buffer containing PEM formatted
   * key. You can pass multiple keys. The number of keys must match the number
   * of certs. Each key must be associated to the the corresponding cert chain.
   *
   * Currently multiple key and certificate chains is not supported.
   */
  key?: string | Array<string> | Uint8Array | Array<Uint8Array>;

  /**
   * X.509 certificate chain in PEM format or Uint8Array buffer containing
   * PEM formatted certificate chain. Each string or Uint8Array is a
   * certificate chain in subject to issuer order. Multiple certificate chains
   * can be passed. The number of certificate chains must match the number of
   * keys. Each certificate chain must be associated to the corresponding key.
   *
   * Currently multiple key and certificate chains is not supported.
   */
  cert?: string | Array<string> | Uint8Array | Array<Uint8Array>;

Look at a QUICServer example for how to do this.

[gherkins]

You can provide a key and certificate as part of the QUICConfig when starting the client.

Unfortunately that does not change anything, connection seems to be established, then fails.

const tlsConfig = await generateTLSConfig('RSA');
const client = await QUICClient.createQUICClient({
        host: tpu_address.split(':')[0],
        port: parseInt(tpu_address.split(':')[1]),
        config: {
            key: tlsConfig.leafKeyPairPEM.privateKey,
            cert: tlsConfig.leafCertPEM,
        },
        crypto: {
            ops: {
                randomBytes
            },
        },
    }
)

INFO:QUICClient:Create QUICClient to 141.98.216.83:8009
INFO:QUICSocket:Start QUICSocket on [::]:0
INFO:QUICSocket:Started QUICSocket on [::]:64177
INFO:QUICConnection d13269a7a249aac1a6efc3f2a44e6e433491c161:Connect QUICConnection
INFO:QUICConnection d13269a7a249aac1a6efc3f2a44e6e433491c161:Start QUICConnection
INFO:QUICClient:ErrorQUICConnectionPeerTLS: QUIC Connection local TLS error - Peer closed with transport code 306
INFO:QUICConnection d13269a7a249aac1a6efc3f2a44e6e433491c161:ErrorQUICConnectionPeerTLS: QUIC Connection local TLS error - Peer closed with transport code 306
INFO:QUICConnection d13269a7a249aac1a6efc3f2a44e6e433491c161:ErrorQUICConnectionPeerTLS: QUIC Connection local TLS error - Peer closed with transport code 306
INFO:QUICSocket:Stop QUICSocket on [::]:64177
INFO:QUICSocket:Stopped QUICSocket on [::]:64177
INFO:QUICClient:Destroy QUICClient
INFO:QUICClient:Destroyed QUICClient
ErrorQUICConnectionPeerTLS: Peer closed with transport code 306
    at constructor_.send ([...]/node_modules/@matrixai/quic/src/QUICConnection.ts:947:23)
    at constructor_.send ([...]/node_modules/@matrixai/async-init/src/StartStop.ts:174:20)
    at [...]/node_modules/@matrixai/quic/src/QUICConnection.ts:833:18
    at [...]/node_modules/@matrixai/async-locks/src/Lock.ts:57:63
    at withF ([...]/node_modules/@matrixai/resources/src/utils.ts:24:18)
    at async constructor_.recv ([...]/node_modules/@matrixai/quic/src/QUICConnection.ts:749:5)
    at async Socket.handleSocketMessage ([...]/node_modules/@matrixai/quic/src/QUICSocket.ts:119:7) {
  data: {
    isApp: false,
    errorCode: 306,
    reason: Uint8Array(50) [
      114, 101,  99, 101, 105, 118, 101, 100,  32,
       99, 111, 114, 114, 117, 112, 116,  32, 109,
      101, 115, 115,  97, 103, 101,  32, 111, 102,
       32, 116, 121, 112, 101,  32,  73, 110, 118,
       97, 108, 105, 100,  83, 101, 114, 118, 101,
      114,  78,  97, 109, 101
    ]
  },
  cause: undefined,
  timestamp: 2024-04-09T07:54:08.325Z
}

[CMCDragonkai]

I mean you have to provide the appropriate client certificate - not just any certificate.

[tegefaulkes]

Based on what I can tell, this isn't a problem with the protocol. With QUIC the connection is established before TLS handshaking completes. So if the server rejects the TLS for whatever reason the client will 'establish' and then close with a code and message like you demonstrated. In this case it's a 306 error indicating DecodeError with the message received corrupt message of type InvalidServerName. This means that the server is rejecting the connection because of some requirement it has about the server name.

So it's important to note, whatever the problem is, it's the server taking issue with the connection, likely due to the certificate. I can't really find any information about how it uses QUIC while doing some quick research so I can't really comment on what exactly it could be.

[gherkins]

Thank you very much for your help!
I'm also struggling to find some information about what would be an appropriate client certificate in that context.
I thought providing any certificate might change the error message, but that didn't do much.

I will post an update here, if I find some solution.
Closing for now - thanks again!

[tegefaulkes]

Happy to help.

[CMCDragonkai]

@gherkins the appropriate client certificate would depend on your target Solana TPU node expects. TLS certificates are supposed to be signed by an authority. Perhaps there is a an authority that you need to get your certificate signed by in order to the target server to accept your credentials? This is a core part of MTLS connections - and represents a sort of end to end identity check. Client checks server identity, server checks client identity. You'd have to ask about this wherever Solana TPUs exist.

[gherkins]

hm, seems like in the rust implementation it's done via x509 certificates 🤔

https://github.com/solana-labs/solana/blob/27eff8408b7223bb3c4ab70523f8a8dca3ca6645/quic-client/tests/quic_client.rs#L285

The TPU node is just one of those retrieved via connection.getClusterNodes()
ContactInfo has an undocumented (in the ts library) property tpuQuic, which is just HOST-IP:PORT

https://github.com/solana-labs/solana-web3.js/blob/a0aa8a6f9fcc237c8b014cea7bd9d9616da608f9/packages/library-legacy/src/connection.ts#L705

I guess I'll try a self signed x509 certificate next and see if this changes the error message 🤷‍♂️

[CMCDragonkai]

There may actually be a signed authority required for all client certificates, best to ask someone on the solana team if that's required. These TPU nodes are not "public" nodes are they? (As in intended for just anybody to connect to).