H3 with s2n-quic

[zR-JB]

Hi, sry for asking this here, but I am a bit lost. Is there any example on how to use h3 with s2n-quic as the quic layer? I found the internal crate in the aws/s2n-quic repo: s2n-quic-h3, but I am not sure how I would use this? I only found the quinn examples and was wondering if anyone could provide me with a simple s2n-quic example. And if I understand it correctly, this is the whole point of h3, that with a simple interface we could plug any quic implementation in there, or is quinn the only real supported quic layer? Thanks for any answer :)

Or is this s2n-quic-h3 crate not available, and I just need to copy it to my local code and then use it similar to this here:

https://github.com/aws/s2n-quic/blob/main/quic/s2n-quic-qns/src/server/h3.rs

[Ruben2424]

The builders of the client and server from h3 have a generic argument:

h3/h3/src/client/builder.rs

Lines 115 to 122 in 94cb13f

	pub async fn build<C, O, B>(
	&mut self,
	quic: C,
	) -> Result<(Connection<C, B>, SendRequest<O, B>), ConnectionError>
	where
	C: quic::Connection<B, OpenStreams = O>,
	O: quic::OpenStreams<B>,
	B: Buf,

h3/h3/src/server/builder.rs

Lines 126 to 129 in 94cb13f

	pub async fn build<C, B>(&self, conn: C) -> Result<Connection<C, B>, ConnectionError>
	where
	C: quic::Connection<B>,
	B: Buf,

They accept anything implementing the Connection trait and are used the same way.

h3/h3/src/quic.rs

Lines 121 to 145 in 94cb13f

	pub trait Connection<B: Buf>: OpenStreams<B> {
	/// The type produced by `poll_accept_recv()`
	type RecvStream: RecvStream;
	/// A producer of outgoing Unidirectional and Bidirectional streams.
	type OpenStreams: OpenStreams<B, SendStream = Self::SendStream, BidiStream = Self::BidiStream>;
	/// Accept an incoming unidirectional stream
	///
	/// Returning `None` implies the connection is closing or closed.
	fn poll_accept_recv(
	&mut self,
	cx: &mut task::Context<'_>,
	) -> Poll<Result<Self::RecvStream, ConnectionErrorIncoming>>;
	/// Accept an incoming bidirectional stream
	///
	/// Returning `None` implies the connection is closing or closed.
	fn poll_accept_bidi(
	&mut self,
	cx: &mut task::Context<'_>,
	) -> Poll<Result<Self::BidiStream, ConnectionErrorIncoming>>;
	/// Get an object to open outgoing streams.
	fn opener(&self) -> Self::OpenStreams;
	}

The h3-quinn crate implements the Connection trait for the quinn. s2n-quic-h3 implements the traits for s2n-quic.

Looking at the client example:
Here the quinn client is created:

h3/examples/client.rs

Lines 99 to 106 in 94cb13f

	let mut client_endpoint = h3_quinn::quinn::Endpoint::client("[::]:0".parse().unwrap())?;
	let client_config = quinn::ClientConfig::new(Arc::new(
	quinn::crypto::rustls::QuicClientConfig::try_from(tls_config)?,
	));
	client_endpoint.set_default_client_config(client_config);
	let conn = client_endpoint.connect(addr, auth.host())?.await?;

The h3-quinn wrapper type is created here:

h3/examples/client.rs

Line 115 in 94cb13f

let quinn_conn = h3_quinn::Connection::new(conn);

Here is the h3 Connection is created. This and the rest of the code is mostly the same for s2n-quic.

h3/examples/client.rs

Line 117 in 94cb13f

let (mut driver, mut send_request) = h3::client::new(quinn_conn).await?;

On how to construct the s2n-quic-h3 Connection implementing h3s traits i would look at their docs.

Since the s2n-quic-h3 crate is not published you could use a git dependency for example.
https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories
https://github.com/aws/s2n-quic/blob/8f510f05ebd904ebda097eaf704066b945c2175d/quic/s2n-quic-h3/Cargo.toml#L10

[zR-JB]

Thank you very much, that clears it up!

[zR-JB]

I have a follow up question @Ruben2424

I implemented now a minimal server example, and it works just fine and serves my html, and I can access it using curl with --http3-only and --insecure.

But for some reason is_h3_no_error in error/ConnectionError always throws a Remote error after curl closes the connection. And I do not really get it, because the ApplicationClose is 0, and I thought 0 was "EVERYTHING IS OKAY". Is this a bug of h3, my implementation, curls implementation or a quirk of the protocol?

https://datatracker.ietf.org/doc/html/rfc9114#name-http-3-error-codes

http3_s2n_quic_server: error chain: Remote(ApplicationClose(0x0))
http3_s2n_quic_server: Connection error: Remote error: ApplicationClose: 0x0

My Implementation:

use std::{path::PathBuf, sync::Arc};

use bytes::Bytes;
use h3::server::builder as h3_server_builder;
use h3::error::ConnectionError;
use http::{Request, Response, StatusCode};
use s2n_quic::Server;
use tokio::{fs, fs::File, io::AsyncReadExt};
use tracing::{error, info};

type H3Conn = h3::server::Connection<s2n_quic_h3::Connection, Bytes>;
type H3RequestStream = h3::server::RequestStream<
    <s2n_quic_h3::Connection as h3::quic::OpenStreams<Bytes>>::BidiStream,
    Bytes,
>;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    tracing_subscriber::fmt::init();

    let cert_pem = std::fs::read_to_string("certs/cert.pem")?;
    let key_pem = std::fs::read_to_string("certs/key.pem")?;

    let mut server = Server::builder()
        .with_tls(
            s2n_quic::provider::tls::default::Server::builder()
                .with_certificate(cert_pem, key_pem)?
                .build()?,
        )?
        .with_io("0.0.0.0:4433")?
        .start()?;

    let www_dir = Arc::new(PathBuf::from("www"));
    info!("Server listening on https://localhost:4433/");

    while let Some(conn) = server.accept().await {
        let www_dir = www_dir.clone();
        tokio::spawn(async move {
            let result = handle_connection(conn, www_dir).await;
            if let Err(e) = result {
                if let Some(h3err) = e.downcast_ref::<ConnectionError>() {
                    if h3err.is_h3_no_error() {
                        tracing::info!("Connection closed cleanly (H3_NO_ERROR)");
                        return;
                    }
                }
                for cause in e.chain() {
                tracing::info!("error chain: {cause:?}");
            }
            error!("Connection error: {e:?}");
            }
        });
    }

    Ok(())
}

async fn handle_connection(
    connection: s2n_quic::Connection,
    www_dir: Arc<PathBuf>,
) -> anyhow::Result<()> {
    use s2n_quic_h3::Connection as H3QuicConnection;

    let mut h3_conn: H3Conn = h3_server_builder()
        .max_field_section_size(16 * 1024)
        .send_grease(false)
        .build(H3QuicConnection::new(connection))
        .await?;

    while let Some(resolver) = h3_conn.accept().await? {
        let www_dir = www_dir.clone();
        tokio::spawn(async move {
            match resolver.resolve_request().await {
                Ok((req, stream)) => {
                    if let Err(e) = handle_request(req, stream, www_dir).await {
                        error!("Request error: {e:?}");
                    }
                }
                Err(e) => {
                    error!("Failed to resolve request: {e:?}");
                }
            }
        });
    }
    Ok(())
}

async fn handle_request(
    req: Request<()>,
    mut stream: H3RequestStream,
    www_dir: Arc<PathBuf>,
) -> anyhow::Result<()> {
    let method = req.method().clone();
    let path = sanitize_path(req.uri().path());
    let file_path = if path == "/" {
        www_dir.join("index.html")
    } else {
        www_dir.join(&path[1..])
    };

    info!("Request: {} {}", method, file_path.display());

    if !file_path.starts_with(&*www_dir) {
        send_response(
            &mut stream,
            StatusCode::FORBIDDEN,
            b"Forbidden\n".as_ref(),
            true,
        )
        .await?;
        return Ok(());
    }

    match fs::metadata(&file_path).await {
        Ok(metadata) if metadata.is_file() => {
            let file_size = metadata.len();
            let resp_builder = Response::builder()
                .status(StatusCode::OK)
                .header("content-length", file_size.to_string());
            let resp = resp_builder.body(())?;
            stream.send_response(resp).await?;
            info!("Sent headers for {}", file_path.display());

            if method != http::Method::HEAD {
                let mut file = File::open(&file_path).await?;
                let mut buf = [0u8; 8192];
                loop {
                    let n = file.read(&mut buf).await?;
                    if n == 0 {
                        break;
                    }
                    stream.send_data(Bytes::copy_from_slice(&buf[..n])).await?;
                }
                info!("Sent body for {}", file_path.display());
            }
            stream.finish().await?;
            info!("Finished stream for {}", file_path.display());
        }
        _ => {
            error!("File not found: {}", file_path.display());
            send_response(
                &mut stream,
                StatusCode::NOT_FOUND,
                b"Not Found\n".as_ref(),
                method == http::Method::HEAD,
            )
            .await?;
        }
    }
    Ok(())
}

async fn send_response(
    stream: &mut H3RequestStream,
    status: StatusCode,
    body: &[u8],
    is_head: bool,
) -> anyhow::Result<()> {
    let mut resp_builder = Response::builder().status(status);
    resp_builder = resp_builder.header("content-length", body.len().to_string());
    let resp = resp_builder.body(())?;
    stream.send_response(resp).await?;
    if !is_head && !body.is_empty() {
        stream.send_data(Bytes::copy_from_slice(body)).await?;
    }
    stream.finish().await?;
    Ok(())
}

fn sanitize_path(path: &str) -> String {
    let mut out = String::new();
    for part in path.split('/') {
        if part == ".." || part.contains('\\') {
            continue;
        }
        if !out.ends_with('/') && !out.is_empty() {
            out.push('/');
        }
        out.push_str(part);
    }
    if out.is_empty() {
        "/".to_string()
    } else {
        out
    }
}

Thanks if anyone takes a look at this and has time to answer :)

[Ruben2424]

The application code 0 is not defined in the http3 spec:
https://datatracker.ietf.org/doc/html/rfc9114#name-http-3-error-codes

This can come from curl, s2n-quic. Maybe even from h3 which would be a bug. Cant say without debugging. You could inspect the traffic with wireshark to see which code is sent by curl. Then you more.