[Bug] ConnectionReset trying to connect to Temporal Server

[andrepastore]

What are you really trying to do?

Using Rust, trying to connect the client to the server using tls_cert and the connect command as the exemple below

    let mut client_options_builder = ClientOptionsBuilder::default();
    client_options_builder
        .target_url(target_url)
        .client_name("name")
        .client_version("version");`

    if client_tls_config.is_some() {
        client_options_builder.tls_cfg(TlsConfig {
            server_root_ca_cert: None,
            domain: None,
            client_tls_config,
        });
    }

    let client_options = client_options_builder
        .build()
        .map_err(|err| TemporalError::ClientError(err.to_string()))?;

    let client = client_options
        .connect(namespace, meter, None)
        .await
        .map_err(|err| TemporalError::ClientError(err.to_string()))?;

Describe the bug

before #698 the code above connects correctly to the server and we can begin to creating workers, after the merge the same code returns the following error

Error: ClientError("'get_system_info call' error after connection: Status { code: Unknown, message: \"transport error\", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Kind(ConnectionReset)))) }")

Minimal Reproduction

In RUST, using any rev after the following:

temporal-client = { git = "https://github.com/temporalio/sdk-core", rev = "c55ba57" }
temporal-sdk = { git = "https://github.com/temporalio/sdk-core", rev = "c55ba57" }
temporal-sdk-core = { git = "https://github.com/temporalio/sdk-core", rev = "c55ba57" }
temporal-sdk-core-api = { git = "https://github.com/temporalio/sdk-core", rev = "c55ba57" }
temporal-sdk-core-protos = { git = "https://github.com/temporalio/sdk-core", rev = "c55ba57" }
temporal-sdk-core-test-utils = { git = "https://github.com/temporalio/sdk-core", rev = "c55ba57" }

and trying to connect a client to the server as of the exemple above

Environment/Versions

• OS and processor: [M1 Mac, Linux]
• Temporal Version: Rust SDK
• The error runs running locally as well as inside a container in k8s

Additional context

the cargo.toml that gives me the error
cargo.toml.txt

[Sushisource]

@andrepastore Hmm, weird. This definitely isn't some fundamental bug as this code./tonic version is running now in our other SDKs and in our tests etc just fine.

Are you using a self hosted server? Cloud? Is there some proxy in the way?

[andrepastore]

@Sushisource we are using temporal cloud, with TLS certificates for auth, there are no proxies in the environment

[happylinks]

@andrepastore did you figure this out? Running into the same issue.

[Sushisource]

@andrepastore Sorry this fell off my rader. @happylinks Are you using the Rust SDK as well? Tonic was upgraded relatively recently here #782 and I also made a change that should read system certs by default here #785

Those both might affect whatever is going on here.

[happylinks]

@Sushisource no problem! to be honest I'm using a fork, because I'm also writing workflows in rust 😅 (I know, not officially supported yet). It is up to date with main though, so not entirely sure yet what's causing it.
I'm still doing more testing, will let you know what I find, so far it only happens when I'm using TLS certs, without it (locally) it works fine.

[happylinks]

Just based on your remark about system certs, this is in my dockerfile:

FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y libssl-dev ca-certificates && rm -rf /var/lib/apt/lists/*

Not sure if there's any other ssl-related things I would need to install.

[Sushisource]

Yeah, I think that would probably do it in terms of having default certs. I think it's got to be something system related, since we do have tests that are running core based sdks against cloud (and, obviously users in real life doing that all the time too), hard for me to say exactly what though.

[happylinks]

OK! I can reproduce it on my macbook now, without the fork, so basically the same as @andrepastore

Cargo.toml:

temporal-client = { git = "https://github.com/temporalio/sdk-core", branch = "master" }
temporal-sdk = { git = "https://github.com/temporalio/sdk-core", branch = "master" }
temporal-sdk-core = { git = "https://github.com/temporalio/sdk-core", branch = "master" }
temporal-sdk-core-api = { git = "https://github.com/temporalio/sdk-core", branch = "master" }
temporal-sdk-core-protos = { git = "https://github.com/temporalio/sdk-core", branch = "master" }

Log

Error: `get_system_info` call error after connection: Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Kind(ConnectionReset)))) }

Code:

pub async fn create_client() -> Result<RetryClient<Client>, anyhow::Error> {
    let env = types::envs::TellaEnv::from_env();

    let address = get_env_var("TEMPORAL_ADDRESS", "localhost:7233", &env);
    let namespace = get_env_var("TEMPORAL_NAMESPACE", "default", &env);

    let temporal_key =
        get_env_var_opt("TEMPORAL_KEY", &env).map(|v| v.replace("\\n", "\n"));
    let temporal_pem =
        get_env_var_opt("TEMPORAL_PEM", &env).map(|v| v.replace("\\n", "\n"));

    let tls_config = match (temporal_key, temporal_pem) {
        (Some(key), Some(pem)) => Some(TlsConfig {
            domain: None,
            server_root_ca_cert: None,
            client_tls_config: Some(ClientTlsConfig {
                client_cert: pem.into_bytes(),
                client_private_key: key.into_bytes(),
            }),
        }),
        _ => None,
    };

    let url = Url::from_str(&format!("http://{address}"))?;
    let server_options = match tls_config {
        Some(tls_config) => sdk_client_options(url).tls_cfg(tls_config).build(),
        None => sdk_client_options(url).build(),
    }?;

    let client = server_options.connect(namespace, None).await?;
    Ok(client)
}

[Sushisource]

Ah, ok but that's against local server - I assume it was configured to use TLS? How is that server configured and running?

[happylinks]

Nah, sorry, I mean local worker configured against Temporal Cloud.

[Sushisource]

Oh, yeah I read the env var getter and assumed the default was what you were hitting. I'll try this out at some point today and see what I get.

[happylinks]

Ah, I see how that's confusing, yeah I set the temporal cloud config via env vars. Thanks for checking, let me know if I can help test anything!

[andrepastore]

hey, I didn't check this in a while, I was going to try the newest version to see if the issue was solved, but apparently it still remains, let me know if I can help with anything =)

[Sushisource]

So I added a simple tls test to Core's CI that connects to cloud, which looks essentially identical to your repro setup and it works fine: #800

I couldn't get anything to repro locally either. That's sort of all expected of course since this works just fine in lots and lots of places.

Can you confirm that you can connect to cloud with the same setup properly with another SDK like Go or Java?