Mac: My cluster isn't trusted

[garyanaplan]

kube-rs: 0.17.1
I'm trying to interact with a GCP hosted cluster. The cluster certificate is self-signed. When I start my application I see errors like this:

Error: Error { inner: Error(Hyper(Error(Connect, Custom { kind: Other, error: Error { code: -67843, message: "The certificate was not trusted." } })), "https://<redacted cluster IP>/api/v1/persistentvolumes?")

Error executing request }

If I update my client OS and tell it to trust the certificate then the problem disappears, so I guess the problem is related to the library not realising that it needs to process the cluster certificate somehow. I had a trawl around in the source, but couldn't see anything obviously wrong. There seemed to be some calls to add_root_certificate, but I wasn't sure if they were being called or if I needed to configure my client somehow or...?

Wish I could file something more useful, but maybe that's enough detail for someone to point me towards a solution.

(BTW: I can't employ my certificate work-around in real life, that was just to help understand the problem.)

[garyanaplan]

Some proper digging around led me to a solution that "works". If I comment out my convenient call to load_kube_config(), then I can build up a Configuration as follows:

// let cfg = config::incluster_config().or_else(|_| config::load_kube_config())?;
    let (builder, loader) = config::create_client_builder(config::ConfigOptions::default())?;
    let mut buf = Vec::new();
    File::open("cluster.crt")?.read_to_end(&mut buf)?;
    let cert = Certificate::from_pem(&buf)?;
    let cfg = config::Configuration::new(
        loader.cluster.server,
        builder.danger_accept_invalid_certs(true).build()?

and then proceed from there. It's nasty but I can work with it for now.

[clux]

Oh, interesting. There's presumably a flag in your ~/.kube/config for accepting invalid certs as well, right?

I think that flag, if it exists, should map to the one exposed by reqwest, but it does not seem to be the case atm 🤔

[garyanaplan]

No (although thank you for the tip about insecure-skip-tls-verify: true). I think (and I'm not an expert in this) that it works by distributing certificate authority data. My kubernetes config looks something like:

- cluster:
    certificate-authority-data: <cluster CA>
    server: https://<cluster ip>
  name: <cluster name>

My understanding is that kubectl makes use of the certificate-authority-data (similar to curl --cacert) which is a base64 encoded PEM to authenticate secured conversations to the server. This appears to be the bit that's missing.
I tried some more experimenting and used reqwest to add_root_certificate() a copy of certificate-authority-data, but I still get the untrusted error.

[clux]

Okay. It sounds like this is incorrectly wired up at the very least. Sorry, I'm not really able to help at the moment.

If you are able to help diagnose / find a fix for it, I'm happy to take PRs for it. I'm not really in possession of a cluster atm to test this out properly.

[garyanaplan]

No problem. I thought I'd bring it to the attention of a wider audience in case a known solution was out there. It looks like it may be a problem in reqwest, since even adding the root certificate manually to build the connection didn't work.
I'll do more investigating when I have time. If I can figure out a fix, I'll file a PR here (or with reqwest). Either way, I'll aim to close the issue by end of December.

[davidB]

I investigated about this issue, because 2 persons report this issue to my kubectl plugin:

• Error trying to connect: The certificate was not trusted
• Doesn't work with (some) self signed certs

I setup a minimal cluster GKE, and a small reqwest app (I'll shutdown the cluster in few day, if you want to try) : GKE access failed via reqwest

What I found :

• The issue is with at least GKE (Google) and EKS (AWS)

• The issue doesn't exist with GKE cluster setup few month ago

• The issue is only for client on mac OSX (Catalina) and not Linux

• The issue is with features native-tls and rusttls (restored 2 days ago on reqwest)

• The issue doesn't exist with kubectl, golang, curl

• The issue also exist on Safari (after register + trust the root certificate on OS keychain)

  

I guess the cause is :

• native-tls & rusttls feature of reqwest seems to delegate part of the certifact check to mac OS security-framework (via hyper-tls)

• curl & golang doesn't delegate to OS

• On mac OS X Catalina, some additional rules are applied Requirements for trusted certificates in iOS 13 and macOS 10.15 - Apple Support. At least one of the rules failed

  Additionally, all TLS server certificates issued after July 1, 2019 (as indicated in the NotBefore field of the certificate) must follow these guidelines:
  ...

  • TLS server certificates must have a validity period of 825 days or fewer (as expressed in the NotBefore and NotAfter fields of the certificate).

  because GKE generate too long certifact for API server: 5 years (see the screenshot) > 825 days

I see few (bad) work around (but no quick solution):

• disable ssl verification of mac OSX ( => lost security)
• request user to use certificate compliant with Catalina's policies ( => loose of user, I don't know how to do it on GKE)
• search how to not use security-framework with hyper-tls
• Other ideas ?

I can work on a PR if we agree about a solution/ work around (working on TLS is out of my skill).

EDIT: I modify my comment (above), because I'm not sur if rusttls use security-framework or if it's a side effect of reqwest, hyper-tls or ???

[davidB]

compementary info extracted from: "Certificate not standards compliant" on macOS Catalina, iOS 13 · Issue #174 · FiloSottile/mkcert

825 comes from the CAB forum bylaws for certificates. All of the major products and CA's follow them.

• https://www.sslshopper.com/cab-forum-reduces-max-cert-validity-to-825-days.html
• https://cabforum.org/2017/03/17/ballot-193-825-day-certificate-lifetimes

[davidB]

An other alternatives: use curl (via its crate) instead of reqwest:

--- with curl ---
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/\"","reason":"Forbidden","details":{},"code":403}
--- with reqwest ---
Error: reqwest::Error { kind: Request, url: "https://35.232.6.83/", source: hyper::Error(Connect, Error { code: -67843, message: "The certificate was not trusted." }) }

with a code like

// curl -i -v https://35.232.6.83 --cacert ./cert3.x509.crt
async fn main_with_curl() -> Result<(), Box<dyn std::error::Error>> {
    use curl::easy::Easy;
    use std::io::{stdout, Write};

    // Write the contents of rust-lang.org to stdout
    let mut easy = Easy::new();
    let cacert = std::path::Path::new("ca.x509.crt");
    std::fs::write(cacert, CACERT_PEM)?;
    easy.cainfo(cacert)?;
    easy.url(SERVERAPI_URL)?;
    easy.write_function(|data| {
        stdout().write_all(data).unwrap();
        Ok(data.len())
    })?;
    easy.perform()?;
    Ok(())
}

[clux]

Yeah, we've run into quite a few catalina issues in general, didn't realize this bug was related to that. Appreciate all the digging you've done here @davidB - lots of really helpful info. Have merged your temporary hack, and will try to make a release with it.