net/http: add docs for getting TLS certs with x/crypto/acme

[bradfitz]

Thanks to @crhym3, we now have a suitably-licensed & CLA-clean ACME implementation in https://godoc.org/golang.org/x/crypto/acme (and a high-level package in https://godoc.org/golang.org/x/crypto/acme/autocert).

I'd like to consider privately vendoring that into Go 1.8 and making HTTPS even easier.

I'd like a complete user program with automatic HTTPS certs to look something like:

package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    srv := &http.Server{
        Addr:         "example.com:443",
        ACMEServer:   http.LetsEncrypt, // non-empty enables autocert support
        ACMEAgreeTOS: func(tosURL string) bool { return true },
        ACMEEmail:    "foo@bar.com", // (but optional)
    }
    log.Fatal(srv.ListenAndServeTLS("", ""))
}

Misc notes:

• The ACMEFoo names are placeholders. Maybe they'd deserve their own struct.
• ACMEServer would be required as the opt-in, and we wouldn't make LetsEncrypt be automatic or preferred, but we would add a constant const LetsEncrypt = "https://acme-v01.api.letsencrypt.org/directory" like the acme package has.
• ACMEAgreeTOS would be required for now. The ACME protocol requires a TOS agreement because the CAB Forum requires cert issuers to have a legal relationship with the people getting certs or something. It's mostly a formality, but we shouldn't make it automatically say "yes" either, even though I don't think LetsEncrypt themselves care. Maybe we could export the https://godoc.org/golang.org/x/crypto/acme#AcceptTOS func in the net/http package to reduce the boilerplate.
• ACMEEmail is optional. If provided, your ACME cert provider can keep you updated on problems or changes.
• the default cache directory would be automatic. We could provide a string to let people pick an alternate directory. If you want to do something more complicated (e.g. cache coordination over a cluster), then you can just import the golang.org/x/crypto/acme/autocert package yourself. This is analogous to the HTTP/2 situation where common HTTP/2 is provided automatically, but weird uses require importing the guts.
• srv.ListenAndServeTLS("", "") is already valid for the past few releases, since 6a208ef for net/http: Server.ListenAndServeTLS not compatible with tls.Config.GetCertificate #14268 requested by @willchan specifically for LetsEncrypt stuff. It's a little ugly but works. Maybe we could provide instead a new method or option which also listens on an cleartext port 80 and redirects HTTP to HTTPS, optionally with a HSTS header.

My goal is for HTTPS to be dead simple out of the box.

Thoughts?

/cc @adg @broady @campoy @quentinmit @rsc @robpike @ianlancetaylor @mholt @crhym3

[mholt]

Like what I'm seeing so far!

I wonder if the ACME configuration should be in a separate struct value -- do we want to tether the http.Server type to ACME concretely?

One of the requests we've had in Caddy is to abstract the way certificates are Obtain()ed and Renew()ed -- in other words, an interface with approximately these two methods. An ACME client would be one implementation, a hashicorp/vault implementation might be another, etc. So I wonder if, instead, there should be some sort of interface type (TLSManager? Not sure what you'd call it), which some type tls.ACMEClient implements (or something like that).

In fact, thinking on these lines, it might be beneficial to drop down to the tls package somehow when working with ACME. In other words, you create a TLS listener that has some notion of ACME, but the HTTP server doesn't care. It's just one application of TLS and uses that listener and goes about its usual business.

FWIW, In Caddy I found myself writing essentially a wrapper type over tls.Config that specifies all the ACME stuff and makes the GetCertificate callback, etc. It all happens within TLS (not just HTTPS), so how about an easy way to make an ACME-capable TLS listener that you pass into http.Serve()?

[nhooyr]

I've been moving all of my Go servers to use ACME (with letsencrypt) and I've been using Russ's letsencrypt package for now because it is so simple and easy. However, I've also contributed to the acme package that @crhym3 wrote because its design was inspired by Russ's package and because I wanted ACME to be in the stdlib. I think it is a very common use case that we should cater towards.

However, I don't like the use of fields in http.Server to configure ACME. It's very easy to just create a tls.Listener using acme.Manager's GetCertificate callback. However for convenience, it just makes more sense for this to be part of crypto/tls rather than purely net/http so that even generic TLS servers can use ACME more easily.

(as I was writing this, I noticed @mholt's comment which I 👍)

[rolandshoemaker]

This is super cool!

With my Let's Encrypt hat on: One important thing to consider though is that currently neither the ACME spec nor the API implementation provided by Let's Encrypt should be considered completely stable. The Let's Encrypt API currently implements a amalgam of the four ACME RFC drafts which we colloquially refer to as v01. Once the RFC is finalized we intend to implement a new API version, v02, which will be a strict implementation of the final specification language which will not be completely backwards compatible with v01.

Once v02 is made public (we are currently aiming for somewhere around the start of 2017) we will provide a timetable for depreciating the v01 implementation . Given this might it make sense to wait until the finalized stable API is available before adding this support to the stdlib in order to prevent churn and breakage in users who don't update?

[tmornini]

My goal is for HTTPS to be dead simple out of the box

👍🏻

[bradfitz]

Once v02 is made public (we are currently aiming for somewhere around the start of 2017) we will provide a timetable for depreciating the v01 implementation . Given this might it make sense to wait until the finalized stable API is available before adding this support to the stdlib in order to prevent churn and breakage in users who don't update?

I totally agree. But Go 1.8 isn't due until 2017 anyway. The timing might work out? But even Go 1.9 is acceptable, if that works out best timing-wse.

[jsha]

Another Let's Encrypt developer here. Wrote out a post saying what @rolandshoemaker said about upcoming ACME changes, but he beat me to it!

Also, ideally the design should encourage disk-based caching as much as possible. If Go programs are designed to issue on each startup, people will very quickly run into Let's Encrypt rate limits. I see that autocert already has a Cache, so it's probably enough to include the Cache mechanism in the library and make sure the examples use it.

Lastly: This is great, and I'm super happy to see it. This is exactly the type of thing Let's Encrypt exists to enable. Thanks!

[bradfitz]

@jsha, as I said in my first post, caching would automatic. There wouldn't even be an option to disable it. Only an option to change the directory it uses.

[jsha]

Got it! I read too quickly. :-)

[dlsniper]

As awesome as this sounds, I think this shouldn't be included in the standard library.

I understand that people will say: it's about security, or other arguments, which, yes, are valid.

But please consider this:

• the ACME standards are in in Draft right now
• the functionality exists outside of the standard library just fine

Considering proposals to include different useful things in the language or standard library have been shut down of over the course of years, I would like to know if this means a change in vision on how the standard library / language should evolve from here on out. Does it mean that I can raise a proposal for having ? : syntax in the language? It is super useful in so many cases but it has been deemed not necessary as we can do if / else.

[bradfitz]

@mholt, @nhooyr, as I wrote:

The ACMEFoo names are placeholders. Maybe they'd deserve their own struct.

I agree they're probably too stuttery. Being a listener is interesting, though. Maybe:

package main

import (
    "crypto/tls"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    var srv http.Server
    log.Fatal(srv.Serve(&tls.ACMEListener{
        Domain:   "foo.com",
        Provider: tls.LetsEncrypt,
        AgreeTOS: tls.YesIAgreeTOS,
    }))
}

[nhooyr]

@bradfitz I like that. But now, I'm not sure if crypto/tls is the best place for such a listener. Perhaps we should add acme.Listener?

[bradfitz]

@dlsniper, the timing with regards to its draft status is discussed above. See comments from @rolandshoemaker and @jsha.

the functionality exists outside of the standard library just fine

Yes. And maybe all we do here is add documentation examples on how to use autocert. Maybe that's enough. I'd like it to be even easier, though.

Does it mean that I can raise a proposal for having ? : syntax in the language?

You can propose anything you like, but that discussion happened a long time ago and was already decided. If you insist on raising it again, please do it elsewhere and not in this issue.

[bradfitz]

@nhooyr, I don't really want to add a new package, though.

We already have https://golang.org/pkg/net/http/httputil/ though. Maybe that's a suitable location.