Support config files and temporary credentials for S3 URLs

[jmarshall]

Handles temporary credentials specified in the URL (as s3://ID:SECRET:TOKEN@bucket/...), via $AWS_SESSION_TOKEN, or as specified in a configuration file.

Reads credentials from the URL, via the usual environment variables, from ~/.aws/credentials, from ~/.s3cfg, or from ~/.awssecret.

[jmarshall]

@DonFreed: What is the usual ballpark lifetime of temporary session tokens? If they're especially short, it seems like a likely setup would be to store your id and secret in .aws/credentials but have the token more easily updated in $AWS_SESSION_TOKEN.

However none of the AWS SDKs or botocore code I looked at appeared to support mixing-'n'-matching like this, so I didn't either. Seems like they all support everything-in-env-vars or everything-in-config-file but not mixtures. Probably simpler that way…

[obenshaindw]

@jmarshall and @DonFreed: the lifetime of a session token can be controlled but Don's cited use case for this (National Database for Autism Research) have tokens lasting ~ 24 hours. You either have a long-term key pair (i.e., key, secret key) or temporary credentials (key, secret key, and token) and the two can't be mixed.

Both s3cmd and aws command-line use boto and provide support for using multiple config files or profiles, respectively. This enables a user to use a set of keys (i.e., their own AWS credentials) for some things and use a different set of credentials for other purposes. I suppose htslib will just look in the default config/profile, respectively?

This may be a corner case for htslib, but I thought it might be worth mentioning:
In addition to the use case where a session token is provided as part of a Federated access model, probably the more common use case is for using IAM roles in combination with EC2 instances. Briefly, this makes temporary credentials available to the instance at a local IP address obviating the need to ever put long term security credentials on an instance. s3cmd supports this method of key delivery and will 'look' for the availability of these keys out of the box.

[DonFreed]

@jmarshall and @obenshaindw: Sorry I missed the earlier message. Temporary tokens lifetimes range from 15 minutes to 36 hours. David is correct that the long-term key pair cannot be mixed with the temporary credentials. When you receive a token, you also receive an id and secret that must be used with that token.

In practice, for short sessions we just request a token and update ~/.aws/credentials. For longer sessions, we track expiration times or handle token expiration programmatically, so it does not matter too much whether the credentials are stored as environmental variables or in ~/.aws/credentials, so long as credentials are not mixed.

The precedence of credentials for the AWS Command Line Interface are:

• command line options
• environmental variables
• $AWS_SHARED_CREDENTIALS_FILE (defaults to ~/.aws/credentials if unset)
• $AWS_CONFIG_FILE (defaults to ~/.aws/config if unset)

To improve consistency with the aws CLI, I would be in favor of handling $AWS_CONFIG_FILE in a similar manner to $AWS_SHARED_CREDENTIALS.

[jmarshall]

When you receive a token, you also receive an id and secret that must be used with that token.

That is the facet I was unaware of. So not mixing environment variables with config files does indeed make sense. (The need for a separate token in the API when you're getting a new id & secret anyway… not so much!)

What's implemented in HTSlib at present follows other AWS SDKs and tools, and does support profiles. This will need documenting soon (probably mostly just a web page saying it supports s3: URLs with the usual configuration vars and files …) but in summary:

• Credentials in the usual username/password part of the URL: s3://id:secret[:token]@bucket/…; or use s3://profile@bucket/… to specify a profile
• Otherwise the usual environment variables ($AWS_ACCESS_KEY_ID, $AWS_SECRET_ACCESS_KEY, $AWS_SESSION_TOKEN); or $AWS_DEFAULT_PROFILE or $AWS_PROFILE to specify a profile
• Otherwise ~/.aws/credentials (or rather $AWS_SHARED_CREDENTIALS_FILE if set), selecting the profile as set in the URL or environment variable, or otherwise default
• Otherwise ~/.s3cfg, selecting the profile similarly
• Otherwise ~/.awssecret

@obenshaindw I see botocore also falls back to contacting the IAM well-known endpoint. HTSlib could also do this, although I would have thought there would be some common script for copying these into environment variables so that such code was not multiplied across clients and SDKs…?

HTSlib-based tools (i.e. samtools) are used on EC2 and especially outside EC2. If you're not on EC2, contacting random link-local IP addresses and asking them for credentials would be rather unfortunate (to put it mildly), so I'd really rather not do this by default. I would have thought there would be a convention of only doing this when $AWS_PROFILE is set to a special :IAM value or something like that?

@DonFreed I followed this AWS blog post, which seems to say that only the AWS CLI looks at ~/.aws/config while all SDKs and the CLI look at the newer common ~/.aws/credentials. As we're just another SDK that's not the CLI, it seemed appropriate not to look at the CLI's config file and if your credentials are in the credentials file there's nothing we would need from the config file anyway.

OTOH we're pandering to the CLI's $AWS_DEFAULT_PROFILE variable and to various legacy config files, and it would only be a few more lines of code… (but also more stuff to document).