Obtain S3 credentials via IAM roles

[jmarshall]

In addition to the use case where a session token is provided as part of a Federated access model, probably the more common use case is for using IAM roles in combination with EC2 instances. Briefly, this makes temporary credentials available to the instance at a local IP address obviating the need to ever put long term security credentials on an instance. s3cmd supports this method of key delivery and will 'look' for the availability of these keys out of the box.

[As noted by @obenshaindw on PR #303]

[jmarshall]

I see botocore also falls back to contacting the IAM well-known endpoint. HTSlib could also do this, although I would have thought there would be some common script for copying these into environment variables so that such code was not multiplied across clients and SDKs…? e.g. eval something like

baseurl='http://169.254.169.254/latest/meta-data/iam/security-credentials'
role=`curl $baseurl`
curl $baseurl/$role | jq -r "\"export AWS_ACCESS_KEY_ID='\(.AccessKeyId)' AWS_SECRET_ACCESS_KEY='\(.SecretAccessKey)' AWS_SESSION_TOKEN='\(.Token)'\""

HTSlib-based tools (i.e. samtools) are used on EC2 and especially outside EC2. If you're not on EC2, contacting random link-local IP addresses and asking them for credentials would be rather unfortunate (to put it mildly), so I'd really rather not do this by default. I would have thought there would be a convention of only doing this when $AWS_PROFILE is set to a special :IAM value or something like that?

[obenshaindw]

Any script that reads the instance metadata for security credentials and sets environment variables for those would need to be run on a schedule or immediately prior to running samtools (at least every hour according to the previously linked page AWS documetation because credentials are automatically rotated.

The advantage here is not having to leave any credentials on the instance, and if using an application that reads directly from instance metadata, not having to set anything environment-specific.

I agree that this would seem like a special case for samtools (i.e., a piece of code that is environment-specific). It makes sense for boto to have this functionality since it is an SDK for Amazon Web Services. As you say, if one is using samtools on EC2 they could write a script to read from the instance metadata endpoint and place the security credentials in a location where samtools will find them.

[delagoya]

The AWS C++ SDK instead of the current customer request signing code would solve this. But that is a big project and would likely be better done by pulling out the htsfile_s3.c plugin from the root source tree and into it's own contrib project.

In the meantime, I've built a Docker container that has S3 protocol support and also a wrapper script that queries the metadata and then calls samtools. This should work for both EC2 instances and ECS Tasks. The wrapper script contents are simply:

#!/bin/bash
B='http://169.254.169.254/latest/meta-data/iam/security-credentials/'
CREDS=''
if [ -n "${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}" ]; then
    CREDS=$(curl -s --connect-timeout 0.1 -s 169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI} |  jq -r "\"AWS_ACCESS_KEY_ID='\(.AccessKeyId)' AWS_SECRET_ACCESS_KEY='\(.SecretAccessKey)' AWS_SESSION_TOKEN='\(.Token)'\"")
elif [ -n "$(R=$(curl -s --connect-timeout 0.1 $B))" ]; then
    CREDS=$(curl -s --connect-timeout 0.1 $B/$R |  jq -r "\"AWS_ACCESS_KEY_ID='\(.AccessKeyId)' AWS_SECRET_ACCESS_KEY='\(.SecretAccessKey)' AWS_SESSION_TOKEN='\(.Token)'\"")
fi
eval ${CREDS} samtools-s3 $*

Docker container is at https://hub.docker.com/r/delagoya/samtools-ecs-s3/

I've tested it out on AWS Batch to pull the header from a private S3 BAM file object.

[hguturu]

Are there any updates to if S3 credentials via IAM roles will be considered? The workaround has trouble since the credentials are rotated so any jobs that run for more than a few minutes can suddenly fail in the middle.

[delagoya]

can you give an example of a failure? The code above should pull fresh credentials every time the docker container is executed.

[hguturu]

The curl command above curl -s --connect-timeout 0.1 $B/$R returns an object with a field such as "Expiration" : "2020-04-14T01:20:09Z". This expiration is a fixed time from when the credentials were last refreshed. So if the command is launched 10-15 minutes before the tokens expires and runs for more than 15 minutes, it breaks in the middle since its tokens have expired. I assume the SDK handles this by refreshing tokens as they expire. Its hard to reproduce since the command itself works fine, just the token refresh timing.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials seems to suggest new tokens are given 5 minutes in advance, so there is a tight window of failure cases, but the window gets longer when streaming large files from s3 (e.g. a 200Gb+ bam file).

[daviesrob]

Is the problem caused by trying to do lots of index look-ups, or by having a very long delay between generating the credentials and actually trying to open the file?

The first one isn't too easy to fix, as currently htslib only reads the credentials on S3 file open (although it does know how to re-sign the request if necessary). The second problem might not be so difficult. Instead of passing the credentials in environment variables, as shown above, you could store them in one of the configuration files supported by hfile_s3. Then you could run a program that occasionally requests a new token and writes it into the configuration file (ensuring you do this safely, i.e. write to a new file and then rename it on top of the old one). That way, when hfile_s3 tries to get the file, it should always have a token with a reasonable amount of lifetime left.

[hguturu]

Ya its unfortunate the first type of use case doesn't have an easy fix with this workaround. I think the use case also applied with larger streams where they can be cut off midstream. But I guess the only fix would be to update hfile_s3 to obtain those credentials so it knows when they are invalid and refresh or switch to the sdk which can handle credentials for you, either of those are bigger tasks, which is why I was wondering if there are any plans for that update?

[daviesrob]

Looking at this, I think the best solution would be to add the AWS JSON metadata format to the list of configuration files the hfile_s3 can use. It can then extract the expiration time and use it to renew the credentials if it needs to.

I'm not sure exactly when we can get it working, but we can certainly put it on out to-do list.

[hguturu]

It would be great if it gets added to the to-do list! Thanks!