Skip to content

Commit cd9879e

Browse files
yhakbarjosh-padnick
yhakbar
and
josh-padnick
authored
docs: Breaking down Pipelines authentication concepts (#2745)
* docs: Nested AWS into `Authenticating to the Cloud` * docs: Adding Azure docs * docs: Adding custom auth * Fix build issues. * Add custom page to sidebar * Update docs/2.0/docs/pipelines/concepts/cloud-auth/aws.mdx Co-authored-by: Josh Padnick <[email protected]> * Update docs/2.0/docs/pipelines/concepts/cloud-auth/aws.mdx Co-authored-by: Josh Padnick <[email protected]> * fix: Use active voice for custom auth * fix: Add examples of secret managers * fix: Explicitly say 'at the root of your repository' * fix: Add callout for risk of custom auth * fix: Shuffle order of tabs for configuration options * fix: Adding a bit of cleanup * fix: Adding preamble for best practices --------- Co-authored-by: Josh Padnick <[email protected]>
1 parent 43e7103 commit cd9879e

File tree

6 files changed

+497
-20
lines changed

6 files changed

+497
-20
lines changed

‎docs/2.0/docs/pipelines/concepts/cloud-auth.md‎ renamed to ‎docs/2.0/docs/pipelines/concepts/cloud-auth/aws.mdx‎

Lines changed: 111 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,108 @@
1-
# Authenticating to the Cloud
1+
# Authenticating to AWS
22

3-
## Authenticating to AWS accounts
3+
import Tabs from '@theme/Tabs';
4+
import TabItem from '@theme/TabItem';
45

5-
Pipelines automatically determines which AWS account to authenticate to based on the infrastructure changes proposed in your pull request.
6+
Pipelines automatically determines which AWS account(s) to authenticate with, and how to authenticate with them, based on the infrastructure changes proposed in your pull request.
67

7-
### How Pipelines authenticates to AWS
8+
## How Pipelines authenticates to AWS
89

9-
To execute the actions detected by Pipelines, each AWS account must assume an AWS IAM Role using Open ID Connect (OIDC). At a high level, OIDC works as follows: AWS recognizes GitHub or GitLab as an "identity provider," trusts GitHub’s or GitLab’s request to assume a temporary IAM Role, and then issues AWS credentials valid for the duration of the GitHub Actions or GitLab CI workflow.
10+
To execute the infrastructure changes detected by Pipelines, each AWS account must have an AWS IAM Role that Pipelines can assume using Open ID Connect (OIDC).
1011

11-
When creating a new AWS account, it is necessary to update the AWS OIDC configuration to include an IAM role that GitHub or GitLab can assume. When using the [Gruntwork Account Factory](/2.0/docs/accountfactory/architecture), this update is performed automatically during the process of [adding a new AWS account](/2.0/docs/accountfactory/guides/vend-aws-account).
12+
At a high level, OIDC works as follows: AWS recognizes GitHub or GitLab as an "identity provider," trusts GitHub's or GitLab's request to assume a temporary IAM Role, and then issues AWS credentials valid for the duration of the GitHub Actions or GitLab CI workflow.
1213

13-
### How Pipelines knows what AWS account to authenticate to
14+
When creating a new AWS account, it is necessary to update the AWS OIDC configuration to include an IAM role that GitHub or GitLab can assume. When using the [Gruntwork Account Factory](/2.0/docs/accountfactory/architecture), this update is performed automatically during the process of [vending a new AWS account](/2.0/docs/accountfactory/guides/vend-aws-account).
1415

15-
Pipelines assumes that each top-level directory in your `infrastructure-live` repository corresponds to a single AWS account, excluding the directory reserved for [module defaults](/2.0/docs/library/concepts/module-defaults). Each account-mapped directory must have an entry in the `accounts.yml` file. The entry should include a key matching the directory name and key/value pairs for the AWS account ID and the root user email address of the account.
16+
## How Pipelines knows what AWS principals to authenticate as
17+
18+
<Tabs>
19+
<TabItem value="hcl" label="HCL">
20+
21+
For HCL configurations, account mappings are defined using environments specified in HCL configuration files in the `.gruntwork` directory (you are using these if you see `.hcl` files in your `.gruntwork` directory).
22+
23+
Whenever Pipelines attempts to authenticate to AWS for a given unit, it will check to see if the unit matches any of the environments specified in your Pipelines HCL configurations. If any do, it will use the corresponding `authentication` block to determine how to authenticate to AWS.
24+
25+
For example, if you have the following environment configuration:
26+
27+
```hcl title=".gruntwork/environments.hcl"
28+
environment "my_cool_environment" {
29+
filter {
30+
paths = ["my-cool-account/*"]
31+
}
32+
33+
authentication {
34+
aws_oidc {
35+
account_id = "123456789012"
36+
plan_iam_role_arn = "arn:aws:iam::123456789012:role/pipelines-plan"
37+
apply_iam_role_arn = "arn:aws:iam::123456789012:role/pipelines-apply"
38+
}
39+
}
40+
}
41+
```
42+
43+
Pipelines will authenticate to AWS using the account with ID `123456789012` when a unit matches the filter `my-cool-account/*`. It will use the `pipelines-plan` role when pull requests are opened/updated, and the `pipelines-apply` role when pull requests are merged. The `pipelines-plan` role typically only has read permissions, while the `pipelines-apply` role typically has both read and write permissions.
44+
45+
Most customers prefer not to have to explicitly track the account IDs of AWS accounts in their configuration files. Instead, they prefer to leverage the `aws` block to parse an `accounts.yml` file that contains the relevant account metadata, then reference the accounts by name in their environment configurations.
46+
47+
For example, you could create an `accounts.yml` file like the following with your account definitions:
48+
49+
```yml title=accounts.yml
50+
"my-cool-account":
51+
"email": "[email protected]"
52+
"id": "123456789012"
53+
```
54+
55+
Then, create an `aws.hcl` file that references this `accounts.yml` file using the `aws` block:
56+
57+
```hcl title=".gruntwork/aws.hcl"
58+
aws {
59+
accounts "all" {
60+
path = "accounts.yml"
61+
}
62+
}
63+
```
64+
65+
You can then reference these accounts in your environment configurations:
66+
67+
```hcl title=".gruntwork/environments.hcl"
68+
environment "my_cool_environment" {
69+
filter {
70+
paths = ["my-cool-account/*"]
71+
}
72+
73+
authentication {
74+
aws_oidc {
75+
account_id = aws.accounts.all.my-cool-account.id
76+
plan_iam_role_arn = "arn:aws:iam::${aws.accounts.all.my-cool-account.id}:role/pipelines-plan"
77+
apply_iam_role_arn = "arn:aws:iam::${aws.accounts.all.my-cool-account.id}:role/pipelines-apply"
78+
}
79+
}
80+
}
81+
```
82+
83+
```bash title="Infrastructure Live"
84+
.
85+
├── .gruntwork/
86+
│ ├── aws.hcl
87+
│ └── environments.hcl
88+
├── accounts.yml
89+
├── my-cool-account
90+
│ └── us-east-1
91+
│ └── dev
92+
│ └── database
93+
│ └── terragrunt.hcl
94+
```
95+
96+
:::info
97+
The HCL configuration approach provides more flexibility for complex authentication scenarios and enables the use of [Configurations as Code](/2.0/reference/pipelines/configurations-as-code/) features.
98+
:::
99+
100+
</TabItem>
101+
<TabItem value="yaml" label="Legacy YAML Configuration" default>
102+
103+
For legacy YAML configurations (you are using these if you see a `.gruntwork/config.yml` file in your repository), account mappings are defined using a combination of the `accounts.yml` file at the root of your repository, and the names of top-level directories in your `infrastructure-live` repository.
104+
105+
Pipelines assumes that each top-level directory in your `infrastructure-live` repository corresponds to a single AWS account, excluding the directory reserved for [module defaults](/2.0/docs/library/concepts/module-defaults) (the one named `_envcommon`). Each account-mapped directory must have an entry in the account configuration. The entry should include a key matching the directory name and key/value pairs for the AWS account ID and the root user email address of the account.
16106

17107
For instance, the following `accounts.yml` entry maps to a directory named `my-cool-account` in your `infrastructure-live` repository:
18108

@@ -25,7 +115,7 @@ For instance, the following `accounts.yml` entry maps to a directory named `my-c
25115
```bash title="Infrastructure Live"
26116
.
27117
├── accounts.yml
28-
├── _module_defaults
118+
├── _envcommon
29119
│ └── services
30120
│ └── my-app.hcl
31121
├── my-cool-account
@@ -35,7 +125,10 @@ For instance, the following `accounts.yml` entry maps to a directory named `my-c
35125
│ └── terragrunt.hcl
36126
```
37127

38-
### AWS account authentication when creating new AWS accounts
128+
</TabItem>
129+
</Tabs>
130+
131+
## AWS account authentication when creating new AWS accounts
39132

40133
:::note
41134

@@ -52,7 +145,7 @@ Pipelines manages two main types of infrastructure-change events:
52145

53146
For the first type (add/change/delete Terragrunt files), Pipelines authenticates directly to the AWS account containing the affected resources. For the second type (creating new AWS accounts), Pipelines uses the Management Account.
54147

55-
#### Management account
148+
### Management account
56149

57150
Gruntwork's Account Factory is built on AWS Control Tower, which requires that new AWS accounts be created through the [Control Tower Management AWS Account](https://docs.aws.amazon.com/controltower/latest/userguide/how-control-tower-works.html#what-is-mgmt).
58151

@@ -66,15 +159,18 @@ The AWS IAM Role in the Management Account must have permissions to provision ne
66159

67160
:::
68161

69-
#### Child accounts
162+
### Child accounts
163+
164+
A child account in the context of Gruntwork Account Factory is an AWS account that is created by AWS Control Tower and is managed by Pipelines. It is a "child" account in that it is considered a child of the Management Account, and Pipelines will perform the initial baselining of the account by first assuming a role in the Management Account (the parent), then use that role to assume a different role in the child account.
70165

71166
Each child account (e.g., `dev`, `stage`, `prod`, etc.) contains an AWS IAM role that Pipelines can assume from GitHub Actions or GitLab CI using OIDC. This role is automatically provisioned during the [account baseline process](/2.0/docs/accountfactory/guides/vend-aws-account). Once the role is established in the child account, users can submit pull requests/merge requests to add, modify, or delete resources in that account.
72167

73168
When a pull request/merge request is created or synchronized, or when changes are pushed to the `main` branch, Pipelines detects the changes, maps them to the appropriate account, assumes the role in the child account, and executes a `terragrunt plan` (for pull requests/merge requests) or `terragrunt apply` (for pushes to `main`).
74169

75-
### Fundamentals of OIDC for Publicly Available and Private CI/CD platforms
170+
## Fundamentals of OIDC for Publicly Available and Private CI/CD platforms
76171

77172
### JWT Token Issuers
173+
78174
A JWT token is a base64-encoded JSON object that contains three parts: a header, a payload, and a signature. The header typically contains metadata about the token, such as the algorithm used to sign it. The payload contains the claims or assertions made by the issuer, such as the subject (user), audience (intended recipient), and expiration time. The signature is used to verify that the token was issued by a trusted authority and has not been tampered with.
79175

80176
Critically, the issuer is a URL that is both specified inside the token, and is used by consumers of the token to fetch the public key used to validate the signature of that same token. Assuming the public key is fetched via HTTPS, there is a valid trust chain that the token was in fact issued by the expected issuer and you have typical cryptographic guarantees it wasn't substituted or tampered with.
@@ -83,8 +179,8 @@ Typically the issuer is the hostname of the CI/CD platform, such as `https://git
83179

84180
If, however, your CI/CD platform is hosted privately, you will need to host the public key and OIDC configuration in a publicly accessible location, such as an S3 bucket, and update the issuer in your CI/CD configuration to point to that location. The diagrams below illustrate both approaches - fetching the keys directly from your CI/CD platform via a public route, or fetching the keys from a public S3 bucket.
85181

182+
### Publicly Available CI/CD Platforms
86183

87-
#### Publicly Available CI/CD Platforms
88184
```mermaid
89185
sequenceDiagram
90186
participant SCM as SCM (GitLab/GitHub etc.)
@@ -102,7 +198,7 @@ sequenceDiagram
102198
103199
```
104200

105-
#### Non-Publicly Available CI/CD Platforms
201+
### Non-Publicly Available CI/CD Platforms
106202

107203
This diagram follows the [recommended approach](https://docs.gitlab.com/ci/cloud_services/aws/#configure-a-non-public-gitlab-instance) from GitLab for private CI/CD platform instances. The guidance is to host the public key in a publicly accessible S3 bucket and update the issuer in the CI/CD configuration.
108204

‎docs/2.0/docs/pipelines/concepts/cloud-auth/azure.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Authenticating to Azure
2+
3+
Pipelines automatically determines which Azure subscription(s) to authenticate with, and how to authenticate with them, based on the infrastructure changes proposed in your pull request.
4+
5+
## How Pipelines authenticates to Azure
6+
7+
To execute the actions detected by Pipelines, each Azure subscription must have one or more Entra ID applications configured that Pipelines can authenticate with using Open ID Connect (OIDC).
8+
9+
At a high level, OIDC works as follows: Entra ID recognizes GitHub or GitLab as an "identity provider," trusts GitHub's or GitLab's request to authenticate with a specific Entra ID application, and then issues Azure credentials valid for the duration of the GitHub Actions or GitLab CI workflow.
10+
11+
When creating a new Azure subscription, it is necessary to configure Entra ID applications and federated identity credentials to enable GitHub or GitLab authentication via OIDC.
12+
13+
## How Pipelines knows what Azure principals to authenticate as
14+
15+
Azure federated identity mappings are defined using environments specified in HCL configuration files in the `.gruntwork` directory.
16+
17+
Whenever Pipelines attempts to authenticate to Azure for a given unit, it will check to see if the unit matches any of the environments specified in your Pipelines HCL configurations. If any do, it will use the corresponding `authentication` block to determine how to authenticate to Azure.
18+
19+
For example, if you have the following environment configuration:
20+
21+
```hcl title=".gruntwork/environments.hcl"
22+
environment "my_azure_subscription" {
23+
filter {
24+
paths = ["my-azure-subscription/*"]
25+
}
26+
27+
authentication {
28+
azure_oidc {
29+
tenant_id = "a-tenant-id"
30+
subscription_id = "a-subscription-id"
31+
plan_client_id = "plan-client-id"
32+
apply_client_id = "apply-client-id"
33+
}
34+
}
35+
}
36+
```
37+
38+
Pipelines will authenticate to Azure using the subscription with ID `a-subscription-id` within tenant `a-tenant-id` when the filepath of the unit matches the filter `my-azure-subscription/*`. It will use the `plan-client-id` application when pull requests are opened/updated, and the `apply-client-id` application when pull requests are merged. The plan application typically only has read permissions, while the apply application typically has both read and write permissions.
39+
40+
```bash title="Infrastructure Live"
41+
.
42+
├── .gruntwork/
43+
│ └── environments.hcl
44+
├── my-azure-subscription
45+
│ └── my-azure-resource-group
46+
│ └── database
47+
│ └── terragrunt.hcl
48+
```
49+
50+
:::info
51+
The HCL configuration approach provides flexibility for complex authentication scenarios and enables the use of [Configurations as Code](/2.0/reference/pipelines/configurations-as-code/) features.
52+
:::
53+
54+
## Azure subscription authentication workflow
55+
56+
Pipelines manages infrastructure changes by authenticating directly to the Azure subscription containing the affected resources using OIDC.
57+
58+
When a pull request is created or synchronized, or when changes are pushed to the `main` branch, Pipelines detects the changes, maps them to the appropriate Azure subscription, authenticates using the configured Entra ID application, and executes a Terragrunt plan (for pull requests) or apply (for pushes to `main`).
59+
60+
## Fundamentals of OIDC for Azure with GitHub Actions and GitLab CI
61+
62+
### Entra ID Federated Identity Credentials
63+
64+
Azure uses federated identity credentials to establish trust between external identity providers (like GitHub or GitLab) and Entra ID applications. This eliminates the need to store long-lived secrets in your CI/CD platform.
65+
66+
The federated identity credential configuration includes:
67+
68+
- **Issuer**: The identity provider URL (e.g., `https://token.actions.githubusercontent.com` for GitHub Actions)
69+
- **Subject identifier**: Specifies which repository, branch, or other criteria must match for the token to be accepted (e.g., `repo:my-org/my-repo:ref:refs/heads/main`)
70+
- **Audience**: The intended recipient of the token (typically `api://AzureADTokenExchange`)
71+
72+
### Publicly Available CI/CD Platforms
73+
74+
```mermaid
75+
sequenceDiagram
76+
participant SCM as SCM (GitLab/GitHub etc.)
77+
participant SCMPublicRoute as SCM Hostname e.g. github.com
78+
participant EntraID as Entra ID
79+
80+
SCM->>SCM: Generate a public/private key pair
81+
SCM->>SCM: Generate a JWT and sign with the private key
82+
SCM->>EntraID: Send JWT to Entra ID requesting an access token
83+
EntraID->>SCMPublicRoute: Fetch public key via HTTPS <br>(which validates that the SCM is who it says it is)
84+
SCMPublicRoute->>EntraID: Return the public key
85+
EntraID->>EntraID: Validate signature on JWT using public key to validate that it was generated by the Issuer
86+
EntraID->>EntraID: Inspect JWT Content and ensure it passes federated identity credential policies
87+
EntraID->>SCM: Return access token for the configured application
88+
```
89+
90+
### Non-Publicly Available CI/CD Platforms
91+
92+
For private CI/CD platform instances, you have a few options to enable OIDC with Azure:
93+
94+
1. **Host OIDC configuration publicly**: Similar to the AWS approach, you can host the OIDC configuration (`.well-known/openid-configuration`) and JWKS (JSON Web Key Set) in a publicly accessible location, such as an Azure Storage Account with static website hosting, and update the issuer in your CI/CD configuration.
95+
96+
2. **Configure firewall exceptions**: Update your application firewalls to specifically allow requests to the `.well-known/openid-configuration` endpoint and the JWKS endpoint from Entra ID.
97+
98+
The diagram below illustrates the first approach - hosting the public key and OIDC configuration in a publicly accessible Azure Storage Account:
99+
100+
```mermaid
101+
sequenceDiagram
102+
participant SCM as SCM (GitLab/GitHub etc.)
103+
participant SCMPublicRoute as Public Azure Storage (e.g. acme-public.z6.web.core.windows.net)
104+
participant EntraID as Entra ID
105+
106+
SCM->>SCM: Generate a public/private key pair
107+
SCM->>SCMPublicRoute: Publish public key and OIDC config to Azure Storage
108+
SCM->>EntraID: Update federated identity credential issuer to Azure Storage public URL
109+
SCM->>SCM: Update issuer to hostname of Azure Storage public URL
110+
SCM->>SCM: Generate a JWT with updated issuer and sign with the private key
111+
SCM->>EntraID: Send JWT to Entra ID requesting an access token
112+
EntraID->>SCMPublicRoute: Fetch public key via HTTPS <br>(HTTPS is important as it validates that the host is in fact the issuer)
113+
SCMPublicRoute->>EntraID: Return the public key
114+
EntraID->>EntraID: Validate signature on JWT using public key to validate that it was generated by the Issuer
115+
EntraID->>EntraID: Inspect JWT Content and ensure it passes federated identity credential policies
116+
EntraID->>SCM: Return access token for the configured application
117+
```
118+
119+
### Environment Variables for Azure Authentication
120+
121+
When Pipelines authenticates to Azure using OIDC, it provides the following environment variables to Terragrunt (and therefore OpenTofu/Terraform):
122+
123+
- `ARM_CLIENT_ID`: The client ID of the Azure AD application
124+
- `ARM_TENANT_ID`: The Azure AD tenant ID
125+
- `ARM_SUBSCRIPTION_ID`: The Azure subscription ID
126+
- `ARM_OIDC_TOKEN`: The OIDC token provided by the CI/CD platform
127+
- `ARM_USE_OIDC`: Set to `true` to enable OIDC authentication
128+
129+
The Azure provider (azurerm) uses these environment variables to authenticate directly with Entra ID using the OIDC token.

0 commit comments

Comments
 (0)