From 3471409129fe755eb5603748d7f6e1f32a6067e6 Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Thu, 31 Jul 2025 17:05:33 +0100 Subject: [PATCH 1/3] docs: [#26] add 12-factor deployment process overview - Add comprehensive 4-step deployment process section to README.md - Enhance cloud deployment guide with detailed workflow explanation - Add ADR reference table to copilot instructions for contributor guidance - Clarify separation between infrastructure provisioning and application deployment - Document Build/Release/Run stages alignment with twelve-factor methodology --- .github/copilot-instructions.md | 13 ++++++++ README.md | 45 +++++++++++++++++++++++++ docs/guides/cloud-deployment-guide.md | 47 +++++++++++++++++++++++++++ 3 files changed, 105 insertions(+) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 52cdc1e..bcef4ec 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -182,6 +182,19 @@ torrust-tracker-demo/ - **Cross-cutting**: Project-wide documentation and ADRs - **Architecture Decisions**: Documented design choices and rationale +#### Architecture Decision Records (ADRs) + +Key design decisions are documented in `docs/adr/`. Contributors should review relevant ADRs when working on related features: + +| ADR | Title | Description | +|-----|-------|-------------| +| [ADR-001](../docs/adr/001-makefile-location.md) | Makefile Location | Why the main Makefile is at repository root | +| [ADR-002](../docs/adr/002-docker-for-all-services.md) | Docker for All Services | Why we use Docker for all services including UDP tracker | +| [ADR-003](../docs/adr/003-use-mysql-over-mariadb.md) | Use MySQL Over MariaDB | Database backend selection rationale | +| [ADR-004](../docs/adr/004-configuration-approach-files-vs-environment-variables.md) | Configuration Approach | Files vs environment variables for configuration | +| [ADR-005](../docs/adr/005-sudo-cache-management-for-infrastructure-operations.md) | Sudo Cache Management | Managing sudo credentials during infrastructure operations | +| [ADR-006](../docs/adr/006-ssl-certificate-generation-strategy.md) | SSL Certificate Generation Strategy | Approach for SSL certificate management | + ## 🛠️ Development Workflow ### Quick Start for Contributors diff --git a/README.md b/README.md index 41ded6d..69fcda0 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,51 @@ clear separation between infrastructure provisioning and application deployment: - **Environment-based Configuration**: Template system with `local.env.tpl` and `production.env.tpl` - **Build/Release/Run Stages**: Proper separation of configuration processing, deployment, and execution +## 🚀 Deployment Process Overview + +This project follows a **four-step deployment process** based on twelve-factor app principles: + +### 1. **Configure Environment Variables** + +Customize deployment settings using environment-specific templates: + +- `infrastructure/config/environments/local.env.tpl` → `local.env` +- `infrastructure/config/environments/production.env.tpl` → `production.env` +- Contains ALL deployment configuration (infrastructure, SSL, backups, application secrets) + +### 2. **Provision Infrastructure** + +Create and configure the target environment: + +- **VM Creation**: Deploy virtual machine with specified resources +- **System Setup**: cloud-init installs dependencies, configures firewall, sets up users +- **Platform Preparation**: Ready environment for application deployment + +### 3. **Deploy Application** (Build + Release + Run Stages) + +Deploy the application stack following twelve-factor principles: + +- **Build**: Prepare application artifacts and configuration files +- **Release**: Generate SSL certificates, create Docker environment files, copy configs +- **Run**: Start Docker Compose stack (tracker, MySQL, Nginx, Grafana, Prometheus) + +### 4. **Validation** + +Verify deployment health and functionality: + +- Service health checks (HTTP/UDP endpoints) +- Database connectivity validation +- Monitoring dashboard accessibility +- End-to-end smoke testing + +### Quick Commands + +```bash +make infra-apply ENVIRONMENT=local # Steps 1-2: Configure + Provision +make app-deploy ENVIRONMENT=local # Step 3: Deploy Application +make app-health-check ENVIRONMENT=local # Step 4: Validation +``` + ## Demo Tracker - **HTTP Tracker**: diff --git a/docs/guides/cloud-deployment-guide.md b/docs/guides/cloud-deployment-guide.md index 2ad2c35..586ee65 100644 --- a/docs/guides/cloud-deployment-guide.md +++ b/docs/guides/cloud-deployment-guide.md @@ -12,6 +12,53 @@ and testing. Hetzner Cloud support is planned as the next implementation target. The process combines Infrastructure as Code with application deployment automation to provide a streamlined deployment experience, following twelve-factor app methodology. +## Deployment Process + +This project implements a **four-step deployment workflow** aligned with twelve-factor app principles: + +### Step 1: Configure Environment Variables + +Create environment-specific configuration from templates: + +- **Local Development**: `infrastructure/config/environments/local.env.tpl` → `local.env` +- **Production**: `infrastructure/config/environments/production.env.tpl` → `production.env` + +The environment file contains **all deployment configuration**, including: + +- Infrastructure settings (VM specs, network configuration) +- Application secrets (database passwords, API tokens) +- SSL certificate configuration (domains, email for Let's Encrypt) +- Backup and monitoring settings + +### Step 2: Provision Infrastructure + +Deploy and configure the target environment: + +- **VM Creation**: Deploy virtual machine with specified resources +- **System Dependencies**: cloud-init installs Docker, configures firewall, creates users +- **Network Setup**: Configure firewall rules, SSH access, system security +- **Platform Readiness**: Environment prepared for application deployment + +### Step 3: Deploy Application (Build + Release + Run) + +Deploy the application stack following twelve-factor methodology: + +- **Build Stage**: Prepare application artifacts and configuration templates +- **Release Stage**: + - Generate SSL certificates (Let's Encrypt or self-signed) + - Create Docker environment files from templates + - Copy application configurations to target VM +- **Run Stage**: Start Docker Compose stack (Torrust Tracker, MySQL, Nginx, Grafana, Prometheus) + +### Step 4: Validation + +Verify deployment health and functionality: + +- **Service Health**: HTTP/UDP endpoint availability checks +- **Database Connectivity**: MySQL connection and schema validation +- **Monitoring Access**: Grafana dashboard accessibility +- **End-to-end Testing**: Tracker announce/scrape functionality + ## Prerequisites ### Local Requirements From 207c4048e7a1b0aa3b81c6b9d8207a0ac7944ff3 Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Thu, 31 Jul 2025 17:12:40 +0100 Subject: [PATCH 2/3] docs: [#26] create ADR-007 for two-level environment variable structure - Add comprehensive ADR documenting security-focused environment variable separation - Level 1: Main environment files for complete deployment configuration - Level 2: Docker environment files with filtered container-only variables - Update ADR table in copilot instructions with new ADR-007 entry - Enhance twelve-factor Release stage documentation with environment structure details - Provide clear guidelines for infrastructure vs container variable management --- .github/copilot-instructions.md | 23 +- ...wo-level-environment-variable-structure.md | 233 ++++++++++++++++++ 2 files changed, 246 insertions(+), 10 deletions(-) create mode 100644 docs/adr/007-two-level-environment-variable-structure.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index bcef4ec..c63467c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -186,14 +186,15 @@ torrust-tracker-demo/ Key design decisions are documented in `docs/adr/`. Contributors should review relevant ADRs when working on related features: -| ADR | Title | Description | -|-----|-------|-------------| -| [ADR-001](../docs/adr/001-makefile-location.md) | Makefile Location | Why the main Makefile is at repository root | -| [ADR-002](../docs/adr/002-docker-for-all-services.md) | Docker for All Services | Why we use Docker for all services including UDP tracker | -| [ADR-003](../docs/adr/003-use-mysql-over-mariadb.md) | Use MySQL Over MariaDB | Database backend selection rationale | -| [ADR-004](../docs/adr/004-configuration-approach-files-vs-environment-variables.md) | Configuration Approach | Files vs environment variables for configuration | -| [ADR-005](../docs/adr/005-sudo-cache-management-for-infrastructure-operations.md) | Sudo Cache Management | Managing sudo credentials during infrastructure operations | -| [ADR-006](../docs/adr/006-ssl-certificate-generation-strategy.md) | SSL Certificate Generation Strategy | Approach for SSL certificate management | +| ADR | Title | Description | +| ----------------------------------------------------------------------------------- | ----------------------------------- | ---------------------------------------------------------- | +| [ADR-001](../docs/adr/001-makefile-location.md) | Makefile Location | Why the main Makefile is at repository root | +| [ADR-002](../docs/adr/002-docker-for-all-services.md) | Docker for All Services | Why we use Docker for all services including UDP tracker | +| [ADR-003](../docs/adr/003-use-mysql-over-mariadb.md) | Use MySQL Over MariaDB | Database backend selection rationale | +| [ADR-004](../docs/adr/004-configuration-approach-files-vs-environment-variables.md) | Configuration Approach | Files vs environment variables for configuration | +| [ADR-005](../docs/adr/005-sudo-cache-management-for-infrastructure-operations.md) | Sudo Cache Management | Managing sudo credentials during infrastructure operations | +| [ADR-006](../docs/adr/006-ssl-certificate-generation-strategy.md) | SSL Certificate Generation Strategy | Approach for SSL certificate management | +| [ADR-007](../docs/adr/007-two-level-environment-variable-structure.md) | Two-Level Environment Variable Structure | Security-focused separation of infrastructure and container variables | ## 🛠️ Development Workflow @@ -297,9 +298,11 @@ The twelve-factor **Build, Release, Run** stages apply to the application deploy - **Release Stage**: Combine built application with environment-specific configuration - - Apply environment variables and configuration files + - Apply environment variables and configuration files using **two-level structure** (see [ADR-007](../docs/adr/007-two-level-environment-variable-structure.md)) + - **Level 1**: Main environment files (`infrastructure/config/environments/`) for all deployment processes + - **Level 2**: Docker environment files (generated from templates) for container runtime only - Combine application artifacts with runtime configuration - - Prepare deployment-ready releases + - Prepare deployment-ready releases with security-focused variable filtering - **Run Stage**: Execute the application in the runtime environment - Start application processes (tracker binary, background jobs) diff --git a/docs/adr/007-two-level-environment-variable-structure.md b/docs/adr/007-two-level-environment-variable-structure.md new file mode 100644 index 0000000..2440fc8 --- /dev/null +++ b/docs/adr/007-two-level-environment-variable-structure.md @@ -0,0 +1,233 @@ +# ADR-007: Two-Level Environment Variable Structure + +## Status + +Accepted + +## Context + +As part of implementing twelve-factor app methodology, we need a clear approach for +managing environment variables throughout the deployment process. The system has +evolved to use a two-level environment variable structure that serves different +purposes and security requirements. + +Currently, the project uses environment variables at two distinct levels: + +1. **Main Environment Variables**: Used for the entire deployment process +2. **Docker Compose Environment Variables**: Used only for running containers + +This separation has emerged organically but lacks clear documentation, leading to +potential confusion about where variables should be defined and how they flow +through the system. + +## Decision + +We will formalize and document a **two-level environment variable structure** +with clear separation of concerns: + +### Level 1: Main Environment Variables + +**Purpose**: Complete deployment configuration +**Location**: `infrastructure/config/environments/` +**Examples**: `local.env`, `production.env` +**Scope**: All deployment processes + +**Contents**: + +- Infrastructure configuration (VM specs, network settings) +- SSL certificate configuration (domains, Let's Encrypt email) +- Database credentials and connection parameters +- Application API tokens and secrets +- Backup and monitoring configuration +- Build and deployment automation settings + +**Usage**: + +- Sourced by deployment scripts (`provision-infrastructure.sh`, `deploy-app.sh`) +- Used for template rendering (cloud-init, configuration files) +- Contains variables for infrastructure operations (SSL generation, backups) +- Includes variables that containers never need to see + +### Level 2: Docker Compose Environment Variables + +**Purpose**: Container runtime configuration +**Template**: `infrastructure/config/templates/docker-compose.env.tpl` +**Generated File**: `.env` (in application directory) +**Scope**: Docker Compose and running containers only + +**Contents** (filtered subset from Level 1): + +- Database connection strings for application containers +- Application API tokens needed by running services +- Docker runtime configuration (USER_ID) +- Service-specific configuration (Grafana admin credentials) +- Container environment overrides + +**Filtering Criteria**: + +- **Include**: Variables directly used by containerized applications +- **Exclude**: Infrastructure-only variables (SSL domains, backup settings) +- **Exclude**: Build-time variables not needed at runtime +- **Security**: Minimize attack surface by only exposing necessary variables + +## Template Transformation Process + +```text +Level 1: Main Environment Variables +├── infrastructure/config/environments/local.env.tpl +├── infrastructure/config/environments/production.env.tpl +└── (user creates) local.env or production.env + │ + ▼ (template processing) +Level 2: Docker Environment Variables +├── infrastructure/config/templates/docker-compose.env.tpl +└── (generated) application/.env +``` + +**Processing Flow**: + +1. User creates environment file from template (e.g., `local.env`) +2. Deployment script sources the main environment file +3. Template processor generates `docker-compose.env` from template +4. Docker Compose uses the generated `.env` file for container variables + +## Rationale + +### Security Benefits + +- **Principle of Least Privilege**: Containers only receive variables they need +- **Reduced Attack Surface**: Infrastructure secrets not exposed to application containers +- **Separation of Concerns**: Infrastructure and application secrets handled differently + +### Operational Benefits + +- **Clear Responsibility**: Infrastructure variables vs. application variables +- **Easier Debugging**: Know where to look for specific types of configuration +- **Template Flexibility**: Can generate different container environments from same base config +- **Deployment Isolation**: Infrastructure operations don't leak sensitive data to containers + +### Examples + +**Level 1 Only (Infrastructure Variables)**: + +```bash +# SSL configuration (not needed in containers) +SSL_DOMAIN="tracker.example.com" +SSL_EMAIL="admin@example.com" +ENABLE_SSL_AUTOMATION="true" + +# Backup configuration (not needed in containers) +ENABLE_DB_BACKUPS="true" +BACKUP_RETENTION_DAYS="30" + +# Infrastructure specifications (not needed in containers) +VM_MEMORY="4096" +VM_VCPUS="4" +``` + +**Level 2 (Container Variables - Filtered from Level 1)**: + +```bash +# Database connection (needed by tracker container) +MYSQL_ROOT_PASSWORD="secure_root_password" +MYSQL_PASSWORD="secure_user_password" +TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH="mysql://torrust:${MYSQL_PASSWORD}@mysql:3306/torrust_tracker" + +# API tokens (needed by application) +TRACKER_ADMIN_TOKEN="secure_api_token" + +# Runtime configuration (needed by containers) +USER_ID="1000" +``` + +## Implementation Guidelines + +### For Infrastructure Scripts + +```bash +# Source the main environment file +source "infrastructure/config/environments/${ENVIRONMENT}.env" + +# Use all variables for infrastructure operations +generate_ssl_certificates "$SSL_DOMAIN" "$SSL_EMAIL" +configure_backups "$ENABLE_DB_BACKUPS" "$BACKUP_RETENTION_DAYS" +``` + +### For Template Processing + +```bash +# Generate Docker environment file from template +envsubst < "infrastructure/config/templates/docker-compose.env.tpl" > "application/.env" +``` + +### For Container Configuration + +```yaml +# docker-compose.yaml +services: + tracker: + env_file: .env # Only contains container-relevant variables + environment: + - TRACKER_ADMIN_TOKEN=${TRACKER_ADMIN_TOKEN} +``` + +## Benefits + +1. **Security**: Reduced container attack surface +2. **Clarity**: Clear separation between infrastructure and application concerns +3. **Maintainability**: Easier to understand what variables are used where +4. **Flexibility**: Can generate different container environments from same base +5. **Compliance**: Aligns with twelve-factor configuration principles +6. **Debugging**: Easier to troubleshoot configuration issues + +## Trade-offs + +### Accepted Complexity + +- **Two Files to Maintain**: Requires keeping template and source in sync +- **Template Processing**: Additional step in deployment process +- **Learning Curve**: Contributors must understand the two-level structure + +### Mitigated Risks + +- **Template Drift**: Validation scripts check template consistency +- **Missing Variables**: Docker Compose will fail fast if required variables are missing +- **Documentation**: This ADR and inline comments clarify the structure + +## Consequences + +### For Contributors + +- Must understand which level to modify for different types of changes +- Infrastructure changes: Edit main environment templates +- Container configuration: Edit Docker environment template +- New variables: Consider which level(s) need the variable + +### For Deployment + +- All deployment scripts use Level 1 (main environment) +- Docker Compose only sees Level 2 (filtered environment) +- Template processing is automatic during deployment +- No manual synchronization required + +### For Security + +- Infrastructure secrets isolated from application containers +- Container compromise doesn't expose infrastructure configuration +- Easier security auditing of container-exposed variables + +## Alternative Considered + +**Single-Level Environment Variables**: Using one environment file for everything. + +**Rejected because**: + +- Security: All variables exposed to containers +- Complexity: Difficult to determine which variables containers actually need +- Maintenance: Changes to infrastructure configuration could affect containers unnecessarily + +## References + +- [Twelve-Factor App: Config](https://12factor.net/config) +- [ADR-004: Configuration Approach](./004-configuration-approach-files-vs-environment-variables.md) +- [Docker Compose Environment Variables](https://docs.docker.com/compose/environment-variables/) From 7ede6e40c54b8a346a367a268b56583346699402 Mon Sep 17 00:00:00 2001 From: Jose Celano Date: Thu, 31 Jul 2025 17:17:11 +0100 Subject: [PATCH 3/3] docs: [#26] fix Docker Compose environment configuration example and improve formatting - Correct ADR-007 container configuration example to show actual --env-file usage - Fix Docker Compose example from incorrect env_file property to correct command-line flag - Improve ADR table formatting in copilot instructions for better readability - Remove trailing spaces and minor formatting improvements --- .github/copilot-instructions.md | 16 ++++++++-------- ...7-two-level-environment-variable-structure.md | 15 +++++++-------- 2 files changed, 15 insertions(+), 16 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c63467c..365e29f 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -186,14 +186,14 @@ torrust-tracker-demo/ Key design decisions are documented in `docs/adr/`. Contributors should review relevant ADRs when working on related features: -| ADR | Title | Description | -| ----------------------------------------------------------------------------------- | ----------------------------------- | ---------------------------------------------------------- | -| [ADR-001](../docs/adr/001-makefile-location.md) | Makefile Location | Why the main Makefile is at repository root | -| [ADR-002](../docs/adr/002-docker-for-all-services.md) | Docker for All Services | Why we use Docker for all services including UDP tracker | -| [ADR-003](../docs/adr/003-use-mysql-over-mariadb.md) | Use MySQL Over MariaDB | Database backend selection rationale | -| [ADR-004](../docs/adr/004-configuration-approach-files-vs-environment-variables.md) | Configuration Approach | Files vs environment variables for configuration | -| [ADR-005](../docs/adr/005-sudo-cache-management-for-infrastructure-operations.md) | Sudo Cache Management | Managing sudo credentials during infrastructure operations | -| [ADR-006](../docs/adr/006-ssl-certificate-generation-strategy.md) | SSL Certificate Generation Strategy | Approach for SSL certificate management | +| ADR | Title | Description | +| ----------------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------- | +| [ADR-001](../docs/adr/001-makefile-location.md) | Makefile Location | Why the main Makefile is at repository root | +| [ADR-002](../docs/adr/002-docker-for-all-services.md) | Docker for All Services | Why we use Docker for all services including UDP tracker | +| [ADR-003](../docs/adr/003-use-mysql-over-mariadb.md) | Use MySQL Over MariaDB | Database backend selection rationale | +| [ADR-004](../docs/adr/004-configuration-approach-files-vs-environment-variables.md) | Configuration Approach | Files vs environment variables for configuration | +| [ADR-005](../docs/adr/005-sudo-cache-management-for-infrastructure-operations.md) | Sudo Cache Management | Managing sudo credentials during infrastructure operations | +| [ADR-006](../docs/adr/006-ssl-certificate-generation-strategy.md) | SSL Certificate Generation Strategy | Approach for SSL certificate management | | [ADR-007](../docs/adr/007-two-level-environment-variable-structure.md) | Two-Level Environment Variable Structure | Security-focused separation of infrastructure and container variables | ## 🛠️ Development Workflow diff --git a/docs/adr/007-two-level-environment-variable-structure.md b/docs/adr/007-two-level-environment-variable-structure.md index 2440fc8..1e17dd5 100644 --- a/docs/adr/007-two-level-environment-variable-structure.md +++ b/docs/adr/007-two-level-environment-variable-structure.md @@ -162,19 +162,18 @@ envsubst < "infrastructure/config/templates/docker-compose.env.tpl" > "applicati ### For Container Configuration -```yaml -# docker-compose.yaml -services: - tracker: - env_file: .env # Only contains container-relevant variables - environment: - - TRACKER_ADMIN_TOKEN=${TRACKER_ADMIN_TOKEN} +```bash +# Docker Compose command (in deploy-app.sh) +docker compose --env-file /var/lib/torrust/compose/.env up -d + +# The .env file contains only container-relevant variables +# and is passed to Docker Compose via the --env-file flag ``` ## Benefits 1. **Security**: Reduced container attack surface -2. **Clarity**: Clear separation between infrastructure and application concerns +2. **Clarity**: Clear separation between infrastructure and application concerns 3. **Maintainability**: Easier to understand what variables are used where 4. **Flexibility**: Can generate different container environments from same base 5. **Compliance**: Aligns with twelve-factor configuration principles