diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 52cdc1e..365e29f 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -182,6 +182,20 @@ torrust-tracker-demo/ - **Cross-cutting**: Project-wide documentation and ADRs - **Architecture Decisions**: Documented design choices and rationale +#### Architecture Decision Records (ADRs) + +Key design decisions are documented in `docs/adr/`. Contributors should review relevant ADRs when working on related features: + +| ADR | Title | Description | +| ----------------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------- | +| [ADR-001](../docs/adr/001-makefile-location.md) | Makefile Location | Why the main Makefile is at repository root | +| [ADR-002](../docs/adr/002-docker-for-all-services.md) | Docker for All Services | Why we use Docker for all services including UDP tracker | +| [ADR-003](../docs/adr/003-use-mysql-over-mariadb.md) | Use MySQL Over MariaDB | Database backend selection rationale | +| [ADR-004](../docs/adr/004-configuration-approach-files-vs-environment-variables.md) | Configuration Approach | Files vs environment variables for configuration | +| [ADR-005](../docs/adr/005-sudo-cache-management-for-infrastructure-operations.md) | Sudo Cache Management | Managing sudo credentials during infrastructure operations | +| [ADR-006](../docs/adr/006-ssl-certificate-generation-strategy.md) | SSL Certificate Generation Strategy | Approach for SSL certificate management | +| [ADR-007](../docs/adr/007-two-level-environment-variable-structure.md) | Two-Level Environment Variable Structure | Security-focused separation of infrastructure and container variables | + ## 🛠️ Development Workflow ### Quick Start for Contributors @@ -284,9 +298,11 @@ The twelve-factor **Build, Release, Run** stages apply to the application deploy - **Release Stage**: Combine built application with environment-specific configuration - - Apply environment variables and configuration files + - Apply environment variables and configuration files using **two-level structure** (see [ADR-007](../docs/adr/007-two-level-environment-variable-structure.md)) + - **Level 1**: Main environment files (`infrastructure/config/environments/`) for all deployment processes + - **Level 2**: Docker environment files (generated from templates) for container runtime only - Combine application artifacts with runtime configuration - - Prepare deployment-ready releases + - Prepare deployment-ready releases with security-focused variable filtering - **Run Stage**: Execute the application in the runtime environment - Start application processes (tracker binary, background jobs) diff --git a/README.md b/README.md index 41ded6d..69fcda0 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,51 @@ clear separation between infrastructure provisioning and application deployment: - **Environment-based Configuration**: Template system with `local.env.tpl` and `production.env.tpl` - **Build/Release/Run Stages**: Proper separation of configuration processing, deployment, and execution +## 🚀 Deployment Process Overview + +This project follows a **four-step deployment process** based on twelve-factor app principles: + +### 1. **Configure Environment Variables** + +Customize deployment settings using environment-specific templates: + +- `infrastructure/config/environments/local.env.tpl` → `local.env` +- `infrastructure/config/environments/production.env.tpl` → `production.env` +- Contains ALL deployment configuration (infrastructure, SSL, backups, application secrets) + +### 2. **Provision Infrastructure** + +Create and configure the target environment: + +- **VM Creation**: Deploy virtual machine with specified resources +- **System Setup**: cloud-init installs dependencies, configures firewall, sets up users +- **Platform Preparation**: Ready environment for application deployment + +### 3. **Deploy Application** (Build + Release + Run Stages) + +Deploy the application stack following twelve-factor principles: + +- **Build**: Prepare application artifacts and configuration files +- **Release**: Generate SSL certificates, create Docker environment files, copy configs +- **Run**: Start Docker Compose stack (tracker, MySQL, Nginx, Grafana, Prometheus) + +### 4. **Validation** + +Verify deployment health and functionality: + +- Service health checks (HTTP/UDP endpoints) +- Database connectivity validation +- Monitoring dashboard accessibility +- End-to-end smoke testing + +### Quick Commands + +```bash +make infra-apply ENVIRONMENT=local # Steps 1-2: Configure + Provision +make app-deploy ENVIRONMENT=local # Step 3: Deploy Application +make app-health-check ENVIRONMENT=local # Step 4: Validation +``` + ## Demo Tracker - **HTTP Tracker**: diff --git a/docs/adr/007-two-level-environment-variable-structure.md b/docs/adr/007-two-level-environment-variable-structure.md new file mode 100644 index 0000000..1e17dd5 --- /dev/null +++ b/docs/adr/007-two-level-environment-variable-structure.md @@ -0,0 +1,232 @@ +# ADR-007: Two-Level Environment Variable Structure + +## Status + +Accepted + +## Context + +As part of implementing twelve-factor app methodology, we need a clear approach for +managing environment variables throughout the deployment process. The system has +evolved to use a two-level environment variable structure that serves different +purposes and security requirements. + +Currently, the project uses environment variables at two distinct levels: + +1. **Main Environment Variables**: Used for the entire deployment process +2. **Docker Compose Environment Variables**: Used only for running containers + +This separation has emerged organically but lacks clear documentation, leading to +potential confusion about where variables should be defined and how they flow +through the system. + +## Decision + +We will formalize and document a **two-level environment variable structure** +with clear separation of concerns: + +### Level 1: Main Environment Variables + +**Purpose**: Complete deployment configuration +**Location**: `infrastructure/config/environments/` +**Examples**: `local.env`, `production.env` +**Scope**: All deployment processes + +**Contents**: + +- Infrastructure configuration (VM specs, network settings) +- SSL certificate configuration (domains, Let's Encrypt email) +- Database credentials and connection parameters +- Application API tokens and secrets +- Backup and monitoring configuration +- Build and deployment automation settings + +**Usage**: + +- Sourced by deployment scripts (`provision-infrastructure.sh`, `deploy-app.sh`) +- Used for template rendering (cloud-init, configuration files) +- Contains variables for infrastructure operations (SSL generation, backups) +- Includes variables that containers never need to see + +### Level 2: Docker Compose Environment Variables + +**Purpose**: Container runtime configuration +**Template**: `infrastructure/config/templates/docker-compose.env.tpl` +**Generated File**: `.env` (in application directory) +**Scope**: Docker Compose and running containers only + +**Contents** (filtered subset from Level 1): + +- Database connection strings for application containers +- Application API tokens needed by running services +- Docker runtime configuration (USER_ID) +- Service-specific configuration (Grafana admin credentials) +- Container environment overrides + +**Filtering Criteria**: + +- **Include**: Variables directly used by containerized applications +- **Exclude**: Infrastructure-only variables (SSL domains, backup settings) +- **Exclude**: Build-time variables not needed at runtime +- **Security**: Minimize attack surface by only exposing necessary variables + +## Template Transformation Process + +```text +Level 1: Main Environment Variables +├── infrastructure/config/environments/local.env.tpl +├── infrastructure/config/environments/production.env.tpl +└── (user creates) local.env or production.env + │ + ▼ (template processing) +Level 2: Docker Environment Variables +├── infrastructure/config/templates/docker-compose.env.tpl +└── (generated) application/.env +``` + +**Processing Flow**: + +1. User creates environment file from template (e.g., `local.env`) +2. Deployment script sources the main environment file +3. Template processor generates `docker-compose.env` from template +4. Docker Compose uses the generated `.env` file for container variables + +## Rationale + +### Security Benefits + +- **Principle of Least Privilege**: Containers only receive variables they need +- **Reduced Attack Surface**: Infrastructure secrets not exposed to application containers +- **Separation of Concerns**: Infrastructure and application secrets handled differently + +### Operational Benefits + +- **Clear Responsibility**: Infrastructure variables vs. application variables +- **Easier Debugging**: Know where to look for specific types of configuration +- **Template Flexibility**: Can generate different container environments from same base config +- **Deployment Isolation**: Infrastructure operations don't leak sensitive data to containers + +### Examples + +**Level 1 Only (Infrastructure Variables)**: + +```bash +# SSL configuration (not needed in containers) +SSL_DOMAIN="tracker.example.com" +SSL_EMAIL="admin@example.com" +ENABLE_SSL_AUTOMATION="true" + +# Backup configuration (not needed in containers) +ENABLE_DB_BACKUPS="true" +BACKUP_RETENTION_DAYS="30" + +# Infrastructure specifications (not needed in containers) +VM_MEMORY="4096" +VM_VCPUS="4" +``` + +**Level 2 (Container Variables - Filtered from Level 1)**: + +```bash +# Database connection (needed by tracker container) +MYSQL_ROOT_PASSWORD="secure_root_password" +MYSQL_PASSWORD="secure_user_password" +TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH="mysql://torrust:${MYSQL_PASSWORD}@mysql:3306/torrust_tracker" + +# API tokens (needed by application) +TRACKER_ADMIN_TOKEN="secure_api_token" + +# Runtime configuration (needed by containers) +USER_ID="1000" +``` + +## Implementation Guidelines + +### For Infrastructure Scripts + +```bash +# Source the main environment file +source "infrastructure/config/environments/${ENVIRONMENT}.env" + +# Use all variables for infrastructure operations +generate_ssl_certificates "$SSL_DOMAIN" "$SSL_EMAIL" +configure_backups "$ENABLE_DB_BACKUPS" "$BACKUP_RETENTION_DAYS" +``` + +### For Template Processing + +```bash +# Generate Docker environment file from template +envsubst < "infrastructure/config/templates/docker-compose.env.tpl" > "application/.env" +``` + +### For Container Configuration + +```bash +# Docker Compose command (in deploy-app.sh) +docker compose --env-file /var/lib/torrust/compose/.env up -d + +# The .env file contains only container-relevant variables +# and is passed to Docker Compose via the --env-file flag +``` + +## Benefits + +1. **Security**: Reduced container attack surface +2. **Clarity**: Clear separation between infrastructure and application concerns +3. **Maintainability**: Easier to understand what variables are used where +4. **Flexibility**: Can generate different container environments from same base +5. **Compliance**: Aligns with twelve-factor configuration principles +6. **Debugging**: Easier to troubleshoot configuration issues + +## Trade-offs + +### Accepted Complexity + +- **Two Files to Maintain**: Requires keeping template and source in sync +- **Template Processing**: Additional step in deployment process +- **Learning Curve**: Contributors must understand the two-level structure + +### Mitigated Risks + +- **Template Drift**: Validation scripts check template consistency +- **Missing Variables**: Docker Compose will fail fast if required variables are missing +- **Documentation**: This ADR and inline comments clarify the structure + +## Consequences + +### For Contributors + +- Must understand which level to modify for different types of changes +- Infrastructure changes: Edit main environment templates +- Container configuration: Edit Docker environment template +- New variables: Consider which level(s) need the variable + +### For Deployment + +- All deployment scripts use Level 1 (main environment) +- Docker Compose only sees Level 2 (filtered environment) +- Template processing is automatic during deployment +- No manual synchronization required + +### For Security + +- Infrastructure secrets isolated from application containers +- Container compromise doesn't expose infrastructure configuration +- Easier security auditing of container-exposed variables + +## Alternative Considered + +**Single-Level Environment Variables**: Using one environment file for everything. + +**Rejected because**: + +- Security: All variables exposed to containers +- Complexity: Difficult to determine which variables containers actually need +- Maintenance: Changes to infrastructure configuration could affect containers unnecessarily + +## References + +- [Twelve-Factor App: Config](https://12factor.net/config) +- [ADR-004: Configuration Approach](./004-configuration-approach-files-vs-environment-variables.md) +- [Docker Compose Environment Variables](https://docs.docker.com/compose/environment-variables/) diff --git a/docs/guides/cloud-deployment-guide.md b/docs/guides/cloud-deployment-guide.md index 2ad2c35..586ee65 100644 --- a/docs/guides/cloud-deployment-guide.md +++ b/docs/guides/cloud-deployment-guide.md @@ -12,6 +12,53 @@ and testing. Hetzner Cloud support is planned as the next implementation target. The process combines Infrastructure as Code with application deployment automation to provide a streamlined deployment experience, following twelve-factor app methodology. +## Deployment Process + +This project implements a **four-step deployment workflow** aligned with twelve-factor app principles: + +### Step 1: Configure Environment Variables + +Create environment-specific configuration from templates: + +- **Local Development**: `infrastructure/config/environments/local.env.tpl` → `local.env` +- **Production**: `infrastructure/config/environments/production.env.tpl` → `production.env` + +The environment file contains **all deployment configuration**, including: + +- Infrastructure settings (VM specs, network configuration) +- Application secrets (database passwords, API tokens) +- SSL certificate configuration (domains, email for Let's Encrypt) +- Backup and monitoring settings + +### Step 2: Provision Infrastructure + +Deploy and configure the target environment: + +- **VM Creation**: Deploy virtual machine with specified resources +- **System Dependencies**: cloud-init installs Docker, configures firewall, creates users +- **Network Setup**: Configure firewall rules, SSH access, system security +- **Platform Readiness**: Environment prepared for application deployment + +### Step 3: Deploy Application (Build + Release + Run) + +Deploy the application stack following twelve-factor methodology: + +- **Build Stage**: Prepare application artifacts and configuration templates +- **Release Stage**: + - Generate SSL certificates (Let's Encrypt or self-signed) + - Create Docker environment files from templates + - Copy application configurations to target VM +- **Run Stage**: Start Docker Compose stack (Torrust Tracker, MySQL, Nginx, Grafana, Prometheus) + +### Step 4: Validation + +Verify deployment health and functionality: + +- **Service Health**: HTTP/UDP endpoint availability checks +- **Database Connectivity**: MySQL connection and schema validation +- **Monitoring Access**: Grafana dashboard accessibility +- **End-to-end Testing**: Tracker announce/scrape functionality + ## Prerequisites ### Local Requirements