ci: Custom GitHub Runners for Nix Builds

[yvan-sraka]

This PR implements a migration from GitHub's standard runners to a hybrid infrastructure combining self-hosted and ephemeral Blacksmith runners for building Nix packages.
The implementation includes runner selection, dynamic build matrix generation, and optimized caching strategies to improve build performance and cost efficiency.

Problem Statement

The previous CI implementation had several limitations:

1. Monolithic build process: A single job attempted to build all packages across all architectures
2. Inefficient resource allocation: All packages used the same runner type regardless of build complexity
3. Limited parallelization: Builds couldn't be efficiently distributed across different runner types
4. Redundant builds: No mechanism to skip packages already available in the binary cache
5. Poor cost optimization: Large, expensive builds ran on the same infrastructure as small, quick builds
6. Poor job output clarity: No separation of build results made it hard to identify issues

Solution Architecture

High-Level Design

┌─────────────────┐
│   nix-eval      │  Evaluates flake, generates build matrix
│   (Blacksmith)  │  Identifies cached vs. uncached packages
└────────┬────────┘  Identifies large packages
         │
         ├──────────────┬──────────────┬
         │              │              │              
         v              v              v              
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ aarch64-linux  │ │ aarch64-darwin │ │ x86_64-linux   │
│ Self-hosted/   │ │ Self-hosted    │ │ Blacksmith     │
│ Blacksmith     │ │ (macOS)        │ │ Ephemeral      │
└────────────────┘ └────────────────┘ └────────────────┘

Architecture Components

1. Nix Evaluation Phase (nix-eval.yml):

   • Runs on powerful ephemeral runner (32vcpu)
   • Evaluates all flake outputs using nix-eval-jobs
   • Checks cache status for each package
   • Generates optimized build matrices per architecture

2. Build Phases (separate jobs per architecture):

   • aarch64-linux: Self-hosted or Blacksmith ARM runners
   • aarch64-darwin: Self-hosted macOS runners
   • x86_64-linux: Blacksmith ephemeral runners

3. Runner Selection Logic:

   • KVM-required packages → Self-hosted runners with KVM support
   • Large packages (Rust, PostGIS) → 32vcpu runners
   • Standard packages → 8vcpu runners
   • Darwin packages → Self-hosted macOS runners

Key Components

1. Dynamic Matrix Generation (github-matrix Package)

Location: nix/packages/github-matrix/

Core Responsibilities:

• Evaluates Nix flake outputs using nix-eval-jobs (https://github.com/nix-community/nix-eval-jobs)
• Determines package dependencies and build order using topological sorting
• Identifies cached packages to skip redundant builds
• Assigns appropriate runners based on package requirements
• Generates GitHub Actions-compatible JSON matrices

Package Size Detection:

• Uses requiredSystemFeatures = ["big-parallel"] in package definitions
• Automatically allocates 32vcpu runners for:
  • Rust-based extensions (pg_graphql, pg_jsonschema, wrappers)
  • PostGIS (complex C++ builds)
  • pgvector with heavy dependencies

Output Format:

{
  "aarch64_linux": {
    "include": [
      {
        "attr": "checks.aarch64-linux.pg_graphql_15",
        "name": "pg_graphql-15.7",
        "system": "aarch64-linux",
        "runs_on": {"labels": ["blacksmith-32vcpu-ubuntu-2404-arm"]},
        "postgresql_version": "15"
      }
    ]
  },
  "x86_64_linux": {...},
  "aarch64_darwin": {...}
}

2. Custom Nix Installation Actions

Unify Nix installation across different runner types with two reusable GitHub Actions.

Ephemeral Runners (nix-install-ephemeral)

Location: .github/actions/nix-install-ephemeral/

Purpose: Set up Nix on fresh Blacksmith runners where Nix is not pre-installed

Features:

• Installs Nix 2.31.2 using cachix/install-nix-action
• Configures binary cache substituters
• Optionally sets up AWS credentials for cache pushing
• Creates post-build hook for automatic cache uploads

Configuration:

- uses: ./.github/actions/nix-install-ephemeral
  with:
    push-to-cache: 'true'  # Enable for build jobs
  env:
    DEV_AWS_ROLE: ${{ secrets.DEV_AWS_ROLE }}
    NIX_SIGN_SECRET_KEY: ${{ secrets.NIX_SIGN_SECRET_KEY }}

Cache Upload Mechanism:

• Post-build hook automatically uploads successful builds to S3
• Uses Nix signing keys for trusted binary cache
• Hook script: /etc/nix/upload-to-cache.sh

Self-Hosted Runners (nix-install-self-hosted)

Location: .github/actions/nix-install-self-hosted/

Purpose: Configure AWS credentials on persistent self-hosted runners where Nix is pre-installed

Features:

• Assumes AWS IAM role via OIDC
• Writes credentials to /etc/nix/aws/nix-aws-credentials
• Supports custom role duration (default 5 hours)

3. Reusable Nix Eval Workflow

Location: .github/workflows/nix-eval.yml

Purpose: Shared workflow for matrix generation

Features:

• Callable from other workflows via workflow_call
• Outputs structured JSON matrix
• Runs on high-performance ephemeral runner
• Handles optional AWS credentials for cache access

4. Restructured Build Workflow

Location: .github/workflows/nix-build.yml

New Structure:

jobs:
  nix-eval:
    # Generate build matrices
    uses: ./.github/workflows/nix-eval.yml

  nix-build-aarch64-linux:
    needs: nix-eval
    strategy:
      matrix: ${{ fromJSON(needs.nix-eval.outputs.matrix).aarch64_linux }}
    # Build ARM Linux packages

  nix-build-aarch64-darwin:
    needs: nix-eval
    strategy:
      matrix: ${{ fromJSON(needs.nix-eval.outputs.matrix).aarch64_darwin }}
    # Build macOS ARM packages

  nix-build-x86_64-linux:
    needs: nix-eval
    strategy:
      matrix: ${{ fromJSON(needs.nix-eval.outputs.matrix).x86_64_linux }}
    # Build x86_64 Linux packages

  run-testinfra:
    needs: [nix-build-aarch64-linux, ...]
    # Only run if all builds succeed or skip

  run-tests:
    needs: [nix-build-aarch64-linux, ...]
    # Run test suite

Key Improvements:

1. Parallel Architecture Builds: Each architecture builds independently
2. Smart Job Skipping: Uses !cancelled() with success/skip conditions
3. Dynamic Job Names: Include PostgreSQL version for clarity

Related PRs

• PR chores: set up nix-github-actions #1742: Base work for custom runners (this PR builds on top of it)

[samrose]

I don’t see any reason we can’t merge reviewing code. Just need to generate images and test

[samrose]

Added request change just to block merge until we test