gitlabci-libsonnet

A Jsonnet library for dynamically generating GitLab CI pipelines.

Overview

This library helps you generate GitLab CI pipeline configurations programmatically using Jsonnet. It's particularly useful for monorepos where you want to selectively run jobs only for modules that have changed or that are affected by changes.

Origin Story

This library was created to solve the challenge of handling dynamic downstream pipelines in a monorepo structure. The goal was to:

• Run build/test/package jobs only for modules that changed
• Trigger jobs for modules affected by changes to shared libraries
• Use a utility to detect impacted modules and generate appropriate pipelines

Prerequisites

• Jsonnet installed on your system.
• Jsonnet Bundler installed for managing dependencies.

Installation

You can install this library for your jsonnet project using jsonnet-bundler or by cloning the repository directly.

# Using jsonnet-bundler
jb install github.com/ziouf/gitlabci-libsonnet

Usage Examples

• Simple case
• Complex case
• Multiple output files case
• Dynamic case

GitLab CI Integration

You can integrate this library with your GitLab CI workflow to dynamically generate pipelines based on changed modules.

Setup

1. Install the dependency using jsonnet-bundler:

mkdir -p .gitlab/ci && cd .gitlab/ci
jb install github.com/ziouf/gitlabci-libsonnet

2. Create a Jsonnet pipeline template:

// ${CI_PROJECT_DIR}/.gitlab/ci/pipeline.jsonnet

// Include GitlabCI library
local gitlabci = import 'vendor/gitlabci-libsonnet/gitlabci.libsonnet';

// Entrypoint: retrieve modules as a string from --tla-str command argument
function(modules="", sep=",")
{
  // Split the modules string into a list of string
  modules:: std.split(modules, sep),
  // Generate the pipeline
  "downstream-pipeline.yml": std.manifestYamlDoc(
    gl.pipeline.new()
      // Add test jobs if module name is not empty
      .withJobs({
        [if std.length(module) > 0 then "test:%s" % [module]]: $.job(module, "test")
          .withScript([
            "cd %s" % [module],
            "make test",
            "make coverage",
          ])
        for module in $.modules
      })
      // Add build jobs if module name is not empty
      .withJobs({
        [if std.length(module) > 0 then "build:%s" % [module]]: $.job(module, "build")
          .withScript([
            "cd %s" % [module],
            "make build",
          ])
        for module in $.modules
      })
  )
}

3. Configure a GitLab CI pipeline to trigger dynamically generated downstream pipelines:

# .gitlab-ci.yml

prepare:
  stage: .pre
  variables:
    # This module list may be dynamically generated by a utility that detects which modules have changed
    MODULES: "module_a,module_b,module_c"
  script:
    - |
      # Generate pipeline file
      # 'pipeline.jsonnet' is your pipeline template file importing the 'gitlabci.libsonnet' library
      # The generated pipeline file will be saved in the 'out' directory
      jsonnet -S -m out -c --tla-str modules="${MODULES}"  \
        ${CI_PROJECT_DIR}/.gitlab/ci/pipeline.jsonnet
  artifacts:
    paths:
    - out/downstream-pipeline.yml

trigger-modules:
  # The trigger job MUST be attached to a stage AFTER the prepare job
  stage: build
  variables: 
    PARENT_PIPELINE_SOURCE: ${CI_PIPELINE_SOURCE}
  trigger:
    strategy: depend
    include: 
    # Retrieve the generated pipeline file from the job artifacts
    # and use it as a template for the downstream pipeline
    - artifact: out/downstream-pipeline.yml
      job: prepare
    forward:
      yaml_variables: true
      pipeline_variables: true

License

[Add license information if applicable]

Contributing

[Add contribution guidelines if applicable]