Skip to content

Conversation

pkoutsovasilis
Copy link
Contributor

@pkoutsovasilis pkoutsovasilis commented Jul 17, 2025

What does this PR do?

This PR reworks the .package-version file format and integrates it into our CI and mage workflows to enable deterministic, snapshot-based builds of Elastic Agent based on published DRA (Downloadable Release Artifacts).

Key changes:

  • .package-version is now a structured JSON file containing version metadata (version, build_id, stack_build_id, manifest_url, etc.).
  • CI scripts and mage targets now conditionally derive the agent version and build metadata from .package-version when USE_PACKAGE_VERSION=true is set.
  • The mage target UpdatePackageVersion replaces previous logic with a structured update mechanism for .package-version.
  • CI integration steps now consistently use the snapshot version and manifest provided in .package-version.

Why is it important?

We previously had a fundamental dependency issue in the release flow: to produce a new Elastic Agent DRA, we needed to bump the agent version — but our CI integration tests required that same DRA to already exist in order to pass. This created a timing problem and made the release workflow fragile.

With this PR:

  • The CI system and mage targets now honour .package-version when USE_PACKAGE_VERSION=true is set, allowing packaging and testing to rely on a known-good, previously published DRA.
  • This ensures that builds and tests only proceed against DRAs that have been published by the unified release process, making test results and releases more predictable and reliable.
  • PRs such as #9163, which update .package-version, now become the authoritative source of truth for the Elastic Agent version used in the repo and must be handled with utmost care.
  • Note: Even if a component publishes its own DRA, Elastic Agent will only package it once the unified release has run and the respective PR to update the .package-version has been merged.

This enables a reproducible and version-pinned CI system while allowing us to automate snapshot bumps in a controlled and testable way.

An experimental version bump under commit a57ee10 verified this flow successfully. The CI run is available here.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Disruptive User Impact

None expected. Existing workflows are unaffected unless USE_PACKAGE_VERSION=true is explicitly set. In that case, .package-version is required and must be up to date.

How to test this PR locally

specify USE_PACKAGE_VERION=true and call any mage target e.g.

USE_PACKAGE_VERSION="true" EXTERNAL="true" PLATFORMS="linux/arm64" PACKAGES="docker" DOCKER_VARIANTS="basic" mage package

Related issues

  • N/A

@pkoutsovasilis pkoutsovasilis self-assigned this Jul 17, 2025
@pkoutsovasilis pkoutsovasilis added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog backport-active-all Automated backport with mergify to all the active branches labels Jul 17, 2025
@pkoutsovasilis pkoutsovasilis force-pushed the ci/build_from_snapshot_dra_automate_bump branch 13 times, most recently from 1ea3e62 to 7d56a08 Compare July 18, 2025 06:50
@pkoutsovasilis pkoutsovasilis force-pushed the ci/build_from_snapshot_dra_automate_bump branch 4 times, most recently from f4883cc to 9cbfc5b Compare July 24, 2025 09:30
@pkoutsovasilis pkoutsovasilis force-pushed the ci/build_from_snapshot_dra_automate_bump branch from 9cbfc5b to 3c6f279 Compare July 28, 2025 03:12
@pkoutsovasilis pkoutsovasilis force-pushed the ci/build_from_snapshot_dra_automate_bump branch from 3c6f279 to a57ee10 Compare July 28, 2025 17:09
Copy link

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @pkoutsovasilis

Copy link
Member

@pchila pchila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked BK run https://buildkite.com/elastic/elastic-agent/builds/24678 and I see the correct dependencies being downloaded for both FIPS and non FIPS packaging.

LGTM

Copy link
Contributor

@swiatekm swiatekm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did the same spot check as Paolo and everything looked correct.

Copy link
Contributor

@fr4nc1sc0-r4m0n fr4nc1sc0-r4m0n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@pkoutsovasilis pkoutsovasilis merged commit a155660 into elastic:main Aug 7, 2025
19 checks passed
Copy link
Contributor

github-actions bot commented Aug 7, 2025

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

Copy link
Contributor

mergify bot commented Aug 7, 2025

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go
mergify bot pushed a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go
mergify bot pushed a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.package-version
#	testing/integration/ess/upgrade_broken_package_test.go
mergify bot pushed a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go
mergify bot pushed a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.package-version
pkoutsovasilis added a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go

# Conflicts:
#	dev-tools/mage/manifest/manifest.go
pkoutsovasilis added a commit that referenced this pull request Aug 7, 2025
* ci: build agent from snapshot DRA (#9048)

* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.package-version

* fix: resolve conflicts

* fix: remove image cloud pushing step

---------

Co-authored-by: Panos Koutsovasilis <[email protected]>
pkoutsovasilis added a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)
pkoutsovasilis added a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go
pkoutsovasilis added a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go
pkoutsovasilis added a commit that referenced this pull request Aug 7, 2025
* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.package-version
#	testing/integration/ess/upgrade_broken_package_test.go
pkoutsovasilis added a commit that referenced this pull request Aug 8, 2025
* Make components in packages configurable (#7602)

* Redefine ExpectedBinaries as YAML config

* Move ExpectedBinaries closer to package spec file

* Fix error formatting in downloadDRAArtifacts

* add packageName template to ExpectedPackages

* use a relaxed dependencies version for IAR releases

* Remove FIPS hack introduced in PR #7486

* Allow for a looser match on relaxing dependencies versions

* Add debug logging when packaging with EXTERNAL=true

* move package tests to dedicated package

* Fips packaging (#7690)

* Add component list to specs

* extract component dependencies from the packages to be built

* Refactor component extraction from package specs

* Fix package tests error handling

* Inject dependencies and remove references to ExpectedBinaries

* Remove ExpectedBinaries global

* Add rootdir to components

* Extract actual version matched on the package file and use it to render RootDir

* Package elastic-agent FIPS specs when FIPS=true is specified

* refactor ResolveManifestPackage

* Move FIPS compile settings in packages.yml

* Add more FIPS components

* Properly handle dependenciesVersion when calling mage package

* Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files

* Rework useDRAAgentBinaryForPackage for repackaging agent

Define elastic-agent-core components (both FIPS and non-FIPS variants)
and define package name and root dir templates.
Implement some filtering on component list to extract the correct
component definition according to the FIPSBuild flag.
Refactor code that downloads pre-compiled elastic-agent binaries and
places them in the golangCrossBuild directory to make use of the new
component definition.

* Write spec FIPS flag into manifest.yaml when packaging

* Add FIPS elastic agent basic and cloud docker images

* Build FIPS docker images in CI packaging

* Fix FIPS .tar.gz package tests

* Restructure package tests

* Extend FIPS check to all binaries in components directory

* Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline

* Cleanup ChecksumsWithManifest and improve godoc

* Improve godoc for BinarySpec

* Correctly inject dependency list when packaging using DROP_PATH (#7795)

* Restore qualifier=core for elastic-agent-core packaging specs (#7805)

Restore qualifier for elastic-agent-core packaging specs to avoid
changing the rootDir name of the archives.
The qualifier had been removed in PR #7690 trying to use the spec name:
this worked to get the desired file name but changed the root Dir name
which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the
template definition instead of '{{.Name}}' which would render the spec
name.

* Modify fips core spec qualifier and name (#7818)

* ci: build agent from snapshot DRA (#9048)

* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

* fix: remove fips-related changes

* fix: resolve QF1004: could use strings.ReplaceAll linter errors

* fix: remove unused imports

---------

Co-authored-by: Paolo Chilà <[email protected]>
Co-authored-by: Panos Koutsovasilis <[email protected]>
pkoutsovasilis added a commit that referenced this pull request Aug 8, 2025
* ci: build agent from snapshot DRA (#9048)

* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go

* [8.x](backport #7690) Fips packaging (#7790)

* Make components in packages configurable (#7602)

* Redefine ExpectedBinaries as YAML config

* Move ExpectedBinaries closer to package spec file

* Fix error formatting in downloadDRAArtifacts

* add packageName template to ExpectedPackages

* use a relaxed dependencies version for IAR releases

* Remove FIPS hack introduced in PR #7486

* Allow for a looser match on relaxing dependencies versions

* Add debug logging when packaging with EXTERNAL=true

* move package tests to dedicated package

* Fips packaging (#7690)

* Add component list to specs

* extract component dependencies from the packages to be built

* Refactor component extraction from package specs

* Fix package tests error handling

* Inject dependencies and remove references to ExpectedBinaries

* Remove ExpectedBinaries global

* Add rootdir to components

* Extract actual version matched on the package file and use it to render RootDir

* Package elastic-agent FIPS specs when FIPS=true is specified

* refactor ResolveManifestPackage

* Move FIPS compile settings in packages.yml

* Add more FIPS components

* Properly handle dependenciesVersion when calling mage package

* Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files

* Rework useDRAAgentBinaryForPackage for repackaging agent

Define elastic-agent-core components (both FIPS and non-FIPS variants)
and define package name and root dir templates.
Implement some filtering on component list to extract the correct
component definition according to the FIPSBuild flag.
Refactor code that downloads pre-compiled elastic-agent binaries and
places them in the golangCrossBuild directory to make use of the new
component definition.

* Write spec FIPS flag into manifest.yaml when packaging

* Add FIPS elastic agent basic and cloud docker images

* Build FIPS docker images in CI packaging

* Fix FIPS .tar.gz package tests

* Restructure package tests

* Extend FIPS check to all binaries in components directory

* Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline

* Cleanup ChecksumsWithManifest and improve godoc

* Improve godoc for BinarySpec

* Correctly inject dependency list when packaging using DROP_PATH (#7795)

* Restore qualifier=core for elastic-agent-core packaging specs (#7805)

Restore qualifier for elastic-agent-core packaging specs to avoid
changing the rootDir name of the archives.
The qualifier had been removed in PR #7690 trying to use the spec name:
this worked to get the desired file name but changed the root Dir name
which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the
template definition instead of '{{.Name}}' which would render the spec
name.

* Modify fips core spec qualifier and name (#7818)

* Reintroduce cloud-defend component

* Filter components by package-type

---------

Co-authored-by: Paolo Chilà <[email protected]>

* fix: 8.18.5 snapshot DRA

* fix: resolve QF1004: could use strings.ReplaceAll linter errors

---------

Co-authored-by: Panos Koutsovasilis <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis added a commit that referenced this pull request Aug 8, 2025
* ci: build agent from snapshot DRA (#9048)

* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.buildkite/integration.pipeline.yml
#	.buildkite/scripts/steps/ess.ps1
#	.package-version
#	dev-tools/mage/manifest/manifest.go

* [8.x](backport #7690) Fips packaging (#7790)

* Make components in packages configurable (#7602)

* Redefine ExpectedBinaries as YAML config

* Move ExpectedBinaries closer to package spec file

* Fix error formatting in downloadDRAArtifacts

* add packageName template to ExpectedPackages

* use a relaxed dependencies version for IAR releases

* Remove FIPS hack introduced in PR #7486

* Allow for a looser match on relaxing dependencies versions

* Add debug logging when packaging with EXTERNAL=true

* move package tests to dedicated package

* Fips packaging (#7690)

* Add component list to specs

* extract component dependencies from the packages to be built

* Refactor component extraction from package specs

* Fix package tests error handling

* Inject dependencies and remove references to ExpectedBinaries

* Remove ExpectedBinaries global

* Add rootdir to components

* Extract actual version matched on the package file and use it to render RootDir

* Package elastic-agent FIPS specs when FIPS=true is specified

* refactor ResolveManifestPackage

* Move FIPS compile settings in packages.yml

* Add more FIPS components

* Properly handle dependenciesVersion when calling mage package

* Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files

* Rework useDRAAgentBinaryForPackage for repackaging agent

Define elastic-agent-core components (both FIPS and non-FIPS variants)
and define package name and root dir templates.
Implement some filtering on component list to extract the correct
component definition according to the FIPSBuild flag.
Refactor code that downloads pre-compiled elastic-agent binaries and
places them in the golangCrossBuild directory to make use of the new
component definition.

* Write spec FIPS flag into manifest.yaml when packaging

* Add FIPS elastic agent basic and cloud docker images

* Build FIPS docker images in CI packaging

* Fix FIPS .tar.gz package tests

* Restructure package tests

* Extend FIPS check to all binaries in components directory

* Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline

* Cleanup ChecksumsWithManifest and improve godoc

* Improve godoc for BinarySpec

* Correctly inject dependency list when packaging using DROP_PATH (#7795)

* Restore qualifier=core for elastic-agent-core packaging specs (#7805)

Restore qualifier for elastic-agent-core packaging specs to avoid
changing the rootDir name of the archives.
The qualifier had been removed in PR #7690 trying to use the spec name:
this worked to get the desired file name but changed the root Dir name
which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the
template definition instead of '{{.Name}}' which would render the spec
name.

* Modify fips core spec qualifier and name (#7818)

* Reintroduce cloud-defend component

* Filter components by package-type

---------

Co-authored-by: Paolo Chilà <[email protected]>

* fix: point .package-version to latest 8.17 SNAPSHOT DRA

* fix: resolve QF1004: could use strings.ReplaceAll linter errors

* fix: remove unused var hintsInputsDFilePattern

---------

Co-authored-by: Panos Koutsovasilis <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis added a commit that referenced this pull request Aug 8, 2025
* ci: build agent from snapshot DRA (#9048)

* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA

* feat: incorporate USE_PACKAGE_VERSION in mage

* experiment: bump version.go

* Revert "experiment: bump version.go"

This reverts commit a57ee10.

* chore: bump .package-version

* feat: allow AGENT_VERSION to be overridden by env var

* fix: use named args for all args in integration_tests_tf.ps1

* feat: panic on err of initPackageVersion

* fix: don't panic when .package-version file doesn't exist, log it instead

* feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE

* feat: use os.WriteFile in writePackageVersion

* chore: bump to latest snapshot DRA

* fix: always DownloadManifest if PackagingFromManifest is set in mage package

* fix: check err of filepath.Abs(dropPath)

(cherry picked from commit a155660)

# Conflicts:
#	.package-version
#	testing/integration/ess/upgrade_broken_package_test.go

* fix: resolve conflicts

* fix: remove redundant agent version pinning

* fix: bump beat to the commit of the snapshot DRA in .package-version

* fix: address beats breaking API func calls

* fix: mage fmt

---------

Co-authored-by: Panos Koutsovasilis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-active-all Automated backport with mergify to all the active branches skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Testing:run:TestUpgradeIntegrationsServer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants