[8.x](backport #7690) Fips packaging #7790
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mergify
bot
added
backport
conflicts
mergify
bot
requested review from
blakerouse and
pkoutsovasilis
and removed request for
a team
|
Cherry-pick of 6abd585 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
github-actions
bot
added
Team:Elastic-Agent-Control-Plane
|
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane) |
|
Pinging @elastic/elastic-agent (Team:Elastic-Agent) |
pchila
force-pushed
the
mergify/bp/8.x/pr-7690
branch
from
a246e8d to
38d1c96
Compare
pchila
force-pushed
the
mergify/bp/8.x/pr-7690
branch
2 times, most recently
from
263d54e to
dd4798e
Compare
* Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package
* Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec
Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name.
pchila
force-pushed
the
mergify/bp/8.x/pr-7690
branch
from
dd4798e to
2c12bea
Compare
|
This pull request has not been merged yet. Could you please review and merge it @pchila? 🙏 |
|
buildkite test this |
pchila
force-pushed
the
mergify/bp/8.x/pr-7690
branch
from
2c12bea to
953a7ff
Compare
|
💚 Build Succeeded
History
cc @pchila |
|
Test packaging build https://buildkite.com/elastic/elastic-agent-package/builds/5384 |
pkoutsovasilis
approved these changes
Apr 16, 2025
8 tasks
pkoutsovasilis
pushed a commit
that referenced
this pull request
Aug 7, 2025
* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]> # Conflicts: # .buildkite/integration.pipeline.yml # dev-tools/mage/checksums.go # dev-tools/mage/dockerbuilder.go # dev-tools/mage/manifest/manifest.go # dev-tools/mage/pkgtypes.go # dev-tools/mage/settings.go # dev-tools/packaging/packages.yml # dev-tools/packaging/testing/package_test.go # testing/integration/ess/upgrade_standalone_same_commit_test.go
pkoutsovasilis
pushed a commit
that referenced
this pull request
Aug 7, 2025
* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis
pushed a commit
that referenced
this pull request
Aug 7, 2025
* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis
pushed a commit
that referenced
this pull request
Aug 7, 2025
* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis
pushed a commit
that referenced
this pull request
Aug 7, 2025
* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis
pushed a commit
that referenced
this pull request
Aug 7, 2025
* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis
added a commit
that referenced
this pull request
Aug 8, 2025
* ci: build agent from snapshot DRA (#9048) * feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA * feat: incorporate USE_PACKAGE_VERSION in mage * experiment: bump version.go * Revert "experiment: bump version.go" This reverts commit a57ee10. * chore: bump .package-version * feat: allow AGENT_VERSION to be overridden by env var * fix: use named args for all args in integration_tests_tf.ps1 * feat: panic on err of initPackageVersion * fix: don't panic when .package-version file doesn't exist, log it instead * feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE * feat: use os.WriteFile in writePackageVersion * chore: bump to latest snapshot DRA * fix: always DownloadManifest if PackagingFromManifest is set in mage package * fix: check err of filepath.Abs(dropPath) (cherry picked from commit a155660) # Conflicts: # .buildkite/integration.pipeline.yml # .buildkite/scripts/steps/ess.ps1 # .package-version # dev-tools/mage/manifest/manifest.go * [8.x](backport #7690) Fips packaging (#7790) * Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]> * fix: 8.18.5 snapshot DRA * fix: resolve QF1004: could use strings.ReplaceAll linter errors --------- Co-authored-by: Panos Koutsovasilis <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Paolo Chilà <[email protected]>
pkoutsovasilis
added a commit
that referenced
this pull request
Aug 8, 2025
* ci: build agent from snapshot DRA (#9048) * feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA * feat: incorporate USE_PACKAGE_VERSION in mage * experiment: bump version.go * Revert "experiment: bump version.go" This reverts commit a57ee10. * chore: bump .package-version * feat: allow AGENT_VERSION to be overridden by env var * fix: use named args for all args in integration_tests_tf.ps1 * feat: panic on err of initPackageVersion * fix: don't panic when .package-version file doesn't exist, log it instead * feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE * feat: use os.WriteFile in writePackageVersion * chore: bump to latest snapshot DRA * fix: always DownloadManifest if PackagingFromManifest is set in mage package * fix: check err of filepath.Abs(dropPath) (cherry picked from commit a155660) # Conflicts: # .buildkite/integration.pipeline.yml # .buildkite/scripts/steps/ess.ps1 # .package-version # dev-tools/mage/manifest/manifest.go * [8.x](backport #7690) Fips packaging (#7790) * Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <[email protected]> * fix: point .package-version to latest 8.17 SNAPSHOT DRA * fix: resolve QF1004: could use strings.ReplaceAll linter errors * fix: remove unused var hintsInputsDFilePattern --------- Co-authored-by: Panos Koutsovasilis <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Paolo Chilà <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR backports FIPS packaging support for elastic-agent.
This change is the combination of 5 PRs:
#7602
#7690
#7795
#7805
#7818
There are 2 extra commits added on top of the 5 backported PRs that deal with
cloud-defendpackaging.What follows is the description of PR #7690:
When specifying "FIPS=true" the agent will pull FIPS-enable components, compile/download a FIPS-enabled agent executable, package only specs that are marked with
fips: trueindev-tools/packaging/packages.yml.It is also possible to package FIPS-compliant
elastic-agent-coreartifact specifyingFIPS=trueIn order to download and use a different set of dependencies, components are now defined in
dev-tools/packaging/packages.ymland referenced directly in the packaging specs.Elastic Agent packaging code will collect all the enabled specs, compute the full set of components needed and pass that list to the download/unpack/flatten/compress steps.
The high-level packaging flow looks like this:
dev-tools/packaging/packages.ymlaccording toPLATFORMS,PACKAGES,FIPSfiltersBinarySpec.GetPackageName()method to locate the correct fileBinarySpec.GetRootDir()packagedirectoryelastic-agent-coreproject with a similar mechanism to the one used for the dependenciesfipsvalue.This PR introduces the new packages below:
elastic-agent-fips-9.1.0-SNAPSHOT-linux-*.tar.gzarchiveelastic-agent-fips-9.1.0-SNAPSHOT-linux-*.docker.tar.gz/docker.elastic.co/elastic-agent/elastic-agent-fips:9.1.0-SNAPSHOTdocker imageelastic-agent-fips-cloud-9.1.0-SNAPSHOT-linux-*.docker.tar.gz/docker.elastic.co/beats-ci/elastic-agent-fips-cloud:9.1.0-SNAPSHOTdocker imageelastic-agent-core-fips-9.1.0-linux-*.tar.gzpackageWhy is it important?
To have FIPS-compliant Elastic Agent packages that contain only FIPS-compliant binaries.
Checklist
[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files[ ] I have added tests that prove my fix is effective or that my feature works[ ] I have added an entry in./changelog/fragmentsusing the changelog tool[ ] I have added an integration test or an E2E testDisruptive User Impact
No disruptive user impact as existing packages (and the command line to package elastic-agent) should be unchanged.
How to test this PR locally
Package a linux/amd64 elastic-agent tar.gz package using local elastic-agent code
Run:
Sample output:
Package a linux/amd64 elastic-agent-core tar.gz package using local elastic-agent code
Run:
FIPS=true SNAPSHOT=true PLATFORMS="linux/amd64" mage clean packageAgentCoreSample output:
Package a linux/amd64 elastic-agent-core tar.gz and docker images using local elastic-agent code
Run:
Sample output:
Package a linux/amd64 elastic-agent tar.gz package using a DRA manifest, downloading the EA executable from it (no local compilation)
We need a small hack as the
elastic-agent-core-fips-*artifacts will only be available after this PR is merged: flipfipsflag for components&comp-elastic-agent-corehere and&comp-elastic-agent-core-fipshere. This will force the packaging code to select the non-FIPS compliant elastic-agent-core artifact and complete the packaging (without this the packaging will fail because theelastic-agent-core-fips-*cannot be downloaded)After modifying the components definitions, run:
Sample output:
Related issues
Questions to ask yourself
This is an automatic backport of pull request #7690 done by [Mergify](https://mergify.com).