[META] Create build pipelines and packaging steps for all Elastic Agent subcomponents

[jlind23]

Purpose of this issue is to design and implement CI pipelines for all the components below:

• Elastic Agent Shipper: https://github.com/elastic/elastic-agent-shipper
• Elastic Agent Inputs: https://github.com/elastic/elastic-agent-inputs

Revisit the proposal to improve the agent build system. Ensure the agent build and packaging system can scale with the agent V2 architecture and make it easier to use. In particular, consider that in the new architecture there may be many more inputs that live in separate repositories, written in different languages.

This ticket will considered as done as soon as the following actions are working:

• Each component can be individually built
• Each component are tested through unit, integration, and end to end tests that are part of the CI steps.
• A release process exists for each component. Each build generate artifacts that are stored and made available to be downloaded and packaged with the agent.
• Each component is appropriately packaged and distributed with the agent
• Builds must be a runnable on a PR basis

Elastic Agent shipper work - elastic/elastic-agent-shipper#74

[cmacknz]

Added some more detail to the description.

[mdelapenya]

Each component are tested through unit, integration, and end to end tests that are part of the CI steps.

I'd like to share my two cents about the different types of testing for each repository/component here. IIUC, there will be libraries (shipper, inputs) that will be consumed by the elastic-agent. For me, the libraries should contain a great coverage on unit tests, but no e2e or integration tests, unless they communicate to other system, which I think it's not the case (please correct me if I'm wrong 🙏 ).

Regarding the elastic-agent, which for me is the central piece here, it consumes the libraries to perform certain actions and send data to an output (elasticsearch). This is the project where, apart form the mandatory unit tests, I'd put integration tests. But for that I'd require first to define the boundaries for all possible interactions:

• agent + shipper
• agent + inputs
• agent + shipper + inputs
• agent + input > output 1 (i.e. elasticsearch)
• agent + shipper > output 1 (i.e elasticsearch)
• ...

It would be of great help having a dependency graph where we can identify consumers and providers of the functionality, so that we can correlate the relationships with those boundaries that I mentioned before, and with that, write the right type of tests for each interaction.

[cmacknz]

At a high level I agree that the agent is the place that aggregates all of this functionality together, and we could write all tests at the agent level.

At an organizational level most of these pieces are by developed by separate teams in separate repositories, so I'm not sure making the agent the place where all integration testing happens would make sense. If we wanted to do that it would probably be easiest to just have an agent monorepo.

I think we will want agent components (inputs, shipper) to have some form of integration testing local to their own repositories that ensures they are highly likely to run under agent without issue.

[mdelapenya]

I think we will want agent components (inputs, shipper) to have some form of integration testing local to their own repositories that ensures they are highly likely to run under agent without issue.

Yes, that's correct, and I totally agree with you about having integration tests at the library level. That's what I tried to express myself when I said:

the libraries should contain a great coverage on unit tests, but no e2e or integration tests, unless they communicate to other system

We probably want to test at the integration layer that a library interacts with components A, B, C, and because of your comment, I think that's the case, so I'm super happy that we are aligned in the test strategy :)

[narph]

Based on the information above and @ph' s proposal, I took a stab at creating some broad and concrete tasks separated by repo. This should help us estimate on how much effort is involved and prioritize for the next releases.
Let me know if I am missing anything else.

Elastic Agent

• clean up current build in preparation for the new way of sourcing binaries (will follow up with detailed proposal)
  • re-evaluate/scrub dev-tools to a bare minimum, decide if anything can be brought to a new solution
  • implement new solution for build (will no longer attempt to build dependencies)
  • reduce complexity, make everything explicit, no longer rely on env variables, strict validation, create migration plan from make to mage, rearchitect to strict set of common targets
• make sure elastic agent can still work with established artifacts from other teams (cloudbeat, apm-server, endpoint-security) for now
• update infra repo elastic-agent templates to reflect the new setup (https://github.com/elastic/release-eng/issues/283)

Beats

• add new magefile containing the new build target that should produce the new beats binaries in the desired format
• update infra repo beats related templates to also publish the new binaries in the new directory

Shipper (elastic/elastic-agent-shipper#74)

• expand build implementation to generate platform based binaries
• align to stack versioning
• update/expand .CI setup
• update infra repo, add shipper as new unified release manager artifact (https://github.com/elastic/release-eng/issues/268)

Inputs

• add to fleet-ci
• same step list as shipper above (related issue Create a minimal packaging process elastic-agent-inputs#5)

Fleet Server

• update build to produce build binaries
• update infra repo fleet server related templates to publish the new binaries

Shared

• ensure tests are run if present
• ensure e2e tests are running

[ph]

This is a really a good starts @narph, could you prioritize the above list and we can work with @pierrehilbert and the team to assign release to them items, I don't think we we can do everything for 8.4. I think experimenting in the shipper is a good way to have the necessary information for the proposal since this need to be done and has little impact to the actual build process.

@andrewkroh fyi, you might be interested to follow this, the main idea is to reduce the complexity of the build of the Elastic Agent and the Beats, we will also separate the build toolings from the contract of what need to be produced by the build.

[cmacknz]

Shipper

expand build implementation to generate platform based binaries
align to stack versioning
update/expand .CI setup
update infra repo, add shipper as new unified release manager artifact

Do we need the shipper to be part of the unified release manager? It will only run as part of the agent. It and the new inputs (from elastic-agent-inputs) can probably be treated the same way fleet-server is from the perspective of releasing the binaries.

[ph]

@cmacknz It need to be in some part, the macOS or Windows binary need to be signed.