Add Reason field to elastic-agent upgrade details metadata

[pchila]

Add "reason" text field to map a new upgrade details field introduced in PR elastic/elastic-agent#8407.

There is no backport needed as the feature will be released for the first time in 9.2.0.

Edit: it seems that we need a version bump for past updates to the x-pack/plugin/core/template-resources/src/main/resources/fleet-agents.json file that did not bump the schema version (see discussion below).
If we want to use the bump included in this PR, we need to backport this to all active branches
cc @ebeahan @cmacknz @pkoutsovasilis

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Hi @pchila, I've created a changelog YAML for you.

[juliaElastic]

We should update the schema version here:

elasticsearch/x-pack/plugin/fleet/src/main/java/org/elasticsearch/xpack/fleet/Fleet.java

Line 82 in c47522c

private static final int FLEET_AGENTS_MAPPINGS_VERSION = 3;

Context: #119087

[pchila]

We should update the schema version here:

elasticsearch/x-pack/plugin/fleet/src/main/java/org/elasticsearch/xpack/fleet/Fleet.java

Line 82 in c47522c

private static final int FLEET_AGENTS_MAPPINGS_VERSION = 3;

Context: #119087

@juliaElastic
Fixed in a1163f1

[juliaElastic]

LGTM

[pkoutsovasilis]

@juliaElastic 👋

I noticed that @pchila bumped FLEET_AGENTS_MAPPINGS_VERSION in this PR (after your comment), but in my previous PR #127651 (which you reviewed) where I added a new field, I didn't bump this version.

Should we retroactively bump FLEET_AGENTS_MAPPINGS_VERSION to cover those previous changes, or has this version bump only now become necessary?

[juliaElastic]

@pkoutsovasilis AFAIK the version field should be bumped after every change, it looks like we missed it in #127651.
We should probably bump the version for the backport branches for #127651

[pkoutsovasilis]

Actually @juliaElastic, we didn’t just miss this once — looking at the file history there are multiple commits where FLEET_AGENTS_MAPPINGS_VERSION wasn’t bumped:

• 33fc566 (missed)
• e822bc4 (bumped)
• a859f37 (missed)
• 6117925 (missed)
• 28f4d87 (missed)
• ffc22b2 (missed)
• 76dc0eb (missed)
• 1fea0d7 (missed)
• 27f31b1 (missed)
• 584d7ea (missed)

If this bump was truly critical, I’d expect the missing bumps in those commits (where we added new fields) to have triggered a large number of SDHs — but I don’t think that has happened.

Just to make sure we handle this correctly going forward; are we certain this bump is strictly required for every field addition? I’d like to double-check the actual impact of missing bumps before we retroactively update all the backport branches.

cc @ebeahan @cmacknz

[juliaElastic]

I think it's not necessarily causing an issue for every change.
If we miss bumping the version, existing deployments won't get the updated mappings on stack upgrade. It can cause a problem if we rely on the mapping for a feature, e.g. we needed the keyword mapping to use a new field for searching in space awareness feature: elastic/kibana#204834

[pkoutsovasilis]

It can cause a problem if we rely on the mapping for a feature, e.g. we needed the keyword mapping to use a new field for searching in space awareness feature: elastic/kibana#204834

I get that point - if we added fields that were optional or not relied upon, missing the bump wouldn’t be a big deal.
However, most of what we’ve added (looking at the commits again) is critical for parsing data correctly, so IMO every change probably deserves a bump to avoid subtle issues later.

I’ll leave it to @ebeahan @cmacknz to weigh in on the actual impact and whether we should retroactively bump in backports - I might be missing some of the fine-print details here.

[cmacknz]

I’ll leave it to @ebeahan @cmacknz to weigh in on the actual impact and whether we should retroactively bump in backports - I might be missing some of the fine-print details here.

Yes I strongly suspect missing this is the reason we've seen SDHs around the complete field at least, we never found out why it suddenly started appearing but this missing revision bump coupled with something starting to use it in the UI (perhaps synthetics UI?) would do it.

Some of the other missing updates aren't used widely yet (e.g. fips). Better safe than sorry on this.

Also this is a pretty big foot gun so it would be ideal if we can figure out some way to have this flagged in CI. I am not familiar enough with ES changes to know what to do about that or if it's even possible. I suspect that relying on this revision instead of having a diff mechanism detect the need for it is a problem.

[cmacknz]

If we want to use the bump included in this PR, we need to backport this to all active branches

My vote is we backport this PR to avoid any issues where we have the same revision number for different change sets. I'm not sure if this would be a problem but I'd rather just avoid it entirely, adding the field here is low enough risk to backport.

[ebeahan]

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	9.0
✅	8.19
✅	8.18

Questions ?

Please refer to the Backport tool documentation

[juliaElastic]

Asked around in es-core-infra channel, there is an open ES issue to catch missing version bumps in CI: https://elasticco.atlassian.net/browse/ES-10832