Fix reproducability of builds against Java EA versions #132847

breskeby · 2025-08-13T15:43:49Z

This fixes a reproducability issue when using the gradle javaToolChain api.

There is no way to test toolchain candidates reliable against the build number in use.

This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection
does not detect the difference between certain builds (or even ea vs. released version)
Therefore we fallback to rely on our custom JDKDownloadPlugin for now here.

Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution
ones something is available. Ultimately we want to get rid of usages of the JDK download plugin.

elasticsearchmachine · 2025-08-13T15:44:15Z

Pinging @elastic/es-delivery (Team:Delivery)

breskeby · 2025-08-13T15:49:12Z

...rg/elasticsearch/gradle/internal/toolchain/EarlyAccessCatalogJdkToolchainResolverSpec.groovy

+import static org.gradle.platform.Architecture.X86_64
+import static org.gradle.platform.OperatingSystem.*
+
+class EarlyAccessCatalogJdkToolchainResolverSpec extends AbstractToolchainResolverSpec {


This is not set active for now.

breskeby · 2025-08-13T15:49:44Z

test/immutable-collections-patch/build.gradle


  classpath = sourceSets.main.runtimeClasspath
  mainClass = 'org.elasticsearch.jdk.patch.ImmutableCollectionsPatcher'
-  if (buildParams.getIsRuntimeJavaHomeSet()) {


I think this should not use the runtime jdk but whatever we use for compiling.

IIRC we use the runtime jdk because it's trying to match what we will actually run with (it's just for use with tests after all).

one issue I faced with this was that the asm patcher failed due to unexpected class file version 🤔

the patcher relies on looking at implementation class details within the jdk itself. So I think the runtime jdk is correct, the classes we patch won't necessarily be the same with the compile jdk, we want to actually patch the classes that we are about to run with.

I've changed it back. I guess we need an asm update here? https://gradle-enterprise.elastic.co/s/biwfet3luxnxe/console-log/task/:test:immutable-collections-patch:generatePatch?anchor=5&page=1

breskeby · 2025-08-13T15:50:39Z

BUILDING.md


 To test an unreleased development version of a third party dependency you have several options.

+### How do I test against java early access (ea) versions?


Try maintaining a good FAQ here :D

rjernst

Looks pretty good. I just wonder why we are adding a toolchain resolver if we aren't actually using it.

rjernst · 2025-08-13T16:16:39Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

+        Integer runtimeJavaProperty = Integer.getInteger("runtime.java");
        if (runtimeJavaProperty != null) {
-            return resolveJavaHomeFromToolChainService(runtimeJavaProperty);
+            if (runtimeJavaProperty > Integer.parseInt(VersionProperties.getBundledJdkMajorVersion())) {


this isn't necessarily true, eg right now we want to try to run with JDK 25 RC builds, this version is grearter than the bundled, but it's not an EA.

Could we instead have runtime.java take in eg 26-ea and we detect and parse out the -ea?

ah interesting. yeah we could do something like that. I'll take a look. Are there RCs of upcoming java versions available already we can test against the behavior of resolving RCs?

Java 25 has RCs available right now.

rjernst · 2025-08-13T16:19:54Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

+        NamedDomainObjectContainer<Jdk> container = project.getPlugins().apply(JdkDownloadPlugin.class).getContainer(project);
+        Integer buildNumber = Integer.getInteger("runtime.java.build");
+        if (buildNumber == null) {
+            buildNumber = Integer.getInteger("runtime.java." + runtimeJavaProperty + ".build");


Is there a reason we would have multiple simultaneous build numbers? Could we get by with just runtime.java.build?

Agreed, lets decide on a single convention here.

rjernst · 2025-08-13T16:21:51Z

gradle/verification-metadata.xml

         <trust group="beats" name="metricbeat-fips"/>
         <trust group="elasticsearch-distribution" name="elasticsearch"/>
+         <trust group="openjdk_25"/>
+         <trust group="openjdk_26"/>


will we need to add this for every new major version?

I think so atm. I don't see regex working here. I'm hopeful gradle has a better solution by reaching 27

Looks like the docs say regex is supported like so:

<trust group="^com[.]mycompany($|([.].*))" regex="true" reason="We trust all mycompany artifacts"/>

https://docs.gradle.org/current/userguide/dependency_verification.html#sec:trusting-artifacts

rjernst · 2025-08-13T16:23:24Z

test/immutable-collections-patch/build.gradle


  classpath = sourceSets.main.runtimeClasspath
  mainClass = 'org.elasticsearch.jdk.patch.ImmutableCollectionsPatcher'
-  if (buildParams.getIsRuntimeJavaHomeSet()) {


IIRC we use the runtime jdk because it's trying to match what we will actually run with (it's just for use with tests after all).

rjernst · 2025-08-13T16:24:04Z

...java/org/elasticsearch/gradle/internal/toolchain/EarlyAccessCatalogJdkToolchainResolver.java

+ * this only supports resolving the latest early access build for a given language version.
+ * <p>
+ */
+public abstract class EarlyAccessCatalogJdkToolchainResolver extends AbstractCustomJavaToolchainResolver {


Why is this here if we aren't going to use the toolchains?

breskeby · 2025-08-13T16:52:41Z

Looks pretty good. I just wonder why we are adding a toolchain resolver if we aren't actually using it.

it resolves the latest availble EA version. I wanted to keep it and have it tested as we come back to this. also I removed the EA support from the default oracle provider. we test this one though we don't use it so it should keep working.

mark-vieira · 2025-08-14T16:39:38Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

    }

+    private Provider<File> resolveEarlyAccessJavaHome(Integer runtimeJavaProperty) {
+        NamedDomainObjectContainer<Jdk> container = project.getPlugins().apply(JdkDownloadPlugin.class).getContainer(project);


Applying the plugin like this seems wrong but maybe my spidey sense is tingling here for no good reason.

yeah feeling dirty. fixed it :D

mark-vieira · 2025-08-14T16:40:32Z

...ols-internal/src/main/java/org/elasticsearch/gradle/internal/info/GlobalBuildInfoPlugin.java

+        NamedDomainObjectContainer<Jdk> container = project.getPlugins().apply(JdkDownloadPlugin.class).getContainer(project);
+        Integer buildNumber = Integer.getInteger("runtime.java.build");
+        if (buildNumber == null) {
+            buildNumber = Integer.getInteger("runtime.java." + runtimeJavaProperty + ".build");


Agreed, lets decide on a single convention here.

mark-vieira · 2025-08-14T16:45:24Z

gradle/verification-metadata.xml

         <trust group="beats" name="metricbeat-fips"/>
         <trust group="elasticsearch-distribution" name="elasticsearch"/>
+         <trust group="openjdk_25"/>
+         <trust group="openjdk_26"/>


Looks like the docs say regex is supported like so:

<trust group="^com[.]mycompany($|([.].*))" regex="true" reason="We trust all mycompany artifacts"/>

https://docs.gradle.org/current/userguide/dependency_verification.html#sec:trusting-artifacts

This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin.

explicitly requires to add a ea postfix to the runtime sys property

elasticsearchmachine · 2025-08-15T11:39:28Z

💔 Backport failed

Status	Branch	Result
❌	8.17	Commit could not be cherrypicked due to conflicts
❌	8.18	Commit could not be cherrypicked due to conflicts
❌	9.0	Commit could not be cherrypicked due to conflicts
✅	9.1
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 132847

* Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup

* Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup (cherry picked from commit 16fe7db) # Conflicts: # BUILDING.md

* Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup (cherry picked from commit 16fe7db) # Conflicts: # BUILDING.md # build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/JdkDownloadPluginFuncTest.groovy

…2974) * Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup

* Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup (cherry picked from commit 16fe7db) # Conflicts: # BUILDING.md # build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/JdkDownloadPluginFuncTest.groovy # build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/toolchain/OracleOpenJdkToolchainResolver.java # build-tools-internal/src/test/groovy/org/elasticsearch/gradle/internal/toolchain/OracleOpenJdkToolchainResolverSpec.groovy # test/framework/src/main/java/org/elasticsearch/test/junit/listeners/ReproduceInfoPrinter.java

breskeby · 2025-08-15T12:47:14Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19
✅	8.18
✅	8.17

Questions ?

Please refer to the Backport tool documentation

…-stats * upstream/main: (36 commits) Fix reproducability of builds against Java EA versions (elastic#132847) Speed up loading keyword fields with index sorts (elastic#132950) Mute org.elasticsearch.index.mapper.LongFieldMapperTests testSyntheticSourceWithTranslogSnapshot elastic#132964 Simplify EsqlSession (elastic#132848) Implement WriteLoadConstraintDecider#canAllocate (elastic#132041) Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {p0=search/400_synthetic_source/_doc_count} elastic#132965 Switch to PR-based benchmark pipeline defined in ES repo (elastic#132941) Breakdown undesired allocations by shard routing role (elastic#132235) Implement v_magnitude function (elastic#132765) Introduce execution location marker for better handling of remote/local compatibility (elastic#132205) Mute org.elasticsearch.cluster.ClusterInfoServiceIT testMaxQueueLatenciesInClusterInfo elastic#132957 Unmuting simulate index data stream mapping overrides yaml rest test (elastic#132946) Remove CrossClusterCancellationIT.createLocalIndex() (elastic#132952) Mute org.elasticsearch.index.mapper.LongFieldMapperTests testFetch elastic#132956 Fix failing UT by adding a required capability (elastic#132947) Precompute the BitsetCacheKey hashCode (elastic#132875) Adding simulate ingest effective mapping (elastic#132833) Mute org.elasticsearch.index.mapper.LongFieldMapperTests testFetchMany elastic#132948 Rename skipping logic to remove hard link to skip_unavailable (elastic#132861) Store ignored source in unique stored fields per entry (elastic#132142) ...

…2975) * Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup (cherry picked from commit 16fe7db) # Conflicts: # BUILDING.md

… (#132978) * Fix reproducability of builds against Java EA versions (#132847) * Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup (cherry picked from commit 16fe7db) # Conflicts: # BUILDING.md # build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/JdkDownloadPluginFuncTest.groovy * Fix jdkdownload tests

… (#132979) * Fix reproducability of builds against Java EA versions (#132847) * Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup (cherry picked from commit 16fe7db) # Conflicts: # BUILDING.md # build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/JdkDownloadPluginFuncTest.groovy # build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/toolchain/OracleOpenJdkToolchainResolver.java # build-tools-internal/src/test/groovy/org/elasticsearch/gradle/internal/toolchain/OracleOpenJdkToolchainResolverSpec.groovy # test/framework/src/main/java/org/elasticsearch/test/junit/listeners/ReproduceInfoPrinter.java * Fix merge issues

… (#132977) * Fix reproducability of builds against Java EA versions (#132847) * Introduce dedicated toolchain resolver for EA java versions * Fix reproducability of builds against Java EA versions This fixes a reproducability issue when using the gradle javaToolChain api. There is no way to test toolchain candidates reliable against the build number in use. This meant that ones you e.g. have resolved some version of java 25, gradle toolchain detection does not detect the difference between certain builds (or even ea vs. released version) Therefore we fallback to rely on our custom JDKDownloadPlugin for now here. Syncing with the gradle team, they want to fix this somewhen in 9.x. We will revisit our solution ones something is available. Ultimately we want to get rid of usages of the JDK download plugin. * Minor cleanup and fixing non set distribution type in jdk * Fix ea jdk selection not just based on major version explicitly requires to add a ea postfix to the runtime sys property * Fix test reproduce linies when using java ea version * Apply review feedback * Use runtime jdk when patching * More cleanup (cherry picked from commit 16fe7db) # Conflicts: # BUILDING.md # build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/JdkDownloadPluginFuncTest.groovy * Fix jdkdownload tests

breskeby requested a review from a team as a code owner August 13, 2025 15:43

breskeby added >non-issue :Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team auto-backport Automatically create backport pull requests when merged v9.2.0 v8.17.11 v8.18.6 v9.0.6 v9.1.3 v8.19.3 labels Aug 13, 2025

breskeby self-assigned this Aug 13, 2025

breskeby changed the title ~~Introduce dedicated toolchain resolver for EA java versions~~ Fix reproducability of builds against Java EA versions Aug 13, 2025

breskeby commented Aug 13, 2025

View reviewed changes

breskeby requested review from JVerwolf and rjernst August 13, 2025 15:51

breskeby force-pushed the fix-java-ea-reproducability branch from 67b1e66 to 8cfd01c Compare August 13, 2025 16:04

rjernst reviewed Aug 13, 2025

View reviewed changes

breskeby force-pushed the fix-java-ea-reproducability branch 3 times, most recently from 8fce476 to 14255a2 Compare August 14, 2025 08:36

mark-vieira approved these changes Aug 14, 2025

View reviewed changes

breskeby force-pushed the fix-java-ea-reproducability branch 2 times, most recently from d241b51 to 1a0cd1c Compare August 14, 2025 21:29

breskeby added 2 commits August 14, 2025 23:31

Introduce dedicated toolchain resolver for EA java versions

5caaccd

breskeby added 6 commits August 14, 2025 23:31

Minor cleanup and fixing non set distribution type in jdk

bbd4287

Fix ea jdk selection not just based on major version

ba8882d

explicitly requires to add a ea postfix to the runtime sys property

Fix test reproduce linies when using java ea version

81b56f2

Apply review feedback

a2bc8fe

Use runtime jdk when patching

4610d7b

More cleanup

487a053

breskeby force-pushed the fix-java-ea-reproducability branch from 1a0cd1c to 487a053 Compare August 14, 2025 21:31

Merge branch 'main' into fix-java-ea-reproducability

aa4827e

breskeby merged commit 16fe7db into elastic:main Aug 15, 2025
33 of 34 checks passed

breskeby mentioned this pull request Aug 15, 2025

[9.1] Fix reproducability of builds against Java EA versions (#132847) #132974

Merged

elasticsearchmachine added the backport pending label Aug 15, 2025

breskeby mentioned this pull request Aug 15, 2025

[9.0] Fix reproducability of builds against Java EA versions (#132847) #132975

Merged

breskeby mentioned this pull request Aug 15, 2025

[8.19] Fix reproducability of builds against Java EA versions (#132847) #132977

Merged

breskeby mentioned this pull request Aug 15, 2025

[8.18] Fix reproducability of builds against Java EA versions (#132847) #132978

Merged

breskeby mentioned this pull request Aug 15, 2025

[8.17] Fix reproducability of builds against Java EA versions (#132847) #132979

Merged


		To test an unreleased development version of a third party dependency you have several options.

		### How do I test against java early access (ea) versions?

Fix reproducability of builds against Java EA versions #132847

Fix reproducability of builds against Java EA versions #132847

Uh oh!

Conversation

breskeby commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rjernst left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

breskeby commented Aug 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 15, 2025

💔 Backport failed

Uh oh!

breskeby commented Aug 15, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

breskeby commented Aug 13, 2025 •

edited

Loading