HADOOP-19212. (3.4) Hadoop UGI compatible with Java 25

[pan3793]

Port #7886 to branch-3.4

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 12s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	2m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 40s		branch-3.4 passed
+1 💚	compile	8m 59s		branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	8m 52s		branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	checkstyle	0m 45s		branch-3.4 passed
+1 💚	mvnsite	1m 34s		branch-3.4 passed
+1 💚	javadoc	1m 3s		branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 51s		branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	spotbugs	2m 25s		branch-3.4 passed
+1 💚	shadedclient	22m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	9m 14s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	9m 14s		the patch passed
+1 💚	compile	10m 11s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	javac	10m 11s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 49s		the patch passed
+1 💚	mvnsite	1m 25s		the patch passed
+1 💚	javadoc	0m 56s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 52s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	spotbugs	2m 7s		the patch passed
+1 💚	shadedclient	21m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 48s		hadoop-auth in the patch passed.
+1 💚	unit	16m 31s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		151m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7897/1/artifact/out/Dockerfile
GITHUB PR	#7897
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ecc8f405f067 5.15.0-142-generic #152-Ubuntu SMP Mon May 19 10:54:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 4e66b56
Default Java	Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7897/1/testReport/
Max. process+thread count	1271 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common U: hadoop-common-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7897/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[slfan1989]

@steveloughran We want to backport this PR to branch-3.4. Do you think that’s okay?

[slfan1989]

@pan3793 Thanks for the contribution! From my personal perspective, this is a relatively important feature, and I suggest backporting it to branch-3.4. If anyone has different opinions, please feel free to discuss.

cc: @stoty

[stoty]

It's already merged :)

I'm fine with this backport.

The question whether we want to fully support Java 18+ in 3.4.3.

I was working on the assumption that support for Java 18+ was only targeted for 3.5.0.

This is just one (admittedly critical) part of support for newer Javas, I alone have more than a dozen Java 25 patches in trunk which were never backported to 3.4, and the large patch for Subject propagation in threads is still WIP.

IMO None of those patches are particularly controversial, though backporting them would take some work.

The options are:

• Hold back 3.4.2 for this patch - Consensus seems to be not do that (and I kind of agree)
• Merge just this patch, and release in 3.4.3 - This seems to be enough for at least the Spark tests to pass
• Backport full Java 25 support and release in 3.4.3 - This is my preference, as this is the fastest way to unlock the full Hadoop ecosystem for Java 18+

[pan3793]

@stoty

The question whether we want to fully support Java 18+ in 3.4.3.

My current goal is to make Hadoop "client" support Java 18+ in the next 3.4.x. version, fully Java 18+ support should target 3.5.

the large patch for Subject propagation ...

I roughly understand what you are trying to fix, but I haven't seen real failures caused by this, do you have a concrete example that is affected by the lack of change? and is it must be fixed on Hadoop itself, and can not be workarounded by downstream projects?

• Merge just this patch, and release in 3.4.3 - This seems to be enough for at least the Spark tests to pass

I won't block 3.4.2 release if it's already in good shape, I prefer this option.

[stoty]

@stoty

The question whether we want to fully support Java 18+ in 3.4.3.

My current goal is to make Hadoop "client" support Java 18+ in the next 3.4.x. version, fully Java 18+ support should target 3.5.

My problem with that is that I am not at all convinced that this patch alone fixes the client fully.
The only reliable data point we have is that the Spark tests are running with it.

Which parts does Spark even test ? HDFS ? MR/Yarn ? Do Spark tests use secure mode / kerberos ? Do they use doAs/proxyUser ?

the large patch for Subject propagation ...

I roughly understand what you are trying to fix, but I haven't seen real failures caused by this, do you have a concrete example that is affected by the lack of change? and is it must be fixed on Hadoop itself, and can not be workarounded by downstream projects?

Just try and run the tests on JDK24/25 without the Thread fixes (i.e current trunk). Many tests will fail without them.
Unfortunately I don't have notes on the specific tests, and it was a long enough time ago that I can't remember.

• Merge just this patch, and release in 3.4.3 - This seems to be enough for at least the Spark tests to pass

I won't block 3.4.2 release if it's already in good shape, I prefer this option.

I agree with not delaying 3.4.2 further, but have reservations about only adding this patch in 3.4.3, as noted above.
Going for full JDK25 support in 3.4.3 doesn't block 3.4.2 either.

[stoty]

Basically, I am wary of shipping a 3.4.3 that looks like it works with Java 18+ at first glance, but can break down in interesting ways in some use cases.

[pan3793]

Which parts does Spark even test ? HDFS ? MR/Yarn ? Do Spark tests use secure mode / kerberos ? Do they use doAs/proxyUser ?

Spark UT does not cover HDFS/MR tests, it has YARN integration tests by leveraging MiniYARNCluster, the Kerberos test coverage is low, with almost all tests run under SIMPLE mode.

I am wary of shipping a 3.4.3 that looks like it works with Java 18+ at first glance

This is better than nothing, we can continue to fix issues discovered by downstream projects and release new patched versions.

[stoty]

Which parts does Spark even test ? HDFS ? MR/Yarn ? Do Spark tests use secure mode / kerberos ? Do they use doAs/proxyUser ?

Spark UT does not cover HDFS/MR tests, it has YARN integration tests by leveraging MiniYARNCluster, the Kerberos test coverage is low, with almost all tests run under SIMPLE mode.

Thanks

I am wary of shipping a 3.4.3 that looks like it works with Java 18+ at first glance

This is better than nothing, we can continue to fix issues discovered by downstream projects and release new patched versions.

Or we can backport all JDK 18+ patches, and have better confidence that it will work on the first try.

Really there are only two major patches, one is this one, and the other one is the thread patch.
The rest are quite small/simple or only affect tests.

[pan3793]

@stoty WDYT have Hadoop 3.4.3 RC first and hold for a period (at least 3 weeks), to allow downstream projects to integrate and report issues?

[stoty]

I'm OK with that as long as we communicate clearly that the JDK24-25 support is not a properly tested.
Ideally, we'd have the test suite passing with JDK25 before we declare it properly supported.

I understand and respect that you want something for Spark, I just don't want ppl to get the impression that 3.4.3 is properly tested and production ready on JDK18+.

[pan3793]

I'm OK with that as long as we communicate clearly that the JDK24-25 support is not a properly tested.

Absolutely!