HADOOP-19664. S3A: Analytics stream to use Java sync client. #7909

ahmarsuhail · 2025-08-27T14:31:13Z

Description of PR

Moves AAL to use the Java sync client as this is giving us the best performance in internal benchmarks.

How was this patch tested?

Testing in progress.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

ahmarsuhail · 2025-08-27T14:32:04Z

@steveloughran @mukund-thakur @shameersss1

small PR to move to AAL to use the Java sync client

hadoop-yetus · 2025-08-27T16:42:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	9m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 54s		trunk passed
+1 💚	compile	8m 28s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	7m 31s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	2m 1s		trunk passed
+1 💚	mvnsite	1m 1s		trunk passed
+1 💚	javadoc	1m 0s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 58s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 30s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	21m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	8m 9s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	8m 9s		the patch passed
+1 💚	compile	8m 25s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	8m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 25s	/results-checkstyle-root.txt	root: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 56s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 18s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	24m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 25s		hadoop-project in the patch passed.
+1 💚	unit	2m 44s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		130m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/1/artifact/out/Dockerfile
GITHUB PR	#7909
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 9825a7b540bf 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `77adf89`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/1/testReport/
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur · 2025-09-02T16:45:13Z

checkstyle failure.
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/streams/TestStreamFactories.java:26:import software.amazon.awssdk.services.s3.S3AsyncClient;:8: Unused import - software.amazon.awssdk.services.s3.S3AsyncClient. [UnusedImports]

mukund-thakur

+1
pending test run results.

steveloughran

+1 pending results of store test run. I know things play up right now but it's good to see from your runs too

steveloughran · 2025-09-15T11:44:48Z

(oh, and include S3A: in the commit message. THX)

ahmarsuhail · 2025-09-16T10:23:43Z

Tested in eu-west-1, with nohup mvn -Dparallel-tests -DtestsThreadCount=16 clean verify

Test results are all good, except the known ITestS3AContractOpen>AbstractContractOpenTest.testInputStreamReadNegativePosition failure. Will merge.

I don't see any failures in ITestS3AContractAnalyticsStreamVectoredRead that Steve saw in the SDK upgrade PR: #7882, will take a look at that separately.

hadoop-yetus · 2025-09-16T12:00:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	9m 28s		Maven dependency ordering for branch
-1 ❌	mvninstall	26m 11s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	9m 1s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	8m 18s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	2m 24s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	0m 51s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 24s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	23m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 27s		the patch passed
+1 💚	compile	9m 8s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	9m 8s		the patch passed
+1 💚	compile	8m 15s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	8m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 16s		the patch passed
+1 💚	mvnsite	0m 51s		the patch passed
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 48s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 21s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	23m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 18s		hadoop-project in the patch passed.
+1 💚	unit	2m 19s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		136m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/2/artifact/out/Dockerfile
GITHUB PR	#7909
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux f17818206b60 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `112594c`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/2/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-09-16T12:05:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	9m 2s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 25s		trunk passed
+1 💚	compile	9m 31s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	8m 14s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	2m 9s		trunk passed
+1 💚	mvnsite	0m 53s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 44s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 24s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	23m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	9m 5s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	9m 5s		the patch passed
+1 💚	compile	8m 2s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	8m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 8s		the patch passed
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 47s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 45s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 20s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	22m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 26s		hadoop-project in the patch passed.
+1 💚	unit	2m 42s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		135m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/3/artifact/out/Dockerfile
GITHUB PR	#7909
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 304e885133c5 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `65d16df`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/3/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7909/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2025-09-17T18:19:35Z

@ahmarsuhail can you do a followup to add the library to LICENSE-binary? thanks

github-actions bot added build trunk TOOLS AWS labels Aug 27, 2025

rebecanonato89 approved these changes Aug 27, 2025

View reviewed changes

mukund-thakur approved these changes Sep 2, 2025

View reviewed changes

steveloughran changed the title ~~HADOOP-19664. AAL to use Java sync client.~~ HADOOP-19664. S3A. Analytics stream to use Java sync client. Sep 15, 2025

steveloughran changed the title ~~HADOOP-19664. S3A. Analytics stream to use Java sync client.~~ HADOOP-19664. S3A: Analytics stream to use Java sync client. Sep 15, 2025

steveloughran approved these changes Sep 15, 2025

View reviewed changes

ahmarsuhail added 2 commits September 16, 2025 10:48

AAL to use Java sync client

835fbf0

checkstyle failure

65d16df

ahmarsuhail force-pushed the aal-sync-client branch from 112594c to 65d16df Compare September 16, 2025 09:49

ahmarsuhail merged commit 1b152a4 into apache:trunk Sep 16, 2025
1 of 2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19664. S3A: Analytics stream to use Java sync client. #7909

HADOOP-19664. S3A: Analytics stream to use Java sync client. #7909

Uh oh!

ahmarsuhail commented Aug 27, 2025 •

edited

Loading

Uh oh!

ahmarsuhail commented Aug 27, 2025

Uh oh!

hadoop-yetus commented Aug 27, 2025

Uh oh!

mukund-thakur commented Sep 2, 2025

Uh oh!

mukund-thakur left a comment

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran commented Sep 15, 2025

Uh oh!

ahmarsuhail commented Sep 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

hadoop-yetus commented Sep 16, 2025

Uh oh!

hadoop-yetus commented Sep 16, 2025

Uh oh!

steveloughran commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HADOOP-19664. S3A: Analytics stream to use Java sync client. #7909

HADOOP-19664. S3A: Analytics stream to use Java sync client. #7909

Uh oh!

Conversation

ahmarsuhail commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

ahmarsuhail commented Aug 27, 2025

Uh oh!

hadoop-yetus commented Aug 27, 2025

Uh oh!

mukund-thakur commented Sep 2, 2025

Uh oh!

mukund-thakur left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Sep 15, 2025

Uh oh!

ahmarsuhail commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Sep 16, 2025

Uh oh!

hadoop-yetus commented Sep 16, 2025

Uh oh!

steveloughran commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ahmarsuhail commented Aug 27, 2025 •

edited

Loading

ahmarsuhail commented Sep 16, 2025 •

edited

Loading