HADOOP-19624 Thread leak in ABFS AbfsClientThrottlingAnalyzer #7852
+157
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This method closes the throttling analyzer and releases any associated resources.
|
💔 -1 overall
This message was automatically generated. |
anujmodi2021
requested changes
Aug 26, 2025
There was a problem hiding this comment.
Thanks for the patch and thorough testing of the issue @mattkduran
I have a few suggestions and comments. Please do take a look at them.
Also we need at least one test (unit or integration) to be inlcuded in this patch. Can you plan for one? Idea is to have coverage of code impacted here.
Also, I see a few PR checks failing, If you click on the link of each -1 commented by hadoop-yetus, you should be able to see the issue reported and fix them.
Once all of this is done, we can wait for a few more reviews and get this checked in.
Thanks again for all the efforts.
| @InterfaceAudience.Private | ||
| @InterfaceStability.Unstable | ||
| public interface AbfsThrottlingIntercept { | ||
| public interface AbfsThrottlingIntercept extends Closable { |
There was a problem hiding this comment.
Shouldn't this be extends Closeable?
There was a problem hiding this comment.
Fixed -- thank you for catching the typo!
| } | ||
|
|
||
| /** | ||
| * No-op implementation of close method. |
There was a problem hiding this comment.
Nit: javadoc to include @ throws
|
|
||
| /** | ||
| * Closes the throttling intercept and releases associated resources. | ||
| * This method closes both the read and write throttling analyzers. |
There was a problem hiding this comment.
Nit: Javadoc to include @ throws
| /** | ||
| * Closes the throttling analyzer and releases associated resources. | ||
| * This method cancels the internal timer and cleans up any pending timer tasks. | ||
| * It is safe to call this method multiple times. |
There was a problem hiding this comment.
Fix javadoc here a well.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Spotbug issue might be due to this: https://issues.apache.org/jira/browse/HADOOP-19731 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
The ABFS driver's auto-throttling feature (
fs.azure.enable.autothrottling=true) creates Timer threads in AbfsClientThrottlingAnalyzer that are never properly cleaned up, leading to a memory leak that eventually causes OutOfMemoryError in long-running applications like Hive Metastore.Impact:
Root Cause:
AbfsClientThrottlingAnalyzer creates Timer objects in its constructor but provides no mechanism to cancel them. When AbfsClient instances are closed, the associated timer threads continue running indefinitely.
Solution
Implement proper resource cleanup by making the throttling components implement Closeable and ensuring timers are cancelled when ABFS clients are closed.
Changes Made
https://github.com/mattkduran/ABFSleaktest
https://www.mail-archive.com/[email protected]/msg43483.html
How was this patch tested?
Standalone Validation Tool
This fix was validated using a standalone reproduction and testing tool that directly exercises the ABFS auto-throttling components outside of a full Hadoop deployment.
Repository: ABFSLeakTest
Testing Scope
Test Results
Leak Reproduction Evidence
Test Environment
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?