[FLINK-38564][connector] FLIP-537: Enumerator Maintains Global Splits Distribution for Balanced assignment. #27149
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
loserwang1024
force-pushed
the
poc-537
branch
from
c75098a to
6cde026
Compare
… Distribution for Balanced assignment.
loserwang1024
force-pushed
the
poc-537
branch
from
6cde026 to
abe1bc7
Compare
|
@flinkbot run azure |
leonardBang
reviewed
Oct 30, 2025
There was a problem hiding this comment.
Thanks @loserwang1024 for the contribution, I left some comments
.../src/main/java/org/apache/flink/api/connector/source/SupportSplitReassignmentOnRecovery.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/flink/api/connector/source/SupportSplitReassignmentOnRecovery.java
Outdated
Show resolved
Hide resolved
flink-core/src/test/java/org/apache/flink/api/connector/source/mocks/MockSplitEnumerator.java
Outdated
Show resolved
Hide resolved
| this.successfulCheckpoints = new ArrayList<>(); | ||
| this.started = false; | ||
| this.closed = false; | ||
| calculateAndPutPendingAssignments(unassignedSplits); |
There was a problem hiding this comment.
reassignSplits(unassignedSplits) ?
There was a problem hiding this comment.
Maybe recalculateAssignments? Don't assign them until the split is added.
flink-core/src/test/java/org/apache/flink/api/connector/source/mocks/MockSplitEnumerator.java
Outdated
Show resolved
Hide resolved
flink-core/src/test/java/org/apache/flink/api/connector/source/mocks/MockSplitEnumerator.java
Outdated
Show resolved
Hide resolved
flink-runtime/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
Outdated
Show resolved
Hide resolved
|
@leonardBang Thanks for your review. Just modify it. |
leonardBang
reviewed
Oct 30, 2025
|
|
||
| /** | ||
| * A decorative interface for {@link Source}. Implementing this interface indicates that the source | ||
| * operator needs to report splits to the enumerator and receive reassignment. |
There was a problem hiding this comment.
needs to report splits to the enumerator on start up and receive reassignment on recovery？
| <rules> | ||
| <requireMavenVersion> | ||
| <version>[3.8.6]</version> | ||
| <version>3.8.6</version> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
As shown in https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=373886480, add a new source contract that enables the enumerator to have a complete global view of splits.
Brief change log
Extending FLIP-27， this FLIP modified ReaderRegistrationEvent Contract:
If a Source implements SupportsSplitReassignmentOnRecovery, the SourceOperator does not assigned splits from state to the reader. Instead, it reports the splits via a ReaderRegistrationEvent sent to the SplitEnumerator.If a Source is not implements SupportsSplitReassignmentOnRecovery, the SourceOperator assigned splits from state to the reader.The splits won't be included in ReaderRegistrationEvent.
After the SourceCoordinator receives the ReaderRegistrationEvent, it calls addReader on the SplitEnumerator. At this point, the enumerator can access the reported splits through the ReaderInfo obtained from SplitEnumeratorContext#registeredReaders.
Verifying this change
Does this pull request potentially affect one of the following parts:
@Public(Evolving): yesDocumentation