This repository was archived by the owner on Oct 23, 2024. It is now read-only.
[DCOS-39150][SPARK] Support unique Executor IDs in cluster managers #82
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Oct 18, 2020
Author
|
This PR is just for reference, so final and effective change has been moved to #69. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference PR#36
What changes were proposed in this pull request?
Using incremental integers as Executor IDs leads to a situation when Spark Executors launched by different Drivers have same IDs. When running on DCOS that means that Mesos Task IDs for Spark Executors can have duplicates too. This change introduces unique identifiers for both
CoarseGrainedSchedulerBackendandYarnAllocatorwhich are prepended to numeric ID thus allowing to distinguish Executors belonging to different drivers.This PR reverts commit ebe3c7f: "initialize executorIdCounter after ApplicationMaster killed for max n…)" realted to Jira Issue SPARK-12864 because it addressed the similar but YARN-case specific issue.
Combine changes in PR#52: [DCOS-51158] Improved Task ID assignment for Executor tasks
How was this patch tested?
The change is verified by a unit test and external integration test.