[SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo #38030
Conversation
bozhang2820
changed the title
|
How/where are we using the message ? |
|
Ok, so this is mainly to propagate in +CC @dongjoon-hyun |
|
Thank you, @mridulm . |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Please revert the change on MapStatus.
core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala
Outdated
Show resolved
Hide resolved
core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionIntegrationSuite.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
Outdated
Show resolved
Hide resolved
|
Could you address the review comments, @bozhang2820 ? |
Sorry for the late response. I was on vacation last week. Addressed the review comments. |
|
Build passed with a retry: https://github.com/bozhang2820/spark/actions/runs/3209525689/jobs/5251496034 |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for update, @bozhang2820 . It looks good to me.
What do you think about the Today's patch, @mridulm , @Ngone51 , @xuanyuanking?
|
Thanks, merged to master! |
|
Thank you all! :) |
5 participants
What changes were proposed in this pull request?
This change populates
ExecutorDecommissionwith messages inExecutorDecommissionInfo.Why are the changes needed?
Currently the message in
ExecutorDecommissionis a fixed value ("Executor decommission."), so it is the same for all cases, e.g. spot instance interruptions and auto-scaling down. With this change we can better differentiate those cases.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added a unit test.