File tree Expand file tree Collapse file tree 1 file changed +3
-0
lines changed
core/src/main/scala/org/apache/spark/scheduler Expand file tree Collapse file tree 1 file changed +3
-0
lines changed Original file line number Diff line number Diff line change @@ -1190,6 +1190,9 @@ class DAGScheduler(
11901190 // completed successfully from the perspective of the TaskSetManager, mark it as
11911191 // no longer pending (the TaskSetManager may consider the task complete even
11921192 // when the output needs to be ignored because the task's epoch is too small below).
1193+ // This can result in inconsistency between pending partitions and output locations
1194+ // of stage. When pending partitions is empty, the scheduler will check output
1195+ // locations, if there are missing, the stage will be resubmitted.
11931196 shuffleStage.pendingPartitions -= task.partitionId
11941197 }
11951198 if (failedEpoch.contains(execId) && smt.epoch <= failedEpoch(execId)) {
You can’t perform that action at this time.
0 commit comments