Skip to content

Commit e34cd85

Browse files
author
jinxing
committed
Refine comment.
1 parent ab8d13e commit e34cd85

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1190,6 +1190,9 @@ class DAGScheduler(
11901190
// completed successfully from the perspective of the TaskSetManager, mark it as
11911191
// no longer pending (the TaskSetManager may consider the task complete even
11921192
// when the output needs to be ignored because the task's epoch is too small below).
1193+
// This can result in inconsistency between pending partitions and output locations
1194+
// of stage. When pending partitions is empty, the scheduler will check output
1195+
// locations, if there are missing, the stage will be resubmitted.
11931196
shuffleStage.pendingPartitions -= task.partitionId
11941197
}
11951198
if (failedEpoch.contains(execId) && smt.epoch <= failedEpoch(execId)) {

0 commit comments

Comments
 (0)