Skip to content

Commit f01a967

Browse files
holdenksrowen
authored andcommitted
[SPARK-10332] [CORE] Fix yarn spark executor validation
From Jira: Running spark-submit with yarn with number-executors equal to 0 when not using dynamic allocation should error out. In spark 1.5.0 it continues and ends up hanging. yarn.ClientArguments still has the check so something else must have changed. spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi --num-executors 0 .... spark 1.4.1 errors with: java.lang.IllegalArgumentException: Number of executors was 0, but must be at least 1 (or 0 if dynamic executor allocation is enabled). Author: Holden Karau <[email protected]> Closes #8580 from holdenk/SPARK-10332-spark-submit-to-yarn-executors-0-message. (cherry picked from commit 67580f1) Signed-off-by: Sean Owen <[email protected]>
1 parent 94404ee commit f01a967

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,9 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
9696
}
9797

9898
numExecutors = initialNumExecutors
99+
} else {
100+
val numExecutorsConf = "spark.executor.instances"
101+
numExecutors = sparkConf.getInt(numExecutorsConf, numExecutors)
99102
}
100103
principal = Option(principal)
101104
.orElse(sparkConf.getOption("spark.yarn.principal"))

0 commit comments

Comments
 (0)