Skip to content

Commit 3a0ec77

Browse files
committed
Update
1 parent 052e288 commit 3a0ec77

File tree

2 files changed

+3
-3
lines changed

2 files changed

+3
-3
lines changed

docs/submitting-applications.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ files to be distributed with your application. If you depend on multiple Python
2323
packaging them into a `.zip` or `.egg`.
2424

2525
As Python can not read files from assembly jar which packaged by JDK1.7+, so packaging pyspark into a
26-
`.zip`(the name contains "spark-pyspark") and use `--py-files` argument of `spark-submit` to distribute it.
26+
`.zip`(the name contains "pyspark") and use `--py-files` argument of `spark-submit` to distribute it.
2727

2828
# Launching Applications with spark-submit
2929

yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,8 +78,8 @@ class ExecutorRunnable(
7878
// From SPARK-1920 and SPARK-1520 we know PySpark on Yarn can not work when the assembly jar are
7979
// package by JDK 1.7+, so we ship PySpark archives to executors by Yarn with --py-files, and
8080
// add this path to PYTHONPATH.
81-
for ((k, v) <- localResources if k.contains("spark-pyspark")) {
82-
env("PYSPARK_ARCHIVES_PATH") = k
81+
for ((resPath, res) <- localResources if resPath.contains("pyspark")) {
82+
env("PYSPARK_ARCHIVES_PATH") = resPath
8383
}
8484
ctx.setEnvironment(env)
8585

0 commit comments

Comments
 (0)