Skip to content

Commit 871f1db

Browse files
committed
add spark.yarn.dist.* documentation
1 parent 41bce59 commit 871f1db

File tree

1 file changed

+14
-0
lines changed

1 file changed

+14
-0
lines changed

docs/running-on-yarn.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,20 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
6767
The address of the Spark history server (i.e. host.com:18080). The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI.
6868
</td>
6969
</tr>
70+
<tr>
71+
<td><code>spark.yarn.dist.archives</code></td>
72+
<td>(none)</td>
73+
<td>
74+
Comma separated list of archives to be extracted into the working directory of each executor.
75+
</td>
76+
</tr>
77+
<tr>
78+
<td><code>spark.yarn.dist.files</code></td>
79+
<td>(none)</td>
80+
<td>
81+
Comma-separated list of files to be placed in the working directory of each executor.
82+
</td>
83+
</tr>
7084
</table>
7185

7286
By default, Spark on YARN will use a Spark jar installed locally, but the Spark JAR can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to a JAR on HDFS, `export SPARK_JAR=hdfs:///some/path`.

0 commit comments

Comments
 (0)