[SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work #969

witgo · 2014-06-04T17:06:40Z

No description provided.

AmplabJenkins · 2014-06-04T17:07:59Z

Merged build triggered.

AmplabJenkins · 2014-06-04T17:08:08Z

Merged build started.

tgravescs · 2014-06-04T17:41:55Z

Please provide more description or problem and how to reproduce and open a jira.

AmplabJenkins · 2014-06-04T17:48:55Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-04T17:48:55Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15449/

vanzin · 2014-06-04T23:21:56Z

yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

I don't think env variables and conf entries should be handled here like this.

YarnClientSchedulerBackend already deals with the env variable and command line option for client mode. It seems that SparkSubmit might be missing code to handle the env variable for cluster mode, though. Probably better to fix it there, and leave this code to deal only with the command line args (which are already correctly parsed).

@vanzin This is a yarn cluster problems. Don't use YarnClientSchedulerBackend.

Did you see the other PR I mentioned (see #969 (comment))?

There's a bug in the Yarn code where configuration is not propagated correctly in cluster mode, which I fixed there. So you don't need this here because with my fix, these options should be correctly propagated without any extra code.

I see, Weekend, I will test your branch, Please merge the master changes.

witgo · 2014-06-05T14:22:03Z

@tgravescs @vanzin
Spark configuration
conf/spark-defaults.conf =>

spark.yarn.dist.archives     /toona/conf
spark.executor.extraClassPath ./conf
spark.driver.extraClassPath  ./conf

HDFS directory
hadoop dfs -cat /toona/conf/toona.conf =>

 redis.num=4

The following command execution fails

YARN_CONF_DIR=/etc/hadoop/conf ./bin/spark-submit  --num-executors 2 --driver-memory 2g --executor-memory 2g --master yarn-cluster --class toona.DeployTest toona-assembly.jar

The following is the test code

package toona
import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory

object DeployTest {
  def main(args: Array[String]) {
    val conf = ConfigFactory.load("toona.conf")
    val redisNum = conf.getInt("redis.num") // Here will throw an `ConfigException` exception
    assert(redisNum == 4)

  }
}

AmplabJenkins · 2014-06-09T02:57:49Z

Merged build triggered.

AmplabJenkins · 2014-06-09T02:57:55Z

Merged build started.

AmplabJenkins · 2014-06-09T03:38:33Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-09T03:38:33Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15550/

AmplabJenkins · 2014-06-11T14:02:07Z

Merged build triggered.

AmplabJenkins · 2014-06-11T14:02:17Z

Merged build started.

AmplabJenkins · 2014-06-11T14:44:17Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-11T14:44:18Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15680/

vanzin · 2014-06-11T21:56:22Z

@witgo You didn't address my comments. You're duplicating logic that's already mostly there in SparkSubmit, just needs to be fixed a little for yarn cluster mode. See my previous comment.

As for your test, see: #560, more specifically: vanzin@08268f0

tgravescs · 2014-06-12T13:34:32Z

@witgo the [WIP] tag tells me you are still working on this, looking for early comments on the approach, and its not really fully done and ready for review, is that the case? If not can you please remove that tag.

witgo · 2014-06-12T13:48:31Z

@tgravescs
Still have some questions not discussed clearly.
@vanzin doesn't like to do so.

AmplabJenkins · 2014-06-13T07:27:05Z

Merged build triggered.

AmplabJenkins · 2014-06-13T07:27:12Z

Merged build started.

AmplabJenkins · 2014-06-13T08:29:23Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-13T08:29:23Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15756/

AmplabJenkins · 2014-06-17T02:50:48Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15836/

AmplabJenkins · 2014-06-17T02:50:48Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15835/

…ientBase

AmplabJenkins · 2014-06-17T09:09:47Z

Merged build triggered.

AmplabJenkins · 2014-06-17T09:09:55Z

Merged build started.

AmplabJenkins · 2014-06-17T10:20:50Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-17T10:20:50Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15848/

tgravescs · 2014-06-17T15:29:41Z

yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

this logic is already handled by spark-submit, no reason to duplicate

tgravescs · 2014-06-17T15:36:35Z

Sorry if I wan't clear. I didn't mean to update ClientArguments with that logic, I just meant that is the logic we want overall. We don't want to duplicate code if possible.

Overall I think if the --files/--archives are specified to ClientArguments we don't touch them. If they aren't specified we look at the config variables and resolveURI them. Then we need to modify the YarnClientSchedulerBackend to properly handle when spark.yarn.dist.archives/spark.yarn.dist.files configs are set (but not set by spark-submit script) (it should default to file://, not hdfs://).

AmplabJenkins · 2014-06-17T15:39:49Z

Merged build triggered.

AmplabJenkins · 2014-06-17T15:39:57Z

Merged build started.

AmplabJenkins · 2014-06-17T15:40:00Z

Merged build finished.

AmplabJenkins · 2014-06-17T15:40:01Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15849/

tgravescs · 2014-06-17T15:41:06Z

note I think it woudl be easiest just to have YarnClientSchedulerBackend ignore the configs spark.yarn.dist.archives/spark.yarn.dist.files and only do the backwards compatibility for the ENV variables. Then your changes to clientArguments would handle calling resolveURI on them.

witgo · 2014-06-17T15:52:04Z

spark-defaults.conf	command	path
spark.yarn.dist.archives /other/path	`./bin/spark-submit --archives /some/path --master yarn-cluster`	file://some/path
spark.yarn.dist.archives /other/path	`./bin/spark-submit --archives /some/path --master yarn-client`	file://some/path
spark.yarn.dist.archives /other/path	`./bin/spark-submit --master yarn-cluster`	file://other/path
spark.yarn.dist.archives /other/path	`./bin/spark-submit --master yarn-client`	file://other/path
spark.yarn.dist.archives hdfs://other/path	`./bin/spark-submit --master yarn-client`	hdfs://other/path
spark.yarn.dist.archives /other/path	`SPARK_YARN_DIST_ARCHIVES=/some/path ./bin/spark-submit --master yarn-client`	hdfs://some/path
spark.yarn.dist.archives /other/path	`SPARK_YARN_DIST_ARCHIVES=/some/path ./bin/spark-submit --master yarn-cluster`	hdfs://some/path

witgo · 2014-06-17T15:59:03Z

@tgravescs
spark.yarn.dist.* related is concentrated in ClientArguments is a good solution

tgravescs · 2014-06-19T17:10:52Z

Looks good.

Author: witgo <[email protected]> Closes apache#969 from witgo/yarn_ClientBase and squashes the following commits: 8117765 [witgo] review commit 3bdbc52 [witgo] Merge branch 'master' of https://github.com/apache/spark into yarn_ClientBase 5261b6c [witgo] fix sys.props.get("SPARK_YARN_DIST_FILES") e3c1107 [witgo] update docs b6a9aa1 [witgo] merge master c8b4554 [witgo] review commit 2f48789 [witgo] Merge branch 'master' of https://github.com/apache/spark into yarn_ClientBase 8d7b82f [witgo] Merge branch 'master' of https://github.com/apache/spark into yarn_ClientBase 1048549 [witgo] remove Utils.resolveURIs 871f1db [witgo] add spark.yarn.dist.* documentation 41bce59 [witgo] review commit 35d6fa0 [witgo] move to ClientArguments 55d72fc [witgo] Merge branch 'master' of https://github.com/apache/spark into yarn_ClientBase 9cdff16 [witgo] review commit 8bc2f4b [witgo] review commit 20e667c [witgo] Merge branch 'master' into yarn_ClientBase 0961151 [witgo] merge master ce609fc [witgo] Merge branch 'master' into yarn_ClientBase 8362489 [witgo] yarn.ClientBase spark.yarn.dist.* do not work

yarn.ClientBase spark.yarn.dist.* do not work

8362489

vanzin reviewed Jun 4, 2014
View reviewed changes

witgo changed the title ~~[WIP] In yarn.ClientBase spark.yarn.dist.* do not work~~ [WIP] [SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work Jun 6, 2014

Merge branch 'master' into yarn_ClientBase

ce609fc

merge master

0961151

witgo changed the title ~~[WIP] [SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work~~ [SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work Jun 12, 2014

witgo added 2 commits June 13, 2014 14:50

Merge branch 'master' into yarn_ClientBase

20e667c

review commit

8bc2f4b

review commit

9cdff16

witgo added 2 commits June 17, 2014 17:06

fix sys.props.get("SPARK_YARN_DIST_FILES")

5261b6c

Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…

3bdbc52

…ientBase

tgravescs reviewed Jun 17, 2014
View reviewed changes

review commit

8117765

asfgit closed this in bce0897 Jun 19, 2014

witgo deleted the yarn_ClientBase branch June 20, 2014 07:16

andrewor14 mentioned this pull request Sep 18, 2014

[SPARK-3319] [SPARK-3338] Resolve Spark submit config paths #2232

Closed

[SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work #969

[SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work #969

Uh oh!

Conversation

witgo commented Jun 4, 2014

Uh oh!

AmplabJenkins commented Jun 4, 2014

Uh oh!

AmplabJenkins commented Jun 4, 2014

Uh oh!

tgravescs commented Jun 4, 2014

Uh oh!

AmplabJenkins commented Jun 4, 2014

Uh oh!

AmplabJenkins commented Jun 4, 2014

Uh oh!

vanzin Jun 4, 2014

Choose a reason for hiding this comment

Uh oh!

witgo Jun 12, 2014

Choose a reason for hiding this comment

Uh oh!

vanzin Jun 12, 2014

Choose a reason for hiding this comment

Uh oh!

witgo Jun 13, 2014

Choose a reason for hiding this comment

Uh oh!

witgo commented Jun 5, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

vanzin commented Jun 11, 2014

Uh oh!

tgravescs commented Jun 12, 2014

Uh oh!

witgo commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 13, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

tgravescs Jun 17, 2014

Choose a reason for hiding this comment

Uh oh!

tgravescs commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014

Uh oh!

AmplabJenkins commented Jun 17, 2014