[SPARK-1177] Allow SPARK_JAR to be set programmatically in system properties

[dbtsai]

No description provided.

[AmplabJenkins]

Build triggered.

[vanzin]

#560 has what I believe is a better way of handling this.

[AmplabJenkins]

Build triggered.

[dbtsai]

@chesterxgchen

#560 Agree, it's a more throughout way to handle this issue. In the code you have, it seems that the spark jar setting is moved to conf: SparkConf in favor of the CONF_SPARK_JAR. But it will make users difficult to set it up since the Client.scala also has to be changed. Simple question, with your change, how users can submit job with their own spark jar by passing the CONF_SPARK_JAR correctly?

def sparkJar(conf: SparkConf) = {
   if (conf.contains(CONF_SPARK_JAR)) {
     conf.get(CONF_SPARK_JAR)
   } else if (System.getenv(ENV_SPARK_JAR) != null) {
     logWarning(
      s"$ENV_SPARK_JAR detected in the system environment. This variable has been deprecated " 
       s"in favor of the $CONF_SPARK_JAR configuration variable.")
     System.getenv(ENV_SPARK_JAR)
   } else {
     SparkContext.jarOfClass(this.getClass).head
   }
 }

[vanzin]

There's no need to change Client.scala with my change; all you need to do is set "spark.yarn.jar" somewhere - JVM system property, spark-defaults.conf, of in the app's code (modifying the SparkConf instance), and it will be picked up by the Yarn code.

[AmplabJenkins]

Build started.

[dbtsai]

The app's code will only run in the application master in yarn-cluster mode, how can yarn client know which jar will be submitted to distributed cache if we set it in the app's spark conf?

[vanzin]

Ok, in cluster mode you can't use SparkConf.set(), but the other two options work fine. You can't do System.setProperty() in cluster mode to achieve that either, so even with your patch, you'd have to use -DSPARK_JAR=foo in the command line for it to work in yarn-cluster mode.

[AmplabJenkins]

Build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15489/

[dbtsai]

We lunched Spark job inside our tomcat, and we directly use Client.scala API. With my patch, I can setup the spark jar using System.setProperty() before

  val sparkConf = new SparkConf
  val args = getArgsFromConf(conf)
  new Client(new ClientArguments(args, sparkConf), hadoopConfig, sparkConf).run

Do you mean that with your work, I can setup the jar location in the sparkConf which will be passed into the new Client?

Can we have the following in sparkJar method

def sparkJar(conf: SparkConf) = {
   if (conf.contains(CONF_SPARK_JAR)) {
     conf.get(CONF_SPARK_JAR)
   } else if (System.getProperty(ENV_SPARK_JAR) != null) {
     logWarning(
      s"$ENV_SPARK_JAR detected in the system property. This variable has been deprecated " 
       s"in favor of the $CONF_SPARK_JAR configuration variable.")
     System.getProperty(ENV_SPARK_JAR)
   } else if (System.getenv(ENV_SPARK_JAR) != null) {
     logWarning(
      s"$ENV_SPARK_JAR detected in the system environment. This variable has been deprecated " 
       s"in favor of the $CONF_SPARK_JAR configuration variable.")
     System.getenv(ENV_SPARK_JAR)
   } else {
     SparkContext.jarOfClass(this.getClass).head
   }
 }

[vanzin]

I mean you can set system properties the same way. SparkConf initializes its configuration from system properties, so my patch covers not only your case, but also others (like using a spark-defaults.conf file for spark-submit users).

[dbtsai]

Got you. Looking forward to having your patch merged. Thanks.

Sent from my Google Nexus 5
On Jun 6, 2014 9:35 AM, "Marcelo Vanzin" [email protected] wrote:

I mean you can set system properties the same way. SparkConf initializes
its configuration from system properties, so my patch covers not only your
case, but also others (like using a spark-defaults.conf file for
spark-submit users).

—
Reply to this email directly or view it on GitHub
#987 (comment).

[dbtsai]

#560 is merged. Close this PR.