[spark-11026][yarn]spark.yarn.user.classpath.first does work for 'spark-submit --jars hdfs://user/foo.jar'

[lianhuiwang]

when spark.yarn.user.classpath.first=true and using 'spark-submit --jars hdfs://user/foo.jar', it can not put foo.jar to system classpath. so we need to put yarn's linkNames of jars to the system classpath. @vanzin @tgravescs

[SparkQA]

Test build #43466 has finished for PR 9045 at commit 3023e1e.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

classpath is already printed if you turn debug on.

[tgravescs]

I don't quite follow the scenario can you please describe in more detail?

[SparkQA]

Test build #43472 has finished for PR 9045 at commit 4c5290a.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Not sure I understand either. What is this addJars you talk about? Also, the code doesn't even build.

[lianhuiwang]

@vanzin @tgravescs when spark-submit --master yarn-cluster --jars /users/xxx.jar and spark.yarn.user.classpath.first=true, the jars of the command are not in system's classpath.
so when schemes of jars are not LOCAL_SCHEME and fileName is null, it needs to add yarn's linkNames of jars to classpath.

[SparkQA]

Test build #43530 has finished for PR 9045 at commit 6f395d6.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

sorry your description isn't clarifying it for me. Could you perhaps give an example? what do you mean by linkNames?

If I use --jars /users/foo.jar and spark.yarn.user.classpath.first=true. foo.jar gets put into the system classpath (CLASSPATH=.....:foo.jar:...:spark.jar

[lianhuiwang]

@tgravescs if use --jars hdfs://users/foo.jar, now i can not put foo.jar into the system classpath without this PR.

[tgravescs]

ah I understand now, thanks. I'll take a look

[vanzin]

@lianhuiwang can you update the description to match your explanation? Mentioning "addJars" is confusing because it's not the source of the problem at all - rather, spark-submit's "--jars" option is.

[vanzin]

retest this please

[SparkQA]

Test build #43578 has finished for PR 9045 at commit 6f395d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

LGTM pending update of the description.

[tgravescs]

minor nit, add space before {

[tgravescs]

LGTM too, thanks for catching this, I should have tested these when I fixed the other part.

[lianhuiwang]

@tgravescs @vanzin i have updated it. thanks.

[SparkQA]

Test build #43642 has finished for PR 9045 at commit d4049a7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

+1 thanks!