SPARK-4338. [YARN] Ditch yarn-alpha. #3215

Yes, if it's targeted for 1.3, it would be nice to get this in not long after 1.2.
I think there's more that simplifies, like some reflection-related code to bridge the gap between the two versions. That's worth a review.

sryza · 2014-11-12T18:08:25Z

I have a whole set of simplifying changes that I want to go in (e.g. YARN-1714), but thought it would probably be good to break things up a bit for easier review.

vanzin · 2014-11-12T18:48:05Z

+1; cleanups can come later.

sryza · 2014-11-12T21:03:02Z

@tgravescs

SparkQA · 2014-11-12T21:05:11Z

Test build #23279 has started for PR 3215 at commit 8427a9b.

This patch merges cleanly.

tgravescs · 2014-11-12T21:10:31Z

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

is \n still needed?

Oops sorry missed that one

tgravescs · 2014-11-12T21:12:37Z

If we don't pull this in right away (wait til closer to 1.2 release) is it going to delay your other prs? It would be nice to hold off a bit to pull this in if not.

I took a quick look at the pr, we need to update docs/building-spark.md. I still need to download this and try it out.

andrewor14 · 2014-11-12T21:43:21Z

+1. I imagine this opens up a world of opportunities to cleanup, which can come later in separate PRs. In particular we relied on a lot of abstractions that were necessary to avoid duplicate code, but these are no longer needed and it would simplify the code significantly if we get rid of them.

SparkQA · 2014-11-12T22:28:08Z

Test build #23279 has finished for PR 3215 at commit 8427a9b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-12T22:28:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23279/
Test PASSed.

SparkQA · 2014-11-13T00:15:00Z

Test build #23290 has started for PR 3215 at commit 66e731f.

This patch does not merge cleanly.

sryza · 2014-11-13T00:38:35Z

@tgravescs it would delay other PRs, but not a huge deal if you think it's too soon.

SparkQA · 2014-11-13T00:40:41Z

Test build #23293 has started for PR 3215 at commit ec3d8da.

This patch merges cleanly.

SparkQA · 2014-11-13T01:43:44Z

Test build #23290 has finished for PR 3215 at commit 66e731f.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-13T01:43:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23290/
Test PASSed.

SparkQA · 2014-11-13T02:04:45Z

Test build #23293 has finished for PR 3215 at commit ec3d8da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-13T02:04:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23293/
Test PASSed.

tgravescs · 2014-11-14T15:15:32Z

docs/building-spark.md

I know you didn't change this here (so we can file separate jira for it if you want) but perhaps we should add example and some text here about hadoop 2.5 and >. Since there is no hadoop profile 2.5 and the user should use the 2.4 profile.

tgravescs · 2014-11-14T15:17:01Z

was the yarn/stable directory removed with this? I couldn't tell from applying the patch and not sure how directory removals show up here.

SparkQA · 2014-11-17T07:39:58Z

Test build #23469 has started for PR 3215 at commit 8fbf896.

This patch merges cleanly.

sryza · 2014-11-17T17:18:30Z

Successfully ran spark-shell in yarn-client mode and an app in yarn-cluster mode.

ksakellis · 2014-11-22T01:03:17Z

LGTM - can't wait for this to merge in.

SparkQA · 2014-11-24T07:12:44Z

Test build #23774 has started for PR 3215 at commit 7175ae8.

This patch merges cleanly.

SparkQA · 2014-11-24T08:39:48Z

Test build #23774 has finished for PR 3215 at commit 7175ae8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-24T08:39:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23774/
Test PASSed.

SparkQA · 2014-12-03T02:25:09Z

Test build #24069 has started for PR 3215 at commit 3128915.

This patch merges cleanly.

SparkQA · 2014-12-03T03:46:14Z

Test build #24069 has finished for PR 3215 at commit 3128915.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-03T03:46:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24069/
Test PASSed.

sryza · 2014-12-05T09:42:40Z

@tgravescs @andrewor14 do you feel comfortable merging this now that 1.2 is almost out the door?

SparkQA · 2014-12-05T09:50:10Z

Test build #24180 has started for PR 3215 at commit 1c5ac08.

This patch merges cleanly.

AmplabJenkins · 2014-12-05T09:57:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24179/
Test FAILed.

SparkQA · 2014-12-05T11:16:36Z

Test build #24180 has finished for PR 3215 at commit 1c5ac08.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-05T11:16:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24180/
Test PASSed.

tgravescs · 2014-12-08T15:53:01Z

seems like we are pretty close on the rc. I'm good with merging this. @andrewor14 any objections at this point?

andrewor14 · 2014-12-09T18:56:22Z

We actually don't need to wait on the RC to merge this since this is only going into master. I'll take a quick look and will likely merge it after that since this is pretty straightforward.

andrewor14 · 2014-12-09T19:01:06Z

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

do you need to do .stripMargins here?

andrewor14 · 2014-12-09T19:02:05Z

Hey this LGTM. I'm just gonna merge this. My comments are minor enough that we can slip them in later.

sryza · 2014-12-09T19:48:50Z

Woot! I'll make that fix as part of SPARK-4447.

sryza force-pushed the sandy-spark-4338 branch from 52a4a75 to b0bbb13 Compare November 12, 2014 01:00

sryza force-pushed the sandy-spark-4338 branch from f8c90f3 to 8427a9b Compare November 12, 2014 21:01

tgravescs reviewed Nov 12, 2014
View reviewed changes

sryza force-pushed the sandy-spark-4338 branch from 66e731f to ec3d8da Compare November 13, 2014 00:33

tgravescs reviewed Nov 14, 2014
View reviewed changes

sryza force-pushed the sandy-spark-4338 branch from ec3d8da to 8fbf896 Compare November 17, 2014 07:36

sryza force-pushed the sandy-spark-4338 branch from 8fbf896 to 7175ae8 Compare November 24, 2014 07:07

sryza force-pushed the sandy-spark-4338 branch from 7175ae8 to 3128915 Compare December 3, 2014 02:20

sryza force-pushed the sandy-spark-4338 branch 2 times, most recently from 72b63a9 to 3d244f2 Compare December 5, 2014 09:41

sryza added 2 commits December 5, 2014 01:41

SPARK-4338. Ditch yarn-alpha.

9c1421c

Update building Spark docs and remove unnecessary newline

1c5ac08

sryza force-pushed the sandy-spark-4338 branch from 3d244f2 to 1c5ac08 Compare December 5, 2014 09:42

sryza changed the title ~~SPARK-4338. Ditch yarn-alpha.~~ SPARK-4338. [YARN] Ditch yarn-alpha. Dec 5, 2014

andrewor14 reviewed Dec 9, 2014
View reviewed changes

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

Copy link

Contributor

andrewor14 Dec 9, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need to do .stripMargins here?

asfgit closed this in 912563a Dec 9, 2014

SPARK-4338. [YARN] Ditch yarn-alpha. #3215

SPARK-4338. [YARN] Ditch yarn-alpha. #3215

Uh oh!

Conversation

sryza commented Nov 12, 2014

Uh oh!

SparkQA commented Nov 12, 2014

Uh oh!

SparkQA commented Nov 12, 2014

Uh oh!

AmplabJenkins commented Nov 12, 2014

Uh oh!

SparkQA commented Nov 12, 2014

Uh oh!

SparkQA commented Nov 12, 2014

Uh oh!

AmplabJenkins commented Nov 12, 2014

Uh oh!

srowen commented Nov 12, 2014

Uh oh!

sryza commented Nov 12, 2014

Uh oh!

vanzin commented Nov 12, 2014

Uh oh!

sryza commented Nov 12, 2014

Uh oh!

SparkQA commented Nov 12, 2014

Uh oh!

tgravescs Nov 12, 2014

Choose a reason for hiding this comment

Uh oh!

sryza Nov 12, 2014

Choose a reason for hiding this comment

Uh oh!

tgravescs commented Nov 12, 2014

Uh oh!

andrewor14 commented Nov 12, 2014

Uh oh!

SparkQA commented Nov 12, 2014

Uh oh!

AmplabJenkins commented Nov 12, 2014

Uh oh!

SparkQA commented Nov 13, 2014

Uh oh!

sryza commented Nov 13, 2014

Uh oh!

SparkQA commented Nov 13, 2014

Uh oh!

SparkQA commented Nov 13, 2014

Uh oh!

AmplabJenkins commented Nov 13, 2014

Uh oh!

SparkQA commented Nov 13, 2014

Uh oh!

AmplabJenkins commented Nov 13, 2014

Uh oh!

tgravescs Nov 14, 2014

Choose a reason for hiding this comment

Uh oh!

tgravescs commented Nov 14, 2014

Uh oh!

SparkQA commented Nov 17, 2014

Uh oh!

sryza commented Nov 17, 2014

Uh oh!

ksakellis commented Nov 22, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

SparkQA commented Nov 24, 2014

Uh oh!

AmplabJenkins commented Nov 24, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

SparkQA commented Dec 3, 2014

Uh oh!

AmplabJenkins commented Dec 3, 2014

Uh oh!

sryza commented Dec 5, 2014