Skip to content

[SPARK-18662] Move resource managers to separate directory #16092

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-18662] Move resource managers to separate directory #16092

wants to merge 2 commits into from

Conversation

foxish
Copy link
Contributor

@foxish foxish commented Dec 1, 2016

What changes were proposed in this pull request?

Ref: #16061 (comment)

How was this patch tested?

  • Manual tests

/cc @rxin

@foxish
Copy link
Contributor Author

foxish commented Dec 1, 2016

/cc @tnachen

@JoshRosen
Copy link
Contributor

There's a few other places that will need updating, including the source_file_regexes in the test runner configuration (modules.py):

spark/dev/sparktestsupport/modules.py

Line 472 in 0a81121

"yarn/",
and

spark/dev/sparktestsupport/modules.py

Line 488 in 0a81121

source_file_regexes=["mesos/"],

@JoshRosen
Copy link
Contributor

Jenkins, this is ok to test

@SparkQA
Copy link

SparkQA commented Dec 1, 2016

Test build #69448 has finished for PR 16092 at commit c6c7dc2.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Dec 1, 2016

Hm, is this really necessary? I know it's just a move but for 2-3 modules, does it matter enough?

@tnachen
Copy link
Contributor

tnachen commented Dec 1, 2016

Changed suggested by @rxin

@ash211
Copy link
Contributor

ash211 commented Dec 1, 2016

With a k8s backend on the way I do think it adds a nice organization for these 3 clearly grouped modules

@foxish
Copy link
Contributor Author

foxish commented Dec 1, 2016

@JoshRosen Thanks for reviewing.
Updated. PTAL.

@SparkQA
Copy link

SparkQA commented Dec 1, 2016

Test build #69499 has finished for PR 16092 at commit 6763a80.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@foxish
Copy link
Contributor Author

foxish commented Dec 1, 2016

The test failure appears to be an unrelated flake.

@SparkQA
Copy link

SparkQA commented Dec 3, 2016

Test build #3465 has finished for PR 16092 at commit 6763a80.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Dec 6, 2016 •
edited
Loading

@foxish this needs to be rebased.

@foxish foxish force-pushed the fix-scheduler-structure-2 branch from 6763a80 to 3e78591 Compare December 6, 2016 16:47
@foxish
Copy link
Contributor Author

foxish commented Dec 6, 2016

@vanzin Done. Thanks!

@SparkQA
Copy link

SparkQA commented Dec 6, 2016

Test build #69735 has finished for PR 16092 at commit 3e78591.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

vanzin
vanzin reviewed Dec 6, 2016
View reviewed changes
dev/sparktestsupport/modules.py Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to update L477 (and probably a similar change in the mesos entry).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vanzin If I understand correctly, the sbt test goals can stay the same?
I can see them defined the same way for other modules such as sketch (as sketch/test), but it is nested under common/.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. Nevermind, I thought those were file paths like these. So fine the way they are.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vanzin also rebased after recent changes to HEAD. Is this okay to merge?

@foxish foxish force-pushed the fix-scheduler-structure-2 branch from 3e78591 to 05c478c Compare December 7, 2016 00:12
@vanzin
Copy link
Contributor

vanzin commented Dec 7, 2016

LGTM, merging to master.

@asfgit asfgit closed this in 81e5619 Dec 7, 2016
@foxish
Copy link
Contributor Author

foxish commented Dec 7, 2016

Thanks @vanzin :)

@foxish foxish deleted the fix-scheduler-structure-2 branch December 7, 2016 00:30
@foxish foxish restored the fix-scheduler-structure-2 branch December 7, 2016 00:30
@SparkQA
Copy link

SparkQA commented Dec 7, 2016

Test build #69756 has finished for PR 16092 at commit 05c478c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

robert3005 pushed a commit to palantir/spark that referenced this pull request Dec 15, 2016
@foxish
[SPARK-18662] Move resource managers to separate directory
2d39b39 
## What changes were proposed in this pull request?

* Moves yarn and mesos scheduler backends to resource-managers/ sub-directory (in preparation for https://issues.apache.org/jira/browse/SPARK-18278)
* Corresponding change in top-level pom.xml.

Ref: apache#16061 (comment)

## How was this patch tested?

* Manual tests

/cc rxin

Author: Anirudh <[email protected]>

Closes apache#16092 from foxish/fix-scheduler-structure-2.
@yanboliang yanboliang mentioned this pull request Jan 16, 2017
Closed
ghost pushed a commit to dbtsai/spark that referenced this pull request Jan 17, 2017
@yanboliang
[MINOR][YARN] Move YarnSchedulerBackendSuite to resource-managers/yar…
84f0b64 
…n directory.

## What changes were proposed in this pull request?
apache#16092 moves YARN resource manager related code to resource-managers/yarn directory. The test case ```YarnSchedulerBackendSuite``` was added after that but with the wrong place. I move it to correct directory in this PR.

## How was this patch tested?
Existing test.

Author: Yanbo Liang <[email protected]>

Closes apache#16595 from yanboliang/yarn.
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
@foxish @uzadude
[SPARK-18662] Move resource managers to separate directory
62305b2 
## What changes were proposed in this pull request?

* Moves yarn and mesos scheduler backends to resource-managers/ sub-directory (in preparation for https://issues.apache.org/jira/browse/SPARK-18278)
* Corresponding change in top-level pom.xml.

Ref: apache#16061 (comment)

## How was this patch tested?

* Manual tests

/cc rxin

Author: Anirudh <[email protected]>

Closes apache#16092 from foxish/fix-scheduler-structure-2.
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
@yanboliang @uzadude
[MINOR][YARN] Move YarnSchedulerBackendSuite to resource-managers/yar…
643e3d7 
…n directory.

## What changes were proposed in this pull request?
apache#16092 moves YARN resource manager related code to resource-managers/yarn directory. The test case ```YarnSchedulerBackendSuite``` was added after that but with the wrong place. I move it to correct directory in this PR.

## How was this patch tested?
Existing test.

Author: Yanbo Liang <[email protected]>

Closes apache#16595 from yanboliang/yarn.
cmonkey pushed a commit to cmonkey/spark that referenced this pull request Feb 15, 2017
@yanboliang @cmonkey
[MINOR][YARN] Move YarnSchedulerBackendSuite to resource-managers/yar…
2f59c90 
…n directory.

## What changes were proposed in this pull request?
apache#16092 moves YARN resource manager related code to resource-managers/yarn directory. The test case ```YarnSchedulerBackendSuite``` was added after that but with the wrong place. I move it to correct directory in this PR.

## How was this patch tested?
Existing test.

Author: Yanbo Liang <[email protected]>

Closes apache#16595 from yanboliang/yarn.
@mgummelt
Copy link

@foxish @vanzin I wish you would have cc'd me on this. If you're going to do something significant to Mesos code, please cc me.

@foxish
Copy link
Contributor Author

foxish commented Feb 22, 2017

Will cc you on future issues. Apologies if this disrupted your workflow. It was meant to be a quick change only in code structure.

@foxish foxish deleted the fix-scheduler-structure-2 branch February 22, 2017 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vanzin vanzin vanzin left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@foxish @JoshRosen @SparkQA @srowen @tnachen @ash211 @vanzin @mgummelt