Skip to content

[SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. #10086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. #10086

wants to merge 2 commits into from

Conversation

@tnachen
Copy link
Contributor

@tnachen tnachen commented Dec 2, 2015

Adding more documentation about submitting jobs with mesos cluster mode.

@tnachen tnachen changed the title Add documentation about submitting Spark with mesos cluster mode. [MESOS]Add documentation about submitting Spark with mesos cluster mode. Dec 2, 2015
@tnachen
Copy link
Contributor Author

tnachen commented Dec 2, 2015

@andrewor14

andrewor14
andrewor14 reviewed Dec 2, 2015
View reviewed changes
docs/submitting-applications.md Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do they know at this point what a MesosClusterDispatcher is?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnachen can you update this assuming the user doesn't know anything about the dispatcher?

@andrewor14
Copy link
Contributor

looks alright

@SparkQA
Copy link

SparkQA commented Dec 2, 2015

Test build #47028 has finished for PR 10086 at commit 448d706.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@andrewor14
Copy link
Contributor

@dragos you mentioned you had difficulty running mesos in cluster mode. Does this additional documentation help? What were the issues you ran into?

@tnachen
Copy link
Contributor Author

tnachen commented Dec 3, 2015

@andrewor14 I don't tihnk this doc helps his problem, he's running into issues where Spark configuration is not being passed down. I'm pushing a new update here.

@tnachen tnachen force-pushed the mesos_supervise_docs branch from 448d706 to 5d612d0 Compare December 3, 2015 19:29
@SparkQA
Copy link

SparkQA commented Dec 3, 2015

Test build #47158 has finished for PR 10086 at commit 5d612d0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tnachen
Copy link
Contributor Author

tnachen commented Dec 4, 2015

@andrewor14 pushed an update today, PTAL

@dragos
Copy link
Contributor

dragos commented Dec 4, 2015

Thanks, this is a good example! Can you please update also running-on-mesos.md, that part is where most people running on Mesos will look first. I think an example like this one, and one showing how to start the dispatcher would be great.

@dragos
Copy link
Contributor

dragos commented Dec 4, 2015

@andrewor14 the issue in the other PR is not due to missing docs, unfortunately. I will post my errors in that PR (I reached out to Tim via Slack earlier, but maybe it makes sense to have such discussions in the open)

@tnachen tnachen force-pushed the mesos_supervise_docs branch from 5d612d0 to 952f116 Compare December 8, 2015 07:22
@tnachen
Copy link
Contributor Author

tnachen commented Dec 8, 2015

@dragos just updated the docs

@SparkQA
Copy link

SparkQA commented Dec 8, 2015

Test build #47318 has finished for PR 10086 at commit 952f116.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * If you like to run the MesosClusterDispatcher with Marathon, you need to run the MesosClusterDispatcher in the foreground (i.e:bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher).\n

@dragos
Copy link
Contributor

dragos commented Dec 8, 2015

LGTM

srowen
srowen reviewed Dec 8, 2015
View reviewed changes
docs/running-on-mesos.md Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nits: "an daemon" -> "a daemon", url -> URL, "i.e" -> "i.e.". You might back-tick MesosClusterDispatcher too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I would backtick the dispatcher

@andrewor14
Copy link
Contributor

@tnachen can you address the comments? I'll merge this once you do so.

@srowen
Copy link
Member

srowen commented Dec 15, 2015

Ping @tnachen

@tnachen tnachen force-pushed the mesos_supervise_docs branch from 952f116 to 7f8d7b6 Compare December 15, 2015 13:58
@tnachen
Copy link
Contributor Author

tnachen commented Dec 15, 2015

@srowen @andrewor14 sorry for the delay, it's updated now.

@SparkQA
Copy link

SparkQA commented Dec 15, 2015

Test build #47741 has finished for PR 10086 at commit 7f8d7b6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * If you like to run theMesosClusterDispatcherwith Marathon, you need to run theMesosClusterDispatcherin the foreground (i.e:bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher).\n

@srowen
Copy link
Member

srowen commented Dec 15, 2015

Last but not least, should we go ahead and make a JIRA for this? it's pretty minor but not trivial

@tnachen tnachen changed the title [MESOS]Add documentation about submitting Spark with mesos cluster mode. [SPARK-12351][MESOS]Add documentation about submitting Spark with mesos cluster mode. Dec 16, 2015
@tnachen
Copy link
Contributor Author

tnachen commented Dec 16, 2015

Created jira and updated title @srowen

@andrewor14
Copy link
Contributor

Merged into master and 1.6

@asfgit asfgit closed this in c2de99a Dec 16, 2015
asfgit pushed a commit that referenced this pull request Dec 16, 2015
@tnachen
[SPARK-12351][MESOS] Add documentation about submitting Spark with me…
2c324d3 
…sos cluster mode.

Adding more documentation about submitting jobs with mesos cluster mode.

Author: Timothy Chen <[email protected]>

Closes #10086 from tnachen/mesos_supervise_docs.

(cherry picked from commit c2de99a)
Signed-off-by: Andrew Or <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tnachen @andrewor14 @SparkQA @dragos @srowen