Skip to content

[SPARK-13492][MESOS] Configurable Mesos framework webui URL. #11369

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

[SPARK-13492][MESOS] Configurable Mesos framework webui URL. #11369

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9fbc059
Mesos: Configurable Mesos framework webui URL.
Jan 22, 2016
1245008
fix NPE
Feb 26, 2016
06334a4
scala style fix
Feb 26, 2016
1ffc6d9
MesosClusterDispatcher: remove --webui-url config
Feb 29, 2016
46ea0ab
history-server: use relative urls
Feb 29, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
<td class="rowGroupColumn"><span title="{{id}}"><a href="{{url}}">{{id}}</a></span></td>
<td class="rowGroupColumn">{{name}}</td>
{{#attempts}}
<td class="attemptIDSpan"><a href="/history/{{id}}/{{attemptId}}/">{{attemptId}}</a></td>
<td class="attemptIDSpan"><a href="history/{{id}}/{{attemptId}}/">{{attemptId}}</a></td>
<td>{{startTime}}</td>
<td>{{endTime}}</td>
<td><span title="{{duration}}" class="durationClass">{{duration}}</span></td>
Expand Down
8 changes: 4 additions & 4 deletions core/src/main/resources/org/apache/spark/ui/static/historypage.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ $(document).ready(function() {
requestedIncomplete = getParameterByName("showIncomplete", searchString);
requestedIncomplete = (requestedIncomplete == "true" ? true : false);

$.getJSON("/api/v1/applications", function(response,status,jqXHR) {
$.getJSON("api/v1/applications", function(response,status,jqXHR) {
var array = [];
var hasMultipleAttempts = false;
for (i in response) {
Expand Down Expand Up @@ -139,9 +139,9 @@ $(document).ready(function() {

var url = null
if (maxAttemptId == null) {
url = "/history/" + id + "/"
url = "history/" + id + "/"
} else {
url = "/history/" + id + "/" + maxAttemptId + "/"
url = "history/" + id + "/" + maxAttemptId + "/"
}

var app_clone = {"id" : id, "name" : name, "url" : url, "attempts" : [attempt]};
Expand All @@ -150,7 +150,7 @@ $(document).ready(function() {
}

var data = {"applications": array}
$.get("/static/historypage-template.html", function(template) {
$.get("static/historypage-template.html", function(template) {
Copy link
Contributor

@tnachen tnachen Mar 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the problem with the /?

Copy link
Contributor

@tnachen tnachen Mar 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussing with @s-urbaniak directly, I see it's a bug where the link doesn't work when application proxy is set.

historySummary.append(Mustache.render($(template).filter("#history-summary-template").html(),data));
var selector = "#history-summary-table";
var conf = {
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ private[mesos] class MesosClusterDispatcher(

def start(): Unit = {
webUi.bind()
scheduler.frameworkUrl = webUi.activeWebUiUrl
scheduler.frameworkUrl = conf.get("spark.mesos.dispatcher.webui.url", webUi.activeWebUiUrl)
scheduler.start()
server.start()
}
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcherArguments.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ private[mesos] class MesosClusterDispatcherArguments(args: Array[String], conf:
port = value
parse(tail)

case ("--webui-port" | "-p") :: IntParam(value) :: tail =>
case ("--webui-port") :: IntParam(value) :: tail =>
Copy link
Contributor

@tnachen tnachen Mar 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove this?

Copy link
Author

@s-urbaniak s-urbaniak Mar 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This conflicts with the --port setting both having the -p shortcut. I think this is a bug and must have been broken for users anyway.

Copy link
Contributor

@dragos dragos Mar 3, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that -p would always be matched by the pattern above, "--port" | "-p". So removing it here wouldn't have any user-visible effects.

webUiPort = value
parse(tail)

Expand Down
3 changes: 2 additions & 1 deletion ...src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,8 @@ private[spark] class CoarseMesosSchedulerBackend(
sc.sparkUser,
sc.appName,
sc.conf,
sc.ui.map(_.appUIAddress))
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
)
startScheduler(driver)
}

Expand Down
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,8 @@ private[spark] class MesosSchedulerBackend(
sc.sparkUser,
sc.appName,
sc.conf,
sc.ui.map(_.appUIAddress))
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
)
startScheduler(driver)
}

Expand Down
30 changes: 30 additions & 0 deletions ...est/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackendSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,35 @@ class CoarseMesosSchedulerBackendSuite extends SparkFunSuite
verify(driver, times(1)).killTask(createTaskId("0"))
}

test("weburi is set in created scheduler driver") {
setBackend()
val taskScheduler = mock[TaskSchedulerImpl]
when(taskScheduler.sc).thenReturn(sc)
val driver = mock[SchedulerDriver]
when(driver.start()).thenReturn(Protos.Status.DRIVER_RUNNING)
val securityManager = mock[SecurityManager]

val backend = new CoarseMesosSchedulerBackend(taskScheduler, sc, "master", securityManager) {
override protected def createSchedulerDriver(
masterUrl: String,
scheduler: Scheduler,
sparkUser: String,
appName: String,
conf: SparkConf,
webuiUrl: Option[String] = None,
checkpoint: Option[Boolean] = None,
failoverTimeout: Option[Double] = None,
frameworkId: Option[String] = None): SchedulerDriver = {
markRegistered()
assert(webuiUrl.isDefined)
assert(webuiUrl.get.equals("http://webui"))
driver
}
}

backend.start()
}

private def verifyDeclinedOffer(driver: SchedulerDriver,
offerId: OfferID,
filter: Boolean = false): Unit = {
Expand Down Expand Up @@ -316,6 +345,7 @@ class CoarseMesosSchedulerBackendSuite extends SparkFunSuite
.setMaster("local[*]")
.setAppName("test-mesos-dynamic-alloc")
.setSparkHome("/path")
.set("spark.mesos.driver.webui.url", "http://webui")

if (sparkConfVars != null) {
for (attr <- sparkConfVars) {
Expand Down
37 changes: 36 additions & 1 deletion .../src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ import scala.collection.JavaConverters._
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer

import org.apache.mesos.{Protos, Scheduler, SchedulerDriver}
import org.apache.mesos.Protos._
import org.apache.mesos.Protos.Value.Scalar
import org.apache.mesos.SchedulerDriver
import org.mockito.{ArgumentCaptor, Matchers}
import org.mockito.Matchers._
import org.mockito.Mockito._
Expand All @@ -42,6 +42,41 @@ import org.apache.spark.scheduler.cluster.ExecutorInfo

class MesosSchedulerBackendSuite extends SparkFunSuite with LocalSparkContext with MockitoSugar {

test("weburi is set in created scheduler driver") {
val conf = new SparkConf
conf.set("spark.mesos.driver.webui.url", "http://webui")
conf.set("spark.app.name", "name1")

val sc = mock[SparkContext]
when(sc.conf).thenReturn(conf)
when(sc.sparkUser).thenReturn("sparkUser1")
when(sc.appName).thenReturn("appName1")

val taskScheduler = mock[TaskSchedulerImpl]
val driver = mock[SchedulerDriver]
when(driver.start()).thenReturn(Protos.Status.DRIVER_RUNNING)

val backend = new MesosSchedulerBackend(taskScheduler, sc, "master") {
override protected def createSchedulerDriver(
masterUrl: String,
scheduler: Scheduler,
sparkUser: String,
appName: String,
conf: SparkConf,
webuiUrl: Option[String] = None,
checkpoint: Option[Boolean] = None,
failoverTimeout: Option[Double] = None,
frameworkId: Option[String] = None): SchedulerDriver = {
markRegistered()
assert(webuiUrl.isDefined)
assert(webuiUrl.get.equals("http://webui"))
driver
}
}

backend.start()
}

test("Use configured mesosExecutor.cores for ExecutorInfo") {
val mesosExecutorCores = 3
val conf = new SparkConf
Expand Down
16 changes: 16 additions & 0 deletions docs/running-on-mesos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,6 +390,22 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
<tr>
<td><code>spark.mesos.driver.webui.url</code></td>
<td><code>(none)</code></td>
<td>
Set the Spark Mesos driver webui_url for interacting with the framework.
If unset it will point to Spark's internal web UI.
</td>
</tr>
<tr>
<td><code>spark.mesos.dispatcher.webui.url</code></td>
Copy link
Contributor

@skonto skonto Feb 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worths to mention that properties coming from a driver submission request are propagated from here:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala#L87-L97

Also there are properties coming from MesosClusterScheduler's conf (initialized from MesosClusterDispatcher conf- https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala#L62):

eg. spark.mesos.uris at
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L407

Both of these sets of properties seem to merge to form the final set of driver's spark properties IMHO these should be isolated or clarified into the docs the concept seems complicated.

Copy link

@mgummelt mgummelt Feb 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is complicated and opaque. A JIRA for this problem is welcome.

<td><code>(none)</code></td>
<td>
Set the Spark Mesos dispatcher webui_url for interacting with the framework.
If unset it will point to Spark's internal web UI.
</td>
</tr>
</table>

# Troubleshooting and Debugging
Expand Down