Log the driver exit status explicitly #282
Conversation
| private def status: String = pod.map(_.getStatus().getContainerStatuses().toString()) | ||
| .getOrElse("unknown") | ||
|
|
||
| private var driverPodExitCode: Int = 0 |
There was a problem hiding this comment.
make this an Option[Int] and only set when we get an exit code. Otherwise if something goes wrong clients of LoggingPodStatusWatcher might think exit code is 0 when it's not
| }.mkString("") | ||
| } | ||
|
|
||
| def getDriverPodExitCode: Int = { |
|
We shouldn't be making changes to V1 submission. Any changes should be done to V2 submission instead. Incidentally this reminds me that V2 submission currently always runs in fire and forget mode - continuous monitoring mode needs to be re-done there. |
|
@mccheah can we make changes to both? I don't think V2 is ready for people to move over to yet, though coming closer |
|
I think the logging pod status is the last feature that isn't in V2 but is in V1. It would be good to add that to V2 and then remove the V1 code path entirely. |
|
@mccheah I'd ideally like to get confirmation from someone that's not us that they've run V2 and it worked, before deleting V1 |
|
Except that would require us or someone building a custom version of Spark to actually change the code path itself. SparkSubmit currently is coded to use the V1 main class, but we would need it to switch to the V2 main class. And we don't want to make the code path configurable. |
|
One has to change this line in SparkSubmit to point to the V2 submission client instead. Then, the driver image must correspond to what we have in this Dockerfile. Finally, all of the parameters for submission need to correspond to what's used in V2, which is currently largely undocumented. When we transition to V2 we will document everything that is required. |
|
I created #285 to discuss the actual transition. I'll work on a PR that switches the code paths and the documentation. |
|
Included in #283 |
…ache [NOSQUASH] Resync Apache
4 participants
Close #276 .
How is the patch tested?
Manually add logical errors to the SparkPi example and check the output.