This repository was archived by the owner on Jan 9, 2020. It is now read-only.
This repository was archived by the owner on Jan 9, 2020. It is now read-only.
Log the driver exit status explicitly #276
Description
Currently it's not so obvious to tell when the spark job has succeeded or failed in the output. Compare these two:
2017-05-17 02:53:48 INFO LoggingPodStatusWatcher:54 - State changed, new state:
pod name: spark-pi-1494989589236
namespace: default
labels: spark-app-id -> spark-pi-1494989589236, spark-app-name -> spark-pi, spark-driver -> spark-pi-1494989589236
pod uid: f6d399dc-3aab-11e7-99db-08002774bad8
creation time: 2017-05-17T02:53:11Z
service account name: default
volumes: spark-submission-secret-volume, default-token-q8ktn
node name: 172.17.4.201
start time: 2017-05-17T02:53:11Z
container images: example.com/lins05/spark-k8s-driver:latest
phase: Succeeded
status: [ContainerStatus(containerID=docker://4109f34b2e13bd01e69e799d7d90bd3349b497ed357f0e148fefdb0d0f21210a, image=example.com/lins05/spark-k8s-driver:late
st, imageID=docker-pullable://example.com/lins05/spark-k8s-driver@sha256:01ef2073c94954f845b7abe01da2e3bd6f92d18c734a484d09741f334e3a1a95, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://4109f34b2e13bd01e69e799d7d90bd3349b497ed357f0e148fefdb0d0f21210a, exitCode=0, finishedAt=2017-05-17T02:53:48Z, message=null, reason=Completed, signal=null, startedAt=2017-05-17T02:53:12Z, additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
2017-05-17 02:53:48 INFO Client:54 - Application spark-pi-1494989589236 finished.
2017-05-17 02:53:48 INFO ShutdownHookManager:54 - Shutdown hook called
v.s.
2017-05-17 02:57:11 INFO LoggingPodStatusWatcher:54 - Application status for spark-pi-1494989799975 (phase: Running)
2017-05-17 02:57:15 INFO LoggingPodStatusWatcher:54 - State changed, new state:
pod name: spark-pi-1494989799975
namespace: default
labels: spark-app-id -> spark-pi-1494989799975, spark-app-name -> spark-pi, spark-driver -> spark-pi-1494989799975
pod uid: 747368bd-3aac-11e7-99db-08002774bad8
creation time: 2017-05-17T02:56:42Z
service account name: default
volumes: spark-submission-secret-volume, default-token-q8ktn
node name: 172.17.4.201
start time: 2017-05-17T02:56:42Z
container images: example.com/lins05/spark-k8s-driver:latest
phase: Failed
status: [ContainerStatus(containerID=docker://0efd66beb19bd258625567fb34ad5b71f95a7fcf0bb82e0f123cb429b17a0f56, image=example.com/lins05/spark-k8s-driver:latest, imageID=docker-pullable://example.com/lins05/spark-k8s-driver@sha256:01ef2073c94954f845b7abe01da2e3bd6f92d18c734a484d09741f334e3a1a95, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://0efd66beb19bd258625567fb34ad5b71f95a7fcf0bb82e0f123cb429b17a0f56, exitCode=1, finishedAt=2017-05-17T02:57:14Z, message=null, reason=Error, signal=null, startedAt=2017-05-17T02:56:43Z, additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
2017-05-17 02:57:15 INFO Client:54 - Application spark-pi-1494989799975 finished.
2017-05-17 02:57:15 INFO ShutdownHookManager:54 - Shutdown hook called
The former one finished successfully while the latter failed. But you need to look very hard to locate the value of phase: to recognize that.
I suggest add the exit status into this line, e.g.:
2017-05-17 02:57:15 INFO Client:54 - Application spark-pi-1494989799975 finished successfully
Or
2017-05-17 02:57:15 INFO Client:54 - Application spark-pi-1494989799975 failed with exit code 1. You may want to check the driver pod logs.