[SPARK-3150] Fix NullPointerException in in Spark recovery: Add initializing default values in DriverInfo.init() #2062
Conversation
|
Can one of the admins verify this patch? |
|
Jenkins, this is ok to test. |
|
QA tests have started for PR 2062 at commit
|
|
QA tests have finished for PR 2062 at commit
|
There was a problem hiding this comment.
Not sure if I understand your intention. These are all transient and won't actually be serialized, so why do we need to set them every time we readObject?
There was a problem hiding this comment.
When a field is deserialized, transient fields become null value, not None, thats why NullPointerException happens.
There was a problem hiding this comment.
Similar approach is used in WorkerInfo
There was a problem hiding this comment.
Ah I see, thanks for the context.
|
FWIW, we don't have any Jenkins tests for Zookeeper-based multi-master FT. There's a docker-based set of integration tests in We should create a JIRA for proper automated testing of this, including a sub-task to a regression-test for this issue. |
|
@tanyatik Have you verified that this fixes the NPE you ran into? If so this LGTM |
|
Yes I did, this patch fixes NPE and Spark restarts successfully. |
|
This looks good to me, too. I'm merging this into |
…alizing default values in DriverInfo.init() The issue happens when Spark is run standalone on a cluster. When master and driver fall simultaneously on one node in a cluster, master tries to recover its state and restart spark driver. While restarting driver, it falls with NPE exception (stacktrace is below). After falling, it restarts and tries to recover its state and restart Spark driver again. It happens over and over in an infinite cycle. Namely, Spark tries to read DriverInfo state from zookeeper, but after reading it happens to be null in DriverInfo.worker. https://issues.apache.org/jira/browse/SPARK-3150 Author: Tatiana Borisova <[email protected]> Closes #2062 from tanyatik/spark-3150 and squashes the following commits: 9936043 [Tatiana Borisova] Add initializing default values in DriverInfo.init() (cherry picked from commit 70d8146) Signed-off-by: Josh Rosen <[email protected]>
…alizing default values in DriverInfo.init() The issue happens when Spark is run standalone on a cluster. When master and driver fall simultaneously on one node in a cluster, master tries to recover its state and restart spark driver. While restarting driver, it falls with NPE exception (stacktrace is below). After falling, it restarts and tries to recover its state and restart Spark driver again. It happens over and over in an infinite cycle. Namely, Spark tries to read DriverInfo state from zookeeper, but after reading it happens to be null in DriverInfo.worker. https://issues.apache.org/jira/browse/SPARK-3150 Author: Tatiana Borisova <[email protected]> Closes #2062 from tanyatik/spark-3150 and squashes the following commits: 9936043 [Tatiana Borisova] Add initializing default values in DriverInfo.init() (cherry picked from commit 70d8146) Signed-off-by: Josh Rosen <[email protected]>
…alizing default values in DriverInfo.init() The issue happens when Spark is run standalone on a cluster. When master and driver fall simultaneously on one node in a cluster, master tries to recover its state and restart spark driver. While restarting driver, it falls with NPE exception (stacktrace is below). After falling, it restarts and tries to recover its state and restart Spark driver again. It happens over and over in an infinite cycle. Namely, Spark tries to read DriverInfo state from zookeeper, but after reading it happens to be null in DriverInfo.worker. https://issues.apache.org/jira/browse/SPARK-3150 Author: Tatiana Borisova <[email protected]> Closes apache#2062 from tanyatik/spark-3150 and squashes the following commits: 9936043 [Tatiana Borisova] Add initializing default values in DriverInfo.init()
…he#2062) Implement DataFrame handling, matching the existing Catalog/Table handling in uc-spark-authz. apple-cloud-services/[email protected]
5 participants
The issue happens when Spark is run standalone on a cluster.
When master and driver fall simultaneously on one node in a cluster, master tries to recover its state and restart spark driver.
While restarting driver, it falls with NPE exception (stacktrace is below).
After falling, it restarts and tries to recover its state and restart Spark driver again. It happens over and over in an infinite cycle.
Namely, Spark tries to read DriverInfo state from zookeeper, but after reading it happens to be null in DriverInfo.worker.
https://issues.apache.org/jira/browse/SPARK-3150