[SPARK-20165][SS] Resolve state encoder's deserializer in driver in FlatMapGroupsWithStateExec #17488
Conversation
| encoderSerializer | ||
| } | ||
| } | ||
| private val stateDeserializer = stateEncoder.resolveAndBind().deserializer |
There was a problem hiding this comment.
Can you comment that this has to happen on the driver so we don't forget why its here and move it?
| try { | ||
| // Add data and get the source where it was added, and the expected offset of the | ||
| // added data. | ||
| if (currentStream != null && |
There was a problem hiding this comment.
Can you comment here too. I had no idea what this was doing until read the PR description.
|
Test build #75408 has finished for PR 17488 at commit
|
|
Test build #75410 has finished for PR 17488 at commit
|
|
Test build #3623 has finished for PR 17488 at commit
|
|
Test build #3622 has finished for PR 17488 at commit
|
|
Test build #3624 has finished for PR 17488 at commit
|
|
Test build #75412 has finished for PR 17488 at commit
|
|
Test build #3626 has finished for PR 17488 at commit
|
|
Test build #3625 has finished for PR 17488 at commit
|
|
The only failure was in completely unrelated HiveMetadataCatalogSuite |
|
Test build #3627 has finished for PR 17488 at commit
|
|
Test build #3629 has finished for PR 17488 at commit
|
|
Test build #3628 has finished for PR 17488 at commit
|
3 participants
What changes were proposed in this pull request?
Encoder's deserializer must be resolved at the driver where the class is defined. Otherwise there are corner cases using nested classes where resolving at the executor can fail.
Fixed flaky test related to processing time timeout. The flakiness is caused because the test thread (that adds data to memory source) has a race condition with the streaming query thread. When testing the manual clock, the goal is to add data and increment clock together atomically, such that a trigger sees new data AND updated clock simultaneously (both or none). This fix adds additional synchronization in when adding data; it makes sure that the streaming query thread is waiting on the manual clock to be incremented (so no batch is currently running) before adding data.
Added
testQuietlyon some tests that generate a lot of error logs.How was this patch tested?
Multiple runs on existing unit tests