[SPARK-19641][SQL] JSON schema inference in DROPMALFORMED mode produces incorrect schema for non-array/object JSONs #17492
Conversation
|
cc @NathanHowell and @cloud-fan. |
|
Test build #75421 has started for PR 17492 at commit |
|
retest this please |
|
Test build #75425 has finished for PR 17492 at commit
|
| } | ||
| } | ||
|
|
||
| private def withParseMode( |
There was a problem hiding this comment.
shall we embed this method in withCorruptField?
|
Test build #75458 has finished for PR 17492 at commit
|
| struct | ||
| } | ||
|
|
||
| case FailFastMode => |
There was a problem hiding this comment.
When will we hit this branch? Seems never?
There was a problem hiding this comment.
It looks possible to run this line. I added a test in 1936L in JsonSuite. In more details, if the json is a valid one but not a object of an areay of object, it will infer not StructType per record. If one of the types is not a struct type and failfast mode is enabled, we will hit this line.
I just checked as below:
| .collect() | ||
| } | ||
| assert(exceptionOne.getMessage.contains("Failed to parse a value")) | ||
| assert(exceptionOne.getMessage.contains("Failed to infer a common schema")) |
There was a problem hiding this comment.
Up to my knowledge, this test case throws an exception when actually parsing the data before. Now, I intended to check the exception in schema inference as it looks parsing data case is covered below.
| spark.read | ||
| .option("mode", "FAILFAST") | ||
| .json(corruptRecords) | ||
| .collect() |
There was a problem hiding this comment.
I intended to remove this line to make sure it happens in schrma inference. I think this is not related with the change here.
|
thanks, merging to master! |
|
Thank you @cloud-fan. |
3 participants
What changes were proposed in this pull request?
Currently, when we infer the types for vaild JSON strings but object or array, we are producing empty schemas regardless of parse modes as below:
This PR proposes to handle parse modes in type inference.
After this PR,
This PR is based on NathanHowell@e233fd0 and I and @NathanHowell talked about this in https://issues.apache.org/jira/browse/SPARK-19641
How was this patch tested?
Unit tests in
JsonSuitefor bothDROPMALFORMEDandFAILFASTmodes.