Error When Querying Partitioned JSON Table

[devinjdangelo]

Describe the bug

I am seeing the following error when attempting to query a table of hive style partitioned JSON files via datafusion-cli:

Arrow error: Json error: Encountered unmasked nulls in non-nullable StructArray child: Field { name: "a", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }

To Reproduce

I have a table with the following directory structure:

dev@dev:~/arrow-datafusion/test_table$ ls -lhR
.:
total 12K
drwxrwxr-x 2 dev dev 4.0K Oct 12 17:35 'a=2'
drwxrwxr-x 2 dev dev 4.0K Oct 12 17:35 'a=4'
drwxrwxr-x 2 dev dev 4.0K Oct 12 17:35 'a=6'

'./a=2':
total 4.0K
-rw-rw-r-- 1 dev dev 20 Oct 12 17:35 tn0Sfag4abaDm6i2.json

'./a=4':
total 4.0K
-rw-rw-r-- 1 dev dev 20 Oct 12 17:35 tn0Sfag4abaDm6i2.json

'./a=6':
total 4.0K
-rw-rw-r-- 1 dev dev 20 Oct 12 17:35 tn0Sfag4abaDm6i2.json

And the JSON files look like:

dev@dev:~/arrow-datafusion/test_table$ cat a\=2/tn0Sfag4abaDm6i2.json 
{"b":"1"}
{"b":"1"}

Attempting to query like the following fails:

dev@dev:~/arrow-datafusion$ datafusion-cli
DataFusion CLI v32.0.0
❯ CREATE EXTERNAL TABLE
json_test(a string, b string)
STORED AS json
LOCATION './test_table'
PARTITIONED BY (a);
0 rows in set. Query took 0.001 seconds.

❯ select * from json_test;
Arrow error: Json error: Encountered unmasked nulls in non-nullable StructArray child: Field { name: "a", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }
❯ 

Querying without the partitions defined works as expected:

dev@dev:~/arrow-datafusion$ datafusion-cli
DataFusion CLI v32.0.0
❯ CREATE EXTERNAL TABLE
json_test(b string)
STORED AS json
LOCATION './test_table';
0 rows in set. Query took 0.001 seconds.

❯ select * from json_test;
+---+
| b |
+---+
| 3 |
| 3 |
| 1 |
| 1 |
| 5 |
| 5 |
+---+
6 rows in set. Query took 0.002 seconds.

The exact same table structure DDL works for CSV and parquet files, but not JSON.

Expected behavior

The above json query should work.

Additional context

I discovered this while working on https://github.com/apache/arrow-datafusion/pull/7801/files#diff-0580d65ff5db0c78c1fa4cf693f2567e7d2394923412687560614836401c223f, and there are additional relevant tests in this PR.

[alamb]

This error appears to come from arrow-rs: https://github.com/apache/arrow-rs/blob/90bc5ec96b5ae5162f469f9784dde7b1a53a5bdd/arrow-json/src/reader/struct_array.rs#L140

@tustvold any thoughts? Is the issue the json reader was told the schema was not nullable, but it actually was nullable?

[devinjdangelo]

Is the issue the json reader was told the schema was not nullable, but it actually was nullable?

There are no nulls in the table, so the reader shouldn't be encountering nulls. 🤔

[tustvold]

If a field isn't present for a row, it will be interpreted as a null. Perhaps you could loosen the nullability restriction and see where the null turns up?

[theelderbeever]

I think this is related to #7686. If you don't do SELECT * or any of the partition columns it will work. But the projection of the partitioned column causes issue with the NDJsonExec implementation.

[Tangruilin]

@tustvold It seems that the bug is not fixed till now.

[Tangruilin]

If it is, maybe you can assigned it to me, and i will fix it

[alamb]

Assigned -- thank you @Tangruilin