Support for PyArrow parquet partitioning

[bsprenger]

Hi all! 😄 Is there support for loading parquet datasets with partitioning as described in the arrow docs?

Particularly, when I use load_dataset(), it currently seems not to be able to reconstruct the partition columns as described in the doc. For example, if I create a folder structure:

dataset_name/
  year=2007/
    month=01/
       data0.parquet
       data1.parquet
       ...
    month=02/
       data0.parquet
       data1.parquet
       ...
    month=03/
    ...
  year=2008/
    month=01/
    ...
  ...

using Pyarrow datasets I can load this dataset, specify the partition columns, and it will automatically add the year, month etc as columns, but I don't see a way to do this through load_dataset (which ignores the key=value folder structure). Does someone know if there's a way around this?

Thanks so much! 😄

[Nintorac]

Hey, did you ever figure anything out for this?

[bsprenger]

Sorry for the delayed response. Per this thread, it does not appear to be supported :( And from what I can tell looking in the Datasets internals, this appears to still be the case.