1000x slowdown opening parquet file due to partitions

[asayers]

Describe the bug

parquet metadata load time was 15 seconds with default configuration. Setting target_partitions to 1 brought it down to 15 milliseconds.

With the default config I was seeing 20 file groups in the DataSourceExec (since my machine has 20 cores) and loading the metadata was taking forever. Forcing target_partitions to 1 fixed it: I now see just 1 file group in the DataSourceExec, metadata load time is down. (Added bonus: the plan no longer requires a MergeExec, which was the slowest part of the whole query).

To Reproduce

The query I was running is very basic: just filter + sort + select. The parquet file is 130.00 MiB
with 1.08 MiB of metadata (6.42 MiB when expanded in memory).

Expected behavior

I wouldn't have thought going from 1 file group to 20 would slow down metadata parsing by 1000x.

Additional context

No response

[jatin510]

take

[asayers]

I think the correct solution was for me to enable the metadata cache (I haven't confirmed this). So perhaps the "bug" (if there is one) is just that the metadata cache is off by default?

[alamb]

Perhaps this is related to the default value of statistics too:

#16447

Do you have a reproducer you can share @asayers ?

[asayers]

I can't share the data I was hitting this case with. I could try to make a synthetic reproducer, but work is very busy right now so I might not get to it for a few months.

[alamb]

We have added a new metadata cache (will ship in 50.0.0):

• [Epic] Enable parquet metadata cache by default #17000

I wonder if you could try your reproducer with what is on main and see if the query runs faster