Wrong results when `pushdown_filters` is enabled (starting in 48.0.0)

[alamb]

Describe the bug

There are a lot of linked issues and discussions here, so it's quite possible that I missed an existing bug report, but I wanted to report an issue I encountered when enabling datafusion.execution.parquet.pushdown_filters.

Originally posted by @ianthetechie in #3463

To Reproduce

I've put together an MRE that demonstrates it with what seems like a trivial query (I initially had something MUCH more complex in a real app, but even this gets wrong results): https://github.com/ianthetechie/datafusion-predicate-pushdown-mre.

Expected behavior

I am expecting the number to be 72, but the count comes to 1245 with datafusion.execution.parquet.pushdown_filters set to true.

Additional context

No response

[ianthetechie]

@xudong963 asked in the other issue whether the issue existed in 47.0. I just tested and 47.0 gives the correct output. So the issue is somewhere between the 47.0 and 48.0 releases.

[alamb]

Thanks @ianthetechie -- I have marked this issue as a regression in 48

[adriangb]

@ianthetechie I looked at your example repo (thanks so much for providing an MRE) but I don't see the data anywhere. Maybe I'm missing something but is categories a known dataset? Without that it may be hard to repro. The query looks relatively simple and I feel like similar queries are extensively tested.

[ianthetechie]

@adriangb The dataset is loaded here: https://github.com/ianthetechie/datafusion-predicate-pushdown-mre/blob/0abc24cd2f728a88e7496f0294554f0858d7c79b/src/context.rs#L12. It's coming from a public S3 bucket, and is a bit large to put in a git repo (though not excessive; the query is very fast), so I didn't bother committing the data. The example should run as-is though. Let me know if you have any issues / find otherwise!

[adriangb]

Sorry I missed that!

[ianthetechie]

No worries; I thought I was doing a favor trying to separate things into different files, but obviously that didn't work out ;)

[adriangb]

I can reproduce running this SQL in datafusion-cli:

-- Create external table pointing to the provided parquet file
CREATE EXTERNAL TABLE categories 
STORED AS PARQUET
LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-06-10/categories/parquet/categories.zstd.parquet';

-- First, inspect the data structure
SELECT * FROM categories LIMIT 5;

-- Test with pushdown DISABLED
SET datafusion.execution.parquet.pushdown_filters = false;

-- Query with pushdown DISABLED - establish baseline
SELECT COUNT(*)
FROM categories
WHERE level1_category_name = 'Arts and Entertainment';

-- Enable pushdown
SET datafusion.execution.parquet.pushdown_filters = true;

-- Same query with pushdown ENABLED - should match baseline
SELECT COUNT(*)
FROM categories
WHERE level1_category_name = 'Arts and Entertainment';

-- Cleanup
DROP TABLE categories;

[adriangb]

A bisect revealed the offending commit is #15769. I am investigating further.

[adriangb]

This small change to the query fixes it:

-- Enable pushdown
SET datafusion.execution.parquet.pushdown_filters = true;

-- Create external table pointing to the provided parquet file
CREATE EXTERNAL TABLE categories 
STORED AS PARQUET
LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-06-10/categories/parquet/categories.zstd.parquet';

-- Same query with pushdown ENABLED - should match baseline
SELECT COUNT(*)
FROM categories
WHERE level1_category_name = 'Arts and Entertainment';

-- Cleanup
DROP TABLE categories;

[alamb]

Thank you for looking into this @adriangb