pipe column orderings into pruning predicate creation #15821
Conversation
github-actions
bot
added
optimizer
| let pruning_predicate = PruningPredicate::try_new( | ||
| Arc::clone(predicate), | ||
| self.schema(), | ||
| vec![ColumnOrdering::Unknown; self.schema().fields().len()], | ||
| )?; |
There was a problem hiding this comment.
We could add a new signature to avoid API churn, but I wanted to make it explicit for now to see all of the callsites
adriangb
force-pushed
the
column-orders-parquet-stats
branch
from
3e15afe to
8c2ceb1
Compare
|
@adriangb please check out pydantic#28 |
| return expr; | ||
| } | ||
|
|
||
| // Special handlng for floats. Because current Parquet statistics do not allow NaN, and |
There was a problem hiding this comment.
This block is why we need to get the column ordering info passed down. Here we know which column is being pruned and with which operation. For floats we can disallow pruning because the Parquet stats are incomplete. We can also skip pruning if the stats are not valid because an ordering is not defined for the type.
|
@etseidl I'm sorry we haven't made any progress here. I see there are a lot of merge conflicts but we do now finally have all of the building blocks in place since we evaluate predicates against the physical file schema. Should we revive this? |
|
I'll be able to revisit this early in the week. It's been a back burner issue for me lately because of the slow progress on parquet format changes. It seems the sort order keeps popping up, though, so I think we'll want this change eventually. |
|
marking as draft as this doesn't appear to be waiting on review |
3 participants
NaNis present #15812