-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This is an umbrella issue to gather all improvements regarding statistics.
Describe the solution you'd like
The list below should probably be better prioritized:
- Moving cost based optimizations to physical planning #962
- Expressions should also evaluate on statistics #992
- better validate that the
column_statisticsvector is aligned on the schemafieldsvector (same size, same types...) when constructing theExecutionPlaninstance (ex Adapt column statistics API #717) - remove
total_byte_sizeas we are not using it OR better estimate it when we have both a fixed size type and thenum_rowsfor the output columns - replace the
is_exactfield at theStatisticslevel with per-field information - have more granularity in statistics that just
(value, is_exact): possible solutions are histograms (cf Spark CBOs) - fix the way
LocalLimitExecpropagates its inexact statistics (requires more granular statistics) - estimate statistics in CSV datasource
- estimate statistics in JSON dataource
- better estimate output statistics of hash_aggregate
- better estimate output statistics of hash_join
- better estimate output statistics of projection (requires Expressions should also evaluate on statistics #992)
- better estimate output statistics of window_agg
- better estimate output statistics of filters (requires more granular statistics, in particular histograms)
Additional context
Statistics are usually sourced at the datasource level, then propagated through the plan tree according to the types of nodes. They are used to choose between different logically equivalent plans or plan configurations. The more rules are implemented for propagating the statistics, the more information the optimizer will have to take good decisions. But at the same time, an overly complex abstraction that is not used by any optimization rule would bloat the code base and make it harder to maintain. For that reason, extensions of the statistics system should be driven by the addition of concrete optimization rules that require them.