Move nested union optimization from plan builder to logical optimizer #7695
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maruschin
force-pushed
the
add_union_optimization
branch
from
e8e6a32 to
79e20c6
Compare
maruschin
force-pushed
the
add_union_optimization
branch
3 times, most recently
from
4c42dc4 to
770bf79
Compare
maruschin
changed the title
maruschin
force-pushed
the
add_union_optimization
branch
from
a3a2484 to
b926721
Compare
maruschin
force-pushed
the
add_union_optimization
branch
from
b926721 to
7259e9f
Compare
Dandandan
reviewed
Sep 29, 2023
alamb
reviewed
Sep 29, 2023
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you @maruschin -- this is looking quite close. The basic idea and testing looks very solid. I think several of the CI failures on this PR are due to #7701, so merging / rebasing with master will help.
I also double checked this branch gets the same plan as main
❯ explain select * from (select 1 UNION ALL (select 2 UNION ALL select 3));
+---------------+----------------------------------------+
| plan_type | plan |
+---------------+----------------------------------------+
| logical_plan | Union |
| | Projection: Int64(1) AS Int64(1) |
| | EmptyRelation |
| | Projection: Int64(2) AS Int64(1) |
| | EmptyRelation |
| | Projection: Int64(3) AS Int64(1) |
| | EmptyRelation |
| physical_plan | UnionExec |
| | ProjectionExec: expr=[1 as Int64(1)] |
| | EmptyExec: produce_one_row=true |
| | ProjectionExec: expr=[2 as Int64(1)] |
| | EmptyExec: produce_one_row=true |
| | ProjectionExec: expr=[3 as Int64(1)] |
| | EmptyExec: produce_one_row=true |
| | |
+---------------+----------------------------------------+
2 rows in set. Query took 0.008 seconds.I found a few sqllogictests seem to fail, specifically
cargo test --test sqllogictests -- timestamp
Running "timestamps.slt"
External error: query result mismatch:
[SQL] SELECT date_trunc('hour', TIMESTAMPTZ '2000-01-01T00:00:00+00:45') as ts_irregular_offset
UNION ALL
SELECT date_trunc('hour', TIMESTAMPTZ '2000-01-01T00:00:00+00:30') as ts_irregular_offset
UNION ALL
SELECT date_trunc('hour', TIMESTAMPTZ '2000-01-01T00:00:00+00:15') as ts_irregular_offset
UNION ALL
SELECT date_trunc('hour', TIMESTAMPTZ '2000-01-01T00:00:00-00:15') as ts_irregular_offset
UNION ALL
SELECT date_trunc('hour', TIMESTAMPTZ '2000-01-01T00:00:00-00:30') as ts_irregular_offset
UNION ALL
SELECT date_trunc('hour', TIMESTAMPTZ '2000-01-01T00:00:00-00:45') as ts_irregular_offset
[Diff] (-expected|+actual)
- 1999-12-31T23:00:00Z
- 1999-12-31T23:00:00Z
- 1999-12-31T23:00:00Z
- 2000-01-01T00:00:00Z
- 2000-01-01T00:00:00Z
- 2000-01-01T00:00:00Z
+ 1999-12-31T23:00:00
+ 1999-12-31T23:00:00
+ 1999-12-31T23:00:00
+ 2000-01-01T00:00:00
+ 2000-01-01T00:00:00
+ 2000-01-01T00:00:00
at test_files/timestamps.slt:1471Which looks to me like the timestamp logic got lost somehow 🤔
maruschin
force-pushed
the
add_union_optimization
branch
2 times, most recently
from
c4a11b7 to
cdad0be
Compare
maruschin
force-pushed
the
add_union_optimization
branch
from
dfd4dab to
4e116ea
Compare
maruschin
force-pushed
the
add_union_optimization
branch
8 times, most recently
from
d8024ab to
e874a08
Compare
maruschin
force-pushed
the
add_union_optimization
branch
2 times, most recently
from
6258d57 to
e59e54c
Compare
maruschin
force-pushed
the
add_union_optimization
branch
from
e59e54c to
65f09fa
Compare
jackwener
reviewed
Oct 8, 2023
jackwener
reviewed
Oct 8, 2023
added 6 commits
October 9, 2023 16:45
maruschin
force-pushed
the
add_union_optimization
branch
from
65f09fa to
bd6c4d4
Compare
crepererum
reviewed
Oct 10, 2023
| plan: &LogicalPlan, | ||
| _config: &dyn OptimizerConfig, | ||
| ) -> Result<Option<LogicalPlan>> { | ||
| // TODO: Add optimization for nested distinct unions. |
Contributor
There was a problem hiding this comment.
at one point (maybe as a TODO) we probably should also remove unions that have only a single child since they are basically a no-op/pass-through. Not sure if this should be an separate optimizer rule or if this should be done in this rule.
Contributor
There was a problem hiding this comment.
I believe the EliminateOneUnion rule, also added in this PR, handles the one union case
alamb
approved these changes
Oct 10, 2023
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you very much @maruschin -- this is a really nice piece of work. It is well tested and the code is very clean.
Thank you @crepererum and @jackwener for the reviews
| plan: &LogicalPlan, | ||
| _config: &dyn OptimizerConfig, | ||
| ) -> Result<Option<LogicalPlan>> { | ||
| // TODO: Add optimization for nested distinct unions. |
Contributor
There was a problem hiding this comment.
I believe the EliminateOneUnion rule, also added in this PR, handles the one union case
This was referenced Oct 10, 2023
Contributor
|
I took the liberty of merging this branch up from main and fixing clippy as I had it all checked out already I also wrote some end to end tests in #7787 |
Contributor
|
Thanks again @maruschin |
devinjdangelo
pushed a commit
to devinjdangelo/arrow-datafusion
that referenced
this pull request
Oct 11, 2023
…apache#7695) * Add naive implementation of eliminate_nested_union * Remove union optimization from LogicalPlanBuilder::union * Fix propagate_union_children_different_schema test * Add implementation of eliminate_one_union * Simplified eliminate_nested_union test * Fix * clippy --------- Co-authored-by: Evgeny Maruschenko <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #7481.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?