Skip to content

Conversation

simonvandel
Copy link
Contributor

Which issue does this PR close?

Rationale for this change

Reduce planning work for unnest expressions if there are none.

What changes are included in this PR?

Add 2 fast paths for when there are no unnest expressions.

Are these changes tested?

Yes, existing tests

Are there any user-facing changes?

Yes, faster planning (Mostly -2%, but also -90% in extreme cases)

Results Running benches/sql_planner.rs (/Users/svs/code/datafusion/target/profiling/deps/sql_planner-85a953ee0953131c) Gnuplot not found, using plotters backend logical_select_one_from_700 time: [188.10 µs 188.52 µs 188.99 µs] change: [-1.6127% -1.0457% -0.5581%] (p = 0.00 < 0.05) Change within noise threshold. Found 6 outliers among 100 measurements (6.00%) 6 (6.00%) high mild

physical_select_one_from_700
time: [516.76 µs 518.24 µs 519.83 µs]
change: [-1.4567% -1.0211% -0.5687%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
1 (1.00%) low mild
2 (2.00%) high mild
2 (2.00%) high severe

logical_select_all_from_1000
time: [7.0757 ms 7.1052 ms 7.1425 ms]
change: [-90.060% -90.006% -89.929%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) high mild
3 (3.00%) high severe

physical_select_all_from_1000
time: [15.934 ms 15.991 ms 16.052 ms]
change: [-80.508% -80.397% -80.288%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe

logical_trivial_join_low_numbered_columns
time: [163.40 µs 163.74 µs 164.11 µs]
change: [-1.6147% -1.3758% -1.1303%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild

logical_trivial_join_high_numbered_columns
time: [177.59 µs 178.40 µs 179.17 µs]
change: [-1.2130% -0.8972% -0.5485%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

logical_aggregate_with_join
time: [295.51 µs 295.96 µs 296.43 µs]
change: [-12.291% -12.046% -11.818%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

physical_select_aggregates_from_200
time: [9.9497 ms 9.9661 ms 9.9840 ms]
change: [-15.960% -15.736% -15.514%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe

physical_join_consider_sort
time: [621.32 µs 622.51 µs 623.85 µs]
change: [-1.1461% -0.9282% -0.7129%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild

physical_theta_join_consider_sort
time: [778.43 µs 779.82 µs 781.40 µs]
change: [-0.5286% -0.2715% -0.0138%] (p = 0.03 < 0.05)
Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
5 (5.00%) high mild
1 (1.00%) high severe

physical_many_self_joins
time: [5.8886 ms 5.9040 ms 5.9212 ms]
change: [-1.8716% -1.5132% -1.1439%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe

physical_unnest_to_join time: [573.99 µs 575.72 µs 577.77 µs]
change: [+0.3713% +0.8352% +1.3040%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
1 (1.00%) low mild
5 (5.00%) high mild
2 (2.00%) high severe

physical_intersection time: [354.33 µs 354.98 µs 355.75 µs]
change: [-2.1731% -1.7583% -1.3807%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) low mild
3 (3.00%) high mild

physical_join_distinct time: [157.82 µs 158.15 µs 158.52 µs]
change: [-1.7485% -1.5026% -1.2490%] (p = 0.00 < 0.05)
Performance has improved.

physical_sorted_union_orderby
time: [21.052 ms 21.087 ms 21.127 ms]
change: [-2.0502% -1.7808% -1.5199%] (p = 0.00 < 0.05)
Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
5 (5.00%) high mild
8 (8.00%) high severe

physical_plan_tpch_q1 time: [824.70 µs 826.59 µs 829.00 µs]
change: [-3.9654% -3.7157% -3.4670%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
3 (3.00%) high mild
4 (4.00%) high severe

physical_plan_tpch_q2 time: [2.4368 ms 2.4521 ms 2.4695 ms]
change: [-0.4954% +0.1385% +0.8269%] (p = 0.70 > 0.05)
No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
5 (5.00%) high mild
5 (5.00%) high severe

Benchmarking physical_plan_tpch_q3: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, enable flat sampling, or reduce sample count to 60.
physical_plan_tpch_q3 time: [1.0451 ms 1.0465 ms 1.0482 ms]
change: [-2.6864% -2.3598% -2.0654%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe

physical_plan_tpch_q4 time: [576.54 µs 577.46 µs 578.40 µs]
change: [-6.9556% -6.7372% -6.5359%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) low mild
2 (2.00%) high mild

Benchmarking physical_plan_tpch_q5: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.6s, enable flat sampling, or reduce sample count to 60.
physical_plan_tpch_q5 time: [1.3066 ms 1.3096 ms 1.3131 ms]
change: [-1.4216% -1.1349% -0.8458%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) high mild
3 (3.00%) high severe

physical_plan_tpch_q6 time: [314.39 µs 314.98 µs 315.78 µs]
change: [-2.4877% -2.2652% -2.0461%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
2 (2.00%) low severe
3 (3.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_tpch_q7: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 9.0s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q7 time: [1.7688 ms 1.7721 ms 1.7757 ms]
change: [-1.7050% -1.2163% -0.7643%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
6 (6.00%) high mild
4 (4.00%) high severe

physical_plan_tpch_q8 time: [2.2029 ms 2.2083 ms 2.2150 ms]
change: [-1.1974% -0.8351% -0.4715%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_tpch_q9: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.4s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q9 time: [1.6734 ms 1.6759 ms 1.6787 ms]
change: [-2.8591% -2.2345% -1.6973%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_tpch_q10: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.1s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q10 time: [1.6021 ms 1.6047 ms 1.6074 ms]
change: [-1.9149% -1.7249% -1.5277%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
1 (1.00%) low mild
3 (3.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_tpch_q11: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.0s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q11 time: [1.3828 ms 1.3851 ms 1.3875 ms]
change: [-1.4988% -1.1863% -0.8931%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
9 (9.00%) high mild
1 (1.00%) high severe

physical_plan_tpch_q12 time: [716.76 µs 717.82 µs 719.01 µs]
change: [-1.9399% -1.7002% -1.4551%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe

physical_plan_tpch_q13 time: [547.99 µs 548.88 µs 549.88 µs]
change: [-2.4430% -2.1554% -1.8732%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
6 (6.00%) high mild
3 (3.00%) high severe

physical_plan_tpch_q14 time: [738.81 µs 740.21 µs 741.73 µs]
change: [-4.1360% -3.0409% -2.1796%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
3 (3.00%) high mild
4 (4.00%) high severe

Benchmarking physical_plan_tpch_q16: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.2s, enable flat sampling, or reduce sample count to 60.
physical_plan_tpch_q16 time: [1.0266 ms 1.0286 ms 1.0309 ms]
change: [-3.4245% -2.9517% -2.5196%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
3 (3.00%) high mild
4 (4.00%) high severe

physical_plan_tpch_q17 time: [947.77 µs 950.12 µs 952.70 µs]
change: [-2.5652% -2.1409% -1.7306%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_tpch_q18: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.5s, enable flat sampling, or reduce sample count to 60.
physical_plan_tpch_q18 time: [1.0904 ms 1.0931 ms 1.0961 ms]
change: [-3.1180% -2.7558% -2.3739%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_tpch_q19: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.9s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q19 time: [1.5747 ms 1.5822 ms 1.5908 ms]
change: [-1.0349% -0.5964% -0.1350%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
2 (2.00%) high mild
7 (7.00%) high severe

Benchmarking physical_plan_tpch_q20: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.7s, enable flat sampling, or reduce sample count to 60.
physical_plan_tpch_q20 time: [1.3168 ms 1.3200 ms 1.3238 ms]
change: [-2.3486% -1.6985% -1.1326%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_tpch_q21: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 9.1s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q21 time: [1.7821 ms 1.7857 ms 1.7899 ms]
change: [-3.7630% -3.5765% -3.3716%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe

Benchmarking physical_plan_tpch_q22: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.8s, enable flat sampling, or reduce sample count to 60.
physical_plan_tpch_q22 time: [1.1460 ms 1.1482 ms 1.1505 ms]
change: [-2.5916% -2.0457% -1.6040%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe

physical_plan_tpch_all time: [27.972 ms 28.050 ms 28.143 ms]
change: [-1.8048% -1.4756% -1.0540%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
6 (6.00%) high mild
4 (4.00%) high severe

Benchmarking physical_plan_tpcds_all: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 49.3s, or reduce sample count to 10.
physical_plan_tpcds_all time: [491.13 ms 492.10 ms 493.30 ms]
change: [-2.3624% -2.0511% -1.7380%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) high mild
3 (3.00%) high severe

physical_plan_clickbench_q1
time: [627.41 µs 630.81 µs 634.81 µs]
change: [-1.2562% -0.6857% +0.0609%] (p = 0.03 < 0.05)
Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe

physical_plan_clickbench_q2
time: [713.86 µs 718.57 µs 725.86 µs]
change: [+0.4199% +1.6426% +3.3108%] (p = 0.03 < 0.05)
Change within noise threshold.
Found 14 outliers among 100 measurements (14.00%)
7 (7.00%) high mild
7 (7.00%) high severe

physical_plan_clickbench_q3
time: [707.24 µs 709.42 µs 711.94 µs]
change: [-4.3336% -3.4336% -2.6422%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
7 (7.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q4
time: [610.12 µs 613.12 µs 616.72 µs]
change: [-3.1410% -2.5594% -2.0454%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q5
time: [683.17 µs 685.06 µs 687.13 µs]
change: [-2.4521% -1.9761% -1.5303%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q6
time: [685.43 µs 688.74 µs 692.74 µs]
change: [-2.4567% -1.9979% -1.5350%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) high mild
3 (3.00%) high severe

physical_plan_clickbench_q7
time: [631.56 µs 634.26 µs 637.48 µs]
change: [-2.9189% -2.3008% -1.6700%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
6 (6.00%) high mild
3 (3.00%) high severe

physical_plan_clickbench_q8
time: [890.67 µs 893.37 µs 896.38 µs]
change: [-2.5735% -2.0381% -1.4872%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q9
time: [861.47 µs 864.52 µs 868.09 µs]
change: [-2.5807% -2.0021% -1.4986%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
1 (1.00%) low mild
6 (6.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q10
time: [917.07 µs 920.51 µs 924.36 µs]
change: [-1.8845% -1.3715% -0.7353%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q11
time: [977.37 µs 989.10 µs 1.0033 ms]
change: [-2.0585% -1.3076% -0.4211%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
1 (1.00%) high mild
8 (8.00%) high severe

Benchmarking physical_plan_clickbench_q12: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.2s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q12
time: [1.0264 ms 1.0304 ms 1.0352 ms]
change: [-3.7200% -2.5328% -1.4520%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe

physical_plan_clickbench_q13
time: [877.99 µs 880.62 µs 883.56 µs]
change: [-2.8364% -2.3327% -1.8760%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
4 (4.00%) high mild
4 (4.00%) high severe

physical_plan_clickbench_q14
time: [968.57 µs 971.96 µs 975.74 µs]
change: [-2.2840% -1.8788% -1.4818%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
5 (5.00%) high mild

physical_plan_clickbench_q15
time: [921.84 µs 927.21 µs 933.25 µs]
change: [-1.9960% -1.4590% -0.8965%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 13 outliers among 100 measurements (13.00%)
8 (8.00%) high mild
5 (5.00%) high severe

physical_plan_clickbench_q16
time: [877.17 µs 886.26 µs 897.94 µs]
change: [-1.6329% -0.5990% +0.5870%] (p = 0.31 > 0.05)
No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) high mild
6 (6.00%) high severe

physical_plan_clickbench_q17
time: [921.65 µs 924.69 µs 928.68 µs]
change: [-3.7469% -2.9819% -2.3437%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q18
time: [720.86 µs 723.06 µs 725.45 µs]
change: [-2.7554% -2.3624% -1.8984%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
4 (4.00%) high mild
5 (5.00%) high severe

Benchmarking physical_plan_clickbench_q19: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.5s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q19
time: [1.0810 ms 1.0849 ms 1.0893 ms]
change: [-2.3788% -1.9819% -1.5873%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) high mild
2 (2.00%) high severe

physical_plan_clickbench_q20
time: [621.49 µs 624.93 µs 628.88 µs]
change: [-0.1622% +0.5795% +1.3344%] (p = 0.11 > 0.05)
No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
5 (5.00%) high mild
5 (5.00%) high severe

physical_plan_clickbench_q21
time: [727.66 µs 731.97 µs 737.56 µs]
change: [+0.7158% +2.3855% +4.7732%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 12 outliers among 100 measurements (12.00%)
4 (4.00%) high mild
8 (8.00%) high severe

physical_plan_clickbench_q22
time: [967.99 µs 971.21 µs 975.09 µs]
change: [-2.1631% -1.6597% -1.1599%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_clickbench_q23: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.6s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q23
time: [1.0874 ms 1.0905 ms 1.0942 ms]
change: [-6.2478% -5.0640% -4.0157%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe

Benchmarking physical_plan_clickbench_q24: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.5s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q24
time: [1.2687 ms 1.2729 ms 1.2775 ms]
change: [-41.602% -41.240% -40.927%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
1 (1.00%) low mild
6 (6.00%) high mild
3 (3.00%) high severe

physical_plan_clickbench_q25
time: [774.85 µs 779.41 µs 785.00 µs]
change: [-0.4966% +0.0607% +0.6975%] (p = 0.84 > 0.05)
No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe

physical_plan_clickbench_q26
time: [693.36 µs 696.20 µs 699.99 µs]
change: [+2.0366% +3.4191% +5.3205%] (p = 0.00 < 0.05)
Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
4 (4.00%) high mild
5 (5.00%) high severe

physical_plan_clickbench_q27
time: [778.58 µs 781.32 µs 784.40 µs]
change: [-0.3959% +0.1047% +0.5853%] (p = 0.69 > 0.05)
No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
5 (5.00%) high mild

Benchmarking physical_plan_clickbench_q28: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.4s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q28
time: [1.0636 ms 1.0680 ms 1.0731 ms]
change: [-4.0254% -2.9192% -1.9493%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_clickbench_q29: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.8s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q29
time: [1.3263 ms 1.3318 ms 1.3385 ms]
change: [-2.0971% -1.6357% -1.1132%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
2 (2.00%) high mild
8 (8.00%) high severe

physical_plan_clickbench_q30
time: [5.9731 ms 5.9937 ms 6.0169 ms]
change: [-8.3918% -7.9986% -7.5712%] (p = 0.00 < 0.05)
Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
5 (5.00%) high mild
6 (6.00%) high severe

Benchmarking physical_plan_clickbench_q31: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.5s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q31
time: [1.0870 ms 1.0914 ms 1.0962 ms]
change: [-2.8621% -2.3090% -1.6322%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe

Benchmarking physical_plan_clickbench_q32: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.6s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q32
time: [1.0844 ms 1.0887 ms 1.0935 ms]
change: [-2.1629% -1.5653% -0.9550%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 15 outliers among 100 measurements (15.00%)
6 (6.00%) high mild
9 (9.00%) high severe

physical_plan_clickbench_q33
time: [913.88 µs 917.25 µs 921.09 µs]
change: [-2.6126% -2.0549% -1.4655%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
9 (9.00%) high mild
1 (1.00%) high severe

physical_plan_clickbench_q34
time: [781.42 µs 783.79 µs 786.42 µs]
change: [-7.4331% -5.4117% -3.7701%] (p = 0.00 < 0.05)
Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
10 (10.00%) high mild
1 (1.00%) high severe

physical_plan_clickbench_q35
time: [817.75 µs 821.23 µs 825.90 µs]
change: [-2.4789% -1.9706% -1.4428%] (p = 0.00 < 0.05)
Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
1 (1.00%) low mild
9 (9.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_clickbench_q36: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.7s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q36
time: [1.1214 ms 1.1282 ms 1.1373 ms]
change: [-3.4903% -2.5187% -1.6809%] (p = 0.00 < 0.05)
Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
8 (8.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_clickbench_q37: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.0s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q37
time: [1.1411 ms 1.1459 ms 1.1522 ms]
change: [-1.8493% -1.3832% -0.9048%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_clickbench_q38: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.8s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q38
time: [1.1389 ms 1.1448 ms 1.1519 ms]
change: [-2.0118% -1.4113% -0.7730%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
7 (7.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_clickbench_q39: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q39
time: [1.0437 ms 1.0498 ms 1.0568 ms]
change: [-3.3478% -2.3387% -1.4380%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_clickbench_q40: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.6s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q40
time: [1.2844 ms 1.2894 ms 1.2952 ms]
change: [-2.7809% -2.1830% -1.4561%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe

Benchmarking physical_plan_clickbench_q41: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.7s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q41
time: [1.1217 ms 1.1281 ms 1.1365 ms]
change: [-1.0602% -0.1071% +1.3125%] (p = 0.89 > 0.05)
No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
1 (1.00%) low mild
3 (3.00%) high mild
9 (9.00%) high severe

Benchmarking physical_plan_clickbench_q42: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.6s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q42
time: [1.1121 ms 1.1297 ms 1.1489 ms]
change: [-0.4793% +0.3228% +1.2243%] (p = 0.46 > 0.05)
No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) high mild
6 (6.00%) high severe

Benchmarking physical_plan_clickbench_q43: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.2s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q43
time: [1.2304 ms 1.2364 ms 1.2434 ms]
change: [-1.9524% -1.4590% -0.9688%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 12 outliers among 100 measurements (12.00%)
1 (1.00%) low mild
7 (7.00%) high mild
4 (4.00%) high severe

physical_plan_clickbench_q44
time: [669.48 µs 671.77 µs 674.36 µs]
change: [-2.4951% -1.7410% -0.9421%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 15 outliers among 100 measurements (15.00%)
7 (7.00%) high mild
8 (8.00%) high severe

physical_plan_clickbench_q45
time: [672.26 µs 674.42 µs 677.02 µs]
change: [-2.2589% -1.6587% -1.0724%] (p = 0.00 < 0.05)
Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
1 (1.00%) low mild
3 (3.00%) high mild
8 (8.00%) high severe

physical_plan_clickbench_q46
time: [848.40 µs 852.98 µs 858.32 µs]
change: [-1.7496% -1.2928% -0.7739%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
2 (2.00%) high mild
8 (8.00%) high severe

Benchmarking physical_plan_clickbench_q47: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q47
time: [1.0589 ms 1.0778 ms 1.1033 ms]
change: [-2.7585% -1.8340% -0.7198%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
1 (1.00%) low mild
2 (2.00%) high mild
4 (4.00%) high severe

Benchmarking physical_plan_clickbench_q48: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.7s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q48
time: [1.3057 ms 1.3095 ms 1.3135 ms]
change: [-3.1701% -2.5485% -1.9748%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe

Benchmarking physical_plan_clickbench_q49: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.2s, enable flat sampling, or reduce sample count to 50.
physical_plan_clickbench_q49
time: [1.4090 ms 1.4157 ms 1.4256 ms]
change: [-2.8521% -2.0959% -1.3969%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
1 (1.00%) low mild
5 (5.00%) high mild
4 (4.00%) high severe

Benchmarking physical_plan_clickbench_q50: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.7s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q50
time: [1.3219 ms 1.3275 ms 1.3345 ms]
change: [-0.1782% +0.4591% +1.1364%] (p = 0.17 > 0.05)
No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
6 (6.00%) high mild
3 (3.00%) high severe

physical_plan_clickbench_q51
time: [873.26 µs 877.42 µs 882.61 µs]
change: [-1.8386% -1.3198% -0.7899%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
6 (6.00%) high mild
4 (4.00%) high severe

Benchmarking physical_plan_clickbench_q52: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.1s, enable flat sampling, or reduce sample count to 60.
physical_plan_clickbench_q52
time: [1.2162 ms 1.2349 ms 1.2561 ms]
change: [-2.1949% -1.2771% -0.2884%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) high mild
6 (6.00%) high severe

Benchmarking physical_plan_clickbench_all: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.7s, or reduce sample count to 80.
physical_plan_clickbench_all
time: [56.570 ms 56.690 ms 56.817 ms]
change: [-4.5415% -4.0476% -3.5951%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe

with_param_values_many_columns
time: [76.482 µs 76.648 µs 76.816 µs]
change: [-1.5282% -1.2036% -0.9131%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

@github-actions github-actions bot added the sql SQL Planner label Jun 12, 2025
Copy link
Contributor

@jonathanc-n jonathanc-n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, thanks @simonvandel! I'll see if theres a way to refactor that function in the comment and create a follow up pr if possible

.build()
}

fn try_process_aggregate_unnest(&self, input: LogicalPlan) -> Result<LogicalPlan> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can possibly refactor this entire function to use a transform traversal function.

@alamb
Copy link
Contributor

alamb commented Jun 13, 2025

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing opt-process-unnest (2c98cca) to 3764fe7 diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=opt-process-unnest
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Jun 13, 2025

🤖: Benchmark completed

Details

group                                         main                                   opt-process-unnest
-----                                         ----                                   ------------------
logical_aggregate_with_join                   1.16    731.0±4.33µs        ? ?/sec    1.00    632.4±2.72µs        ? ?/sec
logical_select_all_from_1000                  10.80   120.4±0.22ms        ? ?/sec    1.00     11.1±0.06ms        ? ?/sec
logical_select_one_from_700                   1.00    413.6±1.67µs        ? ?/sec    1.01    415.8±2.87µs        ? ?/sec
logical_trivial_join_high_numbered_columns    1.01    378.1±2.15µs        ? ?/sec    1.00    373.1±1.37µs        ? ?/sec
logical_trivial_join_low_numbered_columns     1.01    363.6±1.46µs        ? ?/sec    1.00    359.6±1.17µs        ? ?/sec
physical_intersection                         1.01    844.4±7.77µs        ? ?/sec    1.00    839.8±5.55µs        ? ?/sec
physical_join_consider_sort                   1.02  1406.7±10.78µs        ? ?/sec    1.00   1379.0±6.33µs        ? ?/sec
physical_join_distinct                        1.01    354.9±1.41µs        ? ?/sec    1.00    351.0±1.29µs        ? ?/sec
physical_many_self_joins                      1.01     10.4±0.07ms        ? ?/sec    1.00     10.3±0.04ms        ? ?/sec
physical_plan_clickbench_all                  1.04    148.9±2.29ms        ? ?/sec    1.00    143.7±1.53ms        ? ?/sec
physical_plan_clickbench_q1                   1.04  1764.4±23.74µs        ? ?/sec    1.00  1696.8±17.25µs        ? ?/sec
physical_plan_clickbench_q10                  1.05      2.5±0.04ms        ? ?/sec    1.00      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q11                  1.04      2.7±0.05ms        ? ?/sec    1.00      2.6±0.04ms        ? ?/sec
physical_plan_clickbench_q12                  1.04      2.8±0.05ms        ? ?/sec    1.00      2.7±0.05ms        ? ?/sec
physical_plan_clickbench_q13                  1.04      2.5±0.04ms        ? ?/sec    1.00      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q14                  1.04      2.7±0.04ms        ? ?/sec    1.00      2.6±0.03ms        ? ?/sec
physical_plan_clickbench_q15                  1.04      2.5±0.04ms        ? ?/sec    1.00      2.4±0.03ms        ? ?/sec
physical_plan_clickbench_q16                  1.02      2.4±0.03ms        ? ?/sec    1.00      2.3±0.03ms        ? ?/sec
physical_plan_clickbench_q17                  1.03      2.5±0.04ms        ? ?/sec    1.00      2.5±0.03ms        ? ?/sec
physical_plan_clickbench_q18                  1.03      2.0±0.03ms        ? ?/sec    1.00  1960.5±27.55µs        ? ?/sec
physical_plan_clickbench_q19                  1.02      3.0±0.10ms        ? ?/sec    1.00      2.9±0.05ms        ? ?/sec
physical_plan_clickbench_q2                   1.04      2.0±0.03ms        ? ?/sec    1.00  1934.7±31.78µs        ? ?/sec
physical_plan_clickbench_q20                  1.02  1700.5±24.23µs        ? ?/sec    1.00  1670.9±18.27µs        ? ?/sec
physical_plan_clickbench_q21                  1.03      2.0±0.04ms        ? ?/sec    1.00  1955.4±26.69µs        ? ?/sec
physical_plan_clickbench_q22                  1.02      2.6±0.04ms        ? ?/sec    1.00      2.6±0.03ms        ? ?/sec
physical_plan_clickbench_q23                  1.03      3.0±0.04ms        ? ?/sec    1.00      2.9±0.04ms        ? ?/sec
physical_plan_clickbench_q24                  1.44      4.6±0.06ms        ? ?/sec    1.00      3.2±0.05ms        ? ?/sec
physical_plan_clickbench_q25                  1.00      2.1±0.05ms        ? ?/sec    1.00      2.1±0.03ms        ? ?/sec
physical_plan_clickbench_q26                  1.02  1880.5±29.70µs        ? ?/sec    1.00  1851.2±23.32µs        ? ?/sec
physical_plan_clickbench_q27                  1.00      2.1±0.02ms        ? ?/sec    1.01      2.1±0.04ms        ? ?/sec
physical_plan_clickbench_q28                  1.01      2.9±0.04ms        ? ?/sec    1.00      2.9±0.04ms        ? ?/sec
physical_plan_clickbench_q29                  1.01      3.6±0.04ms        ? ?/sec    1.00      3.6±0.05ms        ? ?/sec
physical_plan_clickbench_q3                   1.05      2.0±0.05ms        ? ?/sec    1.00  1918.4±24.39µs        ? ?/sec
physical_plan_clickbench_q30                  1.07     13.4±0.15ms        ? ?/sec    1.00     12.5±0.11ms        ? ?/sec
physical_plan_clickbench_q31                  1.01      2.9±0.04ms        ? ?/sec    1.00      2.9±0.04ms        ? ?/sec
physical_plan_clickbench_q32                  1.02      2.9±0.04ms        ? ?/sec    1.00      2.9±0.04ms        ? ?/sec
physical_plan_clickbench_q33                  1.02      2.5±0.03ms        ? ?/sec    1.00      2.4±0.04ms        ? ?/sec
physical_plan_clickbench_q34                  1.02      2.2±0.03ms        ? ?/sec    1.00      2.1±0.02ms        ? ?/sec
physical_plan_clickbench_q35                  1.03      2.3±0.04ms        ? ?/sec    1.00      2.2±0.04ms        ? ?/sec
physical_plan_clickbench_q36                  1.03      3.0±0.03ms        ? ?/sec    1.00      2.9±0.03ms        ? ?/sec
physical_plan_clickbench_q37                  1.00      3.0±0.04ms        ? ?/sec    1.01      3.0±0.07ms        ? ?/sec
physical_plan_clickbench_q38                  1.00      3.0±0.03ms        ? ?/sec    1.01      3.0±0.06ms        ? ?/sec
physical_plan_clickbench_q39                  1.00      2.8±0.03ms        ? ?/sec    1.03      2.9±0.05ms        ? ?/sec
physical_plan_clickbench_q4                   1.05  1724.6±29.47µs        ? ?/sec    1.00  1642.0±22.54µs        ? ?/sec
physical_plan_clickbench_q40                  1.00      3.5±0.05ms        ? ?/sec    1.00      3.5±0.07ms        ? ?/sec
physical_plan_clickbench_q41                  1.00      3.0±0.03ms        ? ?/sec    1.01      3.0±0.06ms        ? ?/sec
physical_plan_clickbench_q42                  1.00      2.9±0.03ms        ? ?/sec    1.02      2.9±0.05ms        ? ?/sec
physical_plan_clickbench_q43                  1.00      3.3±0.08ms        ? ?/sec    1.00      3.3±0.06ms        ? ?/sec
physical_plan_clickbench_q44                  1.01  1837.1±18.40µs        ? ?/sec    1.00  1825.4±28.03µs        ? ?/sec
physical_plan_clickbench_q45                  1.00  1853.9±25.35µs        ? ?/sec    1.00  1852.9±40.85µs        ? ?/sec
physical_plan_clickbench_q46                  1.01      2.3±0.04ms        ? ?/sec    1.00      2.3±0.06ms        ? ?/sec
physical_plan_clickbench_q47                  1.01      2.9±0.04ms        ? ?/sec    1.00      2.9±0.06ms        ? ?/sec
physical_plan_clickbench_q48                  1.00      3.5±0.08ms        ? ?/sec    1.00      3.5±0.06ms        ? ?/sec
physical_plan_clickbench_q49                  1.03      3.8±0.06ms        ? ?/sec    1.00      3.7±0.06ms        ? ?/sec
physical_plan_clickbench_q5                   1.04  1918.5±29.29µs        ? ?/sec    1.00  1852.5±22.67µs        ? ?/sec
physical_plan_clickbench_q50                  1.03      3.5±0.07ms        ? ?/sec    1.00      3.4±0.05ms        ? ?/sec
physical_plan_clickbench_q51                  1.03      2.4±0.03ms        ? ?/sec    1.00      2.3±0.03ms        ? ?/sec
physical_plan_clickbench_q52                  1.01      3.3±0.06ms        ? ?/sec    1.00      3.2±0.09ms        ? ?/sec
physical_plan_clickbench_q6                   1.05  1942.4±29.30µs        ? ?/sec    1.00  1848.4±28.29µs        ? ?/sec
physical_plan_clickbench_q7                   1.03  1746.2±25.77µs        ? ?/sec    1.00  1693.1±19.44µs        ? ?/sec
physical_plan_clickbench_q8                   1.06      2.6±0.05ms        ? ?/sec    1.00      2.4±0.04ms        ? ?/sec
physical_plan_clickbench_q9                   1.06      2.4±0.04ms        ? ?/sec    1.00      2.3±0.03ms        ? ?/sec
physical_plan_tpcds_all                       1.02   1099.4±5.03ms        ? ?/sec    1.00   1075.8±6.80ms        ? ?/sec
physical_plan_tpch_all                        1.03     65.8±0.57ms        ? ?/sec    1.00     64.1±0.52ms        ? ?/sec
physical_plan_tpch_q1                         1.04      2.2±0.01ms        ? ?/sec    1.00      2.1±0.01ms        ? ?/sec
physical_plan_tpch_q10                        1.01      3.9±0.02ms        ? ?/sec    1.00      3.9±0.04ms        ? ?/sec
physical_plan_tpch_q11                        1.02      3.4±0.02ms        ? ?/sec    1.00      3.3±0.02ms        ? ?/sec
physical_plan_tpch_q12                        1.01   1883.8±6.92µs        ? ?/sec    1.00  1856.9±10.15µs        ? ?/sec
physical_plan_tpch_q13                        1.02   1491.0±6.51µs        ? ?/sec    1.00   1467.5±9.71µs        ? ?/sec
physical_plan_tpch_q14                        1.01   1998.2±9.60µs        ? ?/sec    1.00   1970.9±8.88µs        ? ?/sec
physical_plan_tpch_q16                        1.02      2.6±0.02ms        ? ?/sec    1.00      2.5±0.01ms        ? ?/sec
physical_plan_tpch_q17                        1.01      2.5±0.02ms        ? ?/sec    1.00      2.4±0.01ms        ? ?/sec
physical_plan_tpch_q18                        1.02      2.8±0.02ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_tpch_q19                        1.01      3.5±0.02ms        ? ?/sec    1.00      3.5±0.02ms        ? ?/sec
physical_plan_tpch_q2                         1.01      5.6±0.05ms        ? ?/sec    1.00      5.6±0.04ms        ? ?/sec
physical_plan_tpch_q20                        1.01      3.2±0.03ms        ? ?/sec    1.00      3.2±0.01ms        ? ?/sec
physical_plan_tpch_q21                        1.03      4.3±0.03ms        ? ?/sec    1.00      4.2±0.03ms        ? ?/sec
physical_plan_tpch_q22                        1.03      2.8±0.02ms        ? ?/sec    1.00      2.8±0.01ms        ? ?/sec
physical_plan_tpch_q3                         1.02      2.6±0.02ms        ? ?/sec    1.00      2.6±0.01ms        ? ?/sec
physical_plan_tpch_q4                         1.06   1626.7±8.90µs        ? ?/sec    1.00  1540.7±10.32µs        ? ?/sec
physical_plan_tpch_q5                         1.02      3.2±0.02ms        ? ?/sec    1.00      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q6                         1.01    895.8±3.77µs        ? ?/sec    1.00    886.7±9.07µs        ? ?/sec
physical_plan_tpch_q7                         1.02      4.4±0.04ms        ? ?/sec    1.00      4.3±0.02ms        ? ?/sec
physical_plan_tpch_q8                         1.02      5.4±0.07ms        ? ?/sec    1.00      5.3±0.03ms        ? ?/sec
physical_plan_tpch_q9                         1.01      4.2±0.04ms        ? ?/sec    1.00      4.2±0.03ms        ? ?/sec
physical_select_aggregates_from_200           1.16     21.1±0.08ms        ? ?/sec    1.00     18.1±0.10ms        ? ?/sec
physical_select_all_from_1000                 5.40    134.7±0.34ms        ? ?/sec    1.00     24.9±0.07ms        ? ?/sec
physical_select_one_from_700                  1.01   1080.3±6.53µs        ? ?/sec    1.00   1071.5±7.01µs        ? ?/sec
physical_sorted_union_orderby                 1.02     44.8±0.29ms        ? ?/sec    1.00     43.9±0.27ms        ? ?/sec
physical_theta_join_consider_sort             1.01   1770.9±8.97µs        ? ?/sec    1.00   1752.2±5.90µs        ? ?/sec
physical_unnest_to_join                       1.01   1317.7±8.64µs        ? ?/sec    1.00   1307.0±6.93µs        ? ?/sec
with_param_values_many_columns                1.00    143.1±1.17µs        ? ?/sec    1.00    143.2±1.14µs        ? ?/sec

@alamb
Copy link
Contributor

alamb commented Jun 13, 2025

logical_select_all_from_1000 10.80 120.4±0.22ms ? ?/sec 1.00 11.1±0.06ms ? ?/sec

🚀

The other planning benchmarks look like pretty much across the board win

THank you @simonvandel and @jonathanc-n

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @simonvandel and @jonathanc-n

fn try_process_aggregate_unnest(&self, input: LogicalPlan) -> Result<LogicalPlan> {
match input {
LogicalPlan::Aggregate(ref agg)
if {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the reason to add { here? It looked kind of strange to me -- i would have expected something more like

 LogicalPlan::Aggregate(ref agg) if !&agg.group_expr.iter().any(has_unnest_expr_recursively) => 

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know of any reason :)
Reformatted in 89c210b

@alamb alamb added the performance Make DataFusion faster label Jun 13, 2025
Copy link
Contributor

@jonathanc-n jonathanc-n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good thanks @simonvandel!

@comphead comphead merged commit 42fe050 into apache:main Jun 15, 2025
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Make DataFusion faster sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve sql planing performance (optimize try_process_unnest)

4 participants