File tree Expand file tree Collapse file tree 1 file changed +7
-0
lines changed
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark Expand file tree Collapse file tree 1 file changed +7
-0
lines changed Original file line number Diff line number Diff line change @@ -394,6 +394,13 @@ object FilterPushdownBenchmark extends BenchmarkBase with SQLHelper {
394394 }
395395
396396 runBenchmark(s " Pushdown benchmark with many filters " ) {
397+ // This benchmark and the next one are similar in that they both test predicate pushdown
398+ // where the filter itself is very large. There have been cases where the filter conversion
399+ // would take minutes to hours for large filters due to it being implemented with exponential
400+ // complexity in the height of the filter tree.
401+ // The difference between these two benchmarks is that this one benchmarks pushdown with a
402+ // large string filter (`a AND b AND c ...`), whereas the next one benchmarks pushdown with
403+ // a large Column-based filter (`col(a) || (col(b) || (col(c)...))`).
397404 val numRows = 1
398405 val width = 500
399406
You can’t perform that action at this time.
0 commit comments