Skip to content

Commit 27ed4f9

Browse files
committed
Add a comment explaining what each large-filter benchmark does
1 parent 67b7a86 commit 27ed4f9

File tree

1 file changed

+7
-0
lines changed

1 file changed

+7
-0
lines changed

sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,6 +394,13 @@ object FilterPushdownBenchmark extends BenchmarkBase with SQLHelper {
394394
}
395395

396396
runBenchmark(s"Pushdown benchmark with many filters") {
397+
// This benchmark and the next one are similar in that they both test predicate pushdown
398+
// where the filter itself is very large. There have been cases where the filter conversion
399+
// would take minutes to hours for large filters due to it being implemented with exponential
400+
// complexity in the height of the filter tree.
401+
// The difference between these two benchmarks is that this one benchmarks pushdown with a
402+
// large string filter (`a AND b AND c ...`), whereas the next one benchmarks pushdown with
403+
// a large Column-based filter (`col(a) || (col(b) || (col(c)...))`).
397404
val numRows = 1
398405
val width = 500
399406

0 commit comments

Comments
 (0)