[SPARK-25662][SQL][TEST] Refactor DataSourceReadBenchmark to use main method #22664
Conversation
Change-Id: Icfd0484c8e0fef2ed0b184e09e52db9432e0a250
| * spark-submit --class <this class> <spark sql test jar> | ||
| * To run this benchmark: | ||
| * {{{ | ||
| * 1. without sbt: bin/spark-submit --class <this class> <spark sql test jar> |
There was a problem hiding this comment.
Hi, @peter-toth . Could you run this command actually?
There was a problem hiding this comment.
bin/spark-submit --class org.apache.spark.sql.execution.benchmark.DataSourceReadBenchmark --jars core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar,sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar sql/core/target/spark-sql_2.11-3.0.0-SNAPSHOT-tests.jar
does work for me, but I checked in FilterPushdownBenchmark and it seems we don't mention other required jars.
Shall I modify the command?
There was a problem hiding this comment.
Yes. We noticed that the required jars are introduced during refactoring. So, we start to fix this guide in recent PRs like this.
Change-Id: I49fd66b225fa4cee6ed163a16f55b32506c00e59
| * spark-submit --class <this class> <spark sql test jar> | ||
| * To run this benchmark: | ||
| * {{{ | ||
| * 1. without sbt: bin/spark-submit --class <this class> --jars <spark core test jar>,<spark catalyst test jar> <spark sql test jar> |
There was a problem hiding this comment.
Could you run dev/scalastyle and fix this in your branch?
|
ok to test |
|
Test build #97227 has finished for PR 22664 at commit
|
|
Hi, @peter-toth . |
Update result
|
Test build #97236 has finished for PR 22664 at commit
|
Change-Id: If4fcfc27eb808c08246a8f7779fbe38a437a41a4
…o SPARK-25662 Change-Id: Ie5b0a3fa70b605c1655e3328f2c92ff179805f7d
|
Could you add |
peter-toth
changed the title
|
Test build #97237 has finished for PR 22664 at commit
|
|
Retest this please. |
|
Hi, @dbtsai . |
|
Test build #97241 has finished for PR 22664 at commit
|
|
Retest this please. |
|
Thanks @dongjoon-hyun for ping me. LGTM too. We're working on some parquet reader improvement, and this will be useful. |
|
Test build #97272 has finished for PR 22664 at commit
|
|
Thank you, @dbtsai and @peter-toth . |
|
Thanks for the review @dongjoon-hyun and @dbtsai . |
|
@peter-toth I assigned to you. Thanks for contribution. |
… method ## What changes were proposed in this pull request? 1. Refactor DataSourceReadBenchmark ## How was this patch tested? Manually tested and regenerated results. ``` SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.DataSourceReadBenchmark" ``` Closes apache#22664 from peter-toth/SPARK-25662. Lead-authored-by: Peter Toth <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: DB Tsai <[email protected]>
4 participants
What changes were proposed in this pull request?
How was this patch tested?
Manually tested and regenerated results.