[SPARK-24206][SQL] Improve DataSource read benchmark code

[maropu]

What changes were proposed in this pull request?

This pr added benchmark code DataSourceReadBenchmark for orc, paruqet, csv, and json based on the existing ParquetReadBenchmark and OrcReadBenchmark.

How was this patch tested?

N/A

[maropu]

I'll add benchmark results just after #21070 merged.

[maropu]

Also, I'll make a follow-up pr for pushdown benchmarks;
master...maropu:UpdateParquetBenchmark

[maropu]

@gatorsmile @dongjoon-hyun

[SparkQA]

Test build #90359 has finished for PR 21266 at commit 09b3920.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

@maropu . Since you are merging ParquetReadBenchmark and OrcReadBenchmark benchmarks, let's remove OrcReadBenchmark.

[maropu]

I feel we still need to OrcReadBenchmark to compare native orc with Hive built-in orc?

[dongjoon-hyun]

I see. Never mind. I deleted the previous comment. The scope is only testing native orc here.

[maropu]

ok, thanks!

[SparkQA]

Test build #90392 has finished for PR 21266 at commit 2813706.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #90437 has finished for PR 21266 at commit 8aedbf0.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #90438 has finished for PR 21266 at commit 1d93d99.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #90572 has finished for PR 21266 at commit fc96adb.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

retest this please

[SparkQA]

Test build #90579 has finished for PR 21266 at commit fc96adb.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[kiszk]

I think that this type of comments should not be included by scala file in this directory.
If you want to put this comment, this scala file should be put into sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ where files are not translated to doc.

[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/target/java/org/apache/spark/sql/DataSourceReadBenchmark.java:5: error: unknown tag: this
[error]  *  spark-submit --class <this class> <spark sql test jar>
[error]                          ^
[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/core/target/java/org/apache/spark/sql/DataSourceReadBenchmark.java:5: error: unknown tag: spark
[error]  *  spark-submit --class <this class> <spark sql test jar>
[error]     

[maropu]

ok

[SparkQA]

Test build #90875 has finished for PR 21266 at commit 3b6f541.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #90877 has finished for PR 21266 at commit fad31b7.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #90879 has finished for PR 21266 at commit d8c308f.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

retest this please

[SparkQA]

Test build #90884 has finished for PR 21266 at commit d8c308f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

Let us explicitly set these confs? Here, we are expecting the perf number when ORC_COPY_BATCH_TO_SPARK is set to false. Please also double check the other benchmarks and add the related confs too?

[maropu]

ok

[maropu]

I checked that ORC_COPY_BATCH_TO_SPARK=false in other tests (I didn't find performance differences after explicitly setting false in line 50.
https://github.com/apache/spark/pull/21266/files#diff-ae11b49db05c9e6829cad071b112a742R50

[gatorsmile]

@maropu Great work! Thanks for helping this!

[maropu]

I'll update the benchmark results soon.

[SparkQA]

Test build #91012 has finished for PR 21266 at commit 5eab1a5.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

retest this please

[SparkQA]

Test build #91017 has finished for PR 21266 at commit 5eab1a5.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[kiszk]

retest this please

[SparkQA]

Test build #91036 has finished for PR 21266 at commit 5eab1a5.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[kiszk]

retest this please

[SparkQA]

Test build #91049 has finished for PR 21266 at commit 5eab1a5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

LGTM

Thanks! Merged to master.