Skip to content

[SPARK-31129][SQL][Tests] Fix IntervalBenchmark and DateTimeBenchmark #27885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-31129][SQL][Tests] Fix IntervalBenchmark and DateTimeBenchmark #27885

wants to merge 2 commits into from

Conversation

yaooqinn
Copy link
Member

@yaooqinn yaooqinn commented Mar 12, 2020 •
edited by dongjoon-hyun
Loading

What changes were proposed in this pull request?

This PR aims to recover IntervalBenchmark and DataTimeBenchmark due to banning intervals as output.

Why are the changes needed?

This PR recovers the benchmark suite.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually, re-run the benchmark.

@yaooqinn
Copy link
Member Author

cc @dongjoon-hyun @cloud-fan, thank you very much.

@dongjoon-hyun
Copy link
Member

Hi, @yaooqinn . If this is not the last one, could you proceed as a single PR for all benchmark?

@dongjoon-hyun
Copy link
Member

Also, please create a new JIRA issue. The old one is already committed as ExtractBenchmark. So, we cannot reuse the ID for IntervalBenchmark.

@yaooqinn
Copy link
Member Author

yaooqinn commented Mar 12, 2020 •
edited
Loading

Hi, @yaooqinn . If this is not the last one, could you proceed as a single PR for all benchmark?

I'll re-run all benchmarks to see if any same issue exists, thank you very much for your advice

@yaooqinn
Copy link
Member Author

org.apache.spark.sql.execution.benchmark.DateTimeBenchmark

[error] Caused by: java.time.format.DateTimeParseException: Text '2019-01-27 11:02:01.0' could not be parsed at index 20
[error] 	at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2046)
[error] 	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
[error] 	at org.apache.spark.sql.catalyst.util.Iso8601TimestampFormatter.$anonfun$parse$1(TimestampFormatter.scala:71)
[error] 	... 19 more

@cloud-fan
Copy link
Contributor

@yaooqinn can you point out what query causes this issue?

@yaooqinn
Copy link
Member Author

  1. the IntervalBenchmark is as same as the last one in ExtractBenchmark
  2. the DateTimeBenchmark fails at to_timestamp('2019-01-27 11:02:01.0', 'yyyy-MM-dd HH:mm:ss.SSS'), which maybe should be to_timestamp('2019-01-27 11:02:01.000', 'yyyy-MM-dd HH:mm:ss.SSS') @cloud-fan

@yaooqinn yaooqinn changed the title [SPARK-31111][SQL][Tests][FOLLOWUP] Fix interval output issue in IntervalBenchmark [SPARK-31129][SQL][Tests] Fix IntervalBenchmark and DateTimeBenchmark Mar 12, 2020
@SparkQA
Copy link

SparkQA commented Mar 12, 2020

Test build #119691 has finished for PR 27885 at commit 7759b80.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Mar 12, 2020
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DateTimeBenchmark.scala
val n = 1000000
val timestampStrExpr = "concat('2019-01-27 11:02:01.', cast(mod(id, 1000) as string))"
val timestampStrExpr = "concat('2019-01-27 11:02:01.', rpad(mod(id, 1000), 3, '0'))"
val pattern = "'yyyy-MM-dd HH:mm:ss.SSS'"
Copy link
Contributor

@cloud-fan cloud-fan Mar 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one more thought: maybe we should change the pattern string to yyyy-M-d H: m:s.S, to match the behavior of 2.4

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will fail with 2019-01-27 11:02:01.12(123)

@SparkQA
Copy link

SparkQA commented Mar 12, 2020

Test build #119700 has finished for PR 27885 at commit 7a52ce7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 12, 2020

Test build #119705 has finished for PR 27885 at commit 4015c76.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes Mar 12, 2020
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for recovering benchmark suites, @yaooqinn and @cloud-fan .
I also checked the failure and the recovery. Merged to master/3.0.

dongjoon-hyun pushed a commit that referenced this pull request Mar 12, 2020
@yaooqinn @dongjoon-hyun
[SPARK-31129][SQL][TESTS] Fix IntervalBenchmark and DateTimeBenchmark
fd56924 
### What changes were proposed in this pull request?

This PR aims to recover `IntervalBenchmark` and `DataTimeBenchmark` due to banning intervals as output.

### Why are the changes needed?

This PR recovers the benchmark suite.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Manually, re-run the benchmark.

Closes #27885 from yaooqinn/SPARK-31111-2.

Authored-by: Kent Yao <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit fbc9dc7)
Signed-off-by: Dongjoon Hyun <[email protected]>
sjincho pushed a commit to sjincho/spark that referenced this pull request Apr 15, 2020
@yaooqinn
[SPARK-31129][SQL][TESTS] Fix IntervalBenchmark and DateTimeBenchmark
26763f5 
### What changes were proposed in this pull request?

This PR aims to recover `IntervalBenchmark` and `DataTimeBenchmark` due to banning intervals as output.

### Why are the changes needed?

This PR recovers the benchmark suite.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Manually, re-run the benchmark.

Closes apache#27885 from yaooqinn/SPARK-31111-2.

Authored-by: Kent Yao <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

TESTS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yaooqinn @dongjoon-hyun @cloud-fan @SparkQA