Skip to content

[SPARK-32466][SQL][FOLLOW-UP] Normalize Location info in explain plan #29537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[SPARK-32466][SQL][FOLLOW-UP] Normalize Location info in explain plan #29537

wants to merge 5 commits into from

Conversation

@Ngone51
Copy link
Member

@Ngone51 Ngone51 commented Aug 25, 2020 •
edited
Loading

What changes were proposed in this pull request?

  1. Extract SQLQueryTestSuite.replaceNotIncludedMsg to PlanTest.

  2. Reuse replaceNotIncludedMsg to normalize the explain plan that generated in PlanStabilitySuite.

Why are the changes needed?

This's a follow-up of #29270.
Eliminates the personal related information (e.g., local directories) in the explain plan.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Updated test.

@Ngone51
Copy link
Member Author

Ngone51 commented Aug 25, 2020

cc: @wangyum @maropu @cloud-fan

(Note this PR has only updated a few golden files yet for easier review as @maropu mentioned previously)

Ngone51
Ngone51 commented Aug 25, 2020
View reviewed changes
sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q11.sf100/explain.txt Outdated

(1) Scan parquet default.store_sales
Output [4]: [ss_sold_date_sk#1, ss_customer_sk#2, ss_ext_discount_amt#3, ss_ext_list_price#4]
Output [4]: [ss_sold_date_sk#x, ss_customer_sk#x, ss_ext_discount_amt#x, ss_ext_list_price#x]
Copy link
Member Author

@Ngone51 Ngone51 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for bringing massive changes again.

Do you think we should avoid this change? If it should, I think we could make a custom copy of replaceNotIncludedMsg in PlanStabilitySuite instead of resue.

Copy link
Contributor

@cloud-fan cloud-fan Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine

Copy link
Contributor

@cloud-fan cloud-fan Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a second thought, this might be useful to understand the query and debug the test failure, let's keep it.

Copy link
Member Author

@Ngone51 Ngone51 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

Copy link
Member

@maropu maropu Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to keep the info.

Copy link
Member Author

@Ngone51 Ngone51 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the formatted explained plan. So it doesn't really have Partition Statistics and codegenStageIds.

Copy link
Member

@maropu maropu Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the formatted explained plan. So it doesn't really have Partition Statistics and codegenStageIds.

Yea, just after I wrote the comment, I noticed that soon... Thanks!

@Ngone51 Ngone51 mentioned this pull request Aug 25, 2020
Closed
@Ngone51 Ngone51 force-pushed the follow-up-plan-stablity branch from 6f46616 to 702bb32 Compare August 25, 2020 06:45
Ngone51
Ngone51 commented Aug 25, 2020
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala
val approved = FileUtils.readFileToString(file, StandardCharsets.UTF_8)
approved == actualSimplifiedPlan
val expected = FileUtils.readFileToString(file, StandardCharsets.UTF_8)
expected == actualSimplifiedPlan
Copy link
Member Author

@Ngone51 Ngone51 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address the comment from #29270 (comment)

Ngone51
Ngone51 commented Aug 25, 2020
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala
* plan is not too useful for debugging
*/
private def generateApprovedPlanFile(plan: SparkPlan, name: String, explain: String): Unit = {
private def generateGoldenFile(plan: SparkPlan, name: String, explain: String): Unit = {
Copy link
Member Author

@Ngone51 Ngone51 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address the comment from #29270 (comment)

Ngone51
Ngone51 commented Aug 25, 2020
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala
* Project [c_customer_id]
*/
def getSimplifiedPlan(node: SparkPlan, depth: Int): String = {
def simplifyNode(node: SparkPlan, depth: Int): String = {
Copy link
Member Author

@Ngone51 Ngone51 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address the comment from #29270 (comment)

@SparkQA
Copy link

SparkQA commented Aug 25, 2020

Test build #127868 has finished for PR 29537 at commit 6f46616.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 25, 2020

Test build #127875 has finished for PR 29537 at commit 702bb32.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@Ngone51
Copy link
Member Author

Ngone51 commented Aug 25, 2020

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 25, 2020

Test build #127878 has finished for PR 29537 at commit 702bb32.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Aug 25, 2020

Merged to master.

@Ngone51 Ngone51 mentioned this pull request Aug 26, 2020
Closed
@Ngone51
Copy link
Member Author

Ngone51 commented Aug 26, 2020

Thank you for your approval and the merging. I created another followup PR to generate all the golden file: #29546 PTAL!

HyukjinKwon pushed a commit that referenced this pull request Aug 26, 2020
@Ngone51 @HyukjinKwon
[SPARK-32466][FOLLOW-UP][TEST][SQL] Regenerate the golden explain fil…
f510d21 
…e for PlanStabilitySuite

### What changes were proposed in this pull request?

This PR regenerates the golden explain file based on the fix: #29537

### Why are the changes needed?

Eliminates the personal related information (e.g., local directories) in the explain plan.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Checked manually.

Closes #29546 from Ngone51/follow-up-gen-golden-file.

Authored-by: yi.wu <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maropu maropu maropu approved these changes

@cloud-fan cloud-fan cloud-fan approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Ngone51 @SparkQA @HyukjinKwon @maropu @cloud-fan