Skip to content

Conversation

@windpiger
Copy link
Contributor

What changes were proposed in this pull request?

current spark can handle this query to infer fitler from join condition:

select * from t1 join t2 where ((t1.a = t2.d and t1.b = 1) or (t1.a = t2.d and t1.b = 5))

(t1.b =1 or t1.b =5) can be infered, and pushed to table t1.

but it can not handle following query:

select * from t1 join t2 where (t1.a = t2.d and t1.b = 1 and t2.e=3 ) or (t1.a = t2.d and t1.b = 5 and t2.e=7)

(t1.b=1 or t1.b=5) should be infered and push to t1
(t2.e=3 or t2.e=7) should be infered and push to t2

How was this patch tested?

unit test added

@windpiger windpiger changed the title infer more filter from join or condition [SPARK-27280][SQL]infer more filter from join or condition Mar 27, 2019
@beliefer
Copy link
Contributor

Is this SQL grammar supported by Hive or standard SQL?

}
}

/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this SQL grammar supported by Hive or standard SQL?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hive can do this

@SparkQA
Copy link

SparkQA commented Mar 27, 2019

Test build #104008 has finished for PR 24228 at commit 879c98e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AngersZhuuuu
Copy link
Contributor

AngersZhuuuu commented Jul 16, 2019

#24973 I want to do the same thing for partition puring

@maropu
Copy link
Member

maropu commented Jul 21, 2019

I also think we need not a band-aid fix, but a general solution for this issue as @cloud-fan suggested in #24973. Have you checked the previous activity for the CNF normalization? https://issues.apache.org/jira/browse/SPARK-6624

@AngersZhuuuu
Copy link
Contributor

AngersZhuuuu commented Jul 21, 2019

I also think we need not a band-aid fix, but a general solution for this issue as @cloud-fan suggested in #24973. Have you checked the previous activity for the CNF normalization? https://issues.apache.org/jira/browse/SPARK-6624

thanks for Issue link. I will work on this in partition predicate part.

@github-actions
Copy link

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just
a way of keeping the PR queue manageable.

If you'd like to revive this PR, please reopen it!

@github-actions github-actions bot added the Stale label Dec 31, 2019
@github-actions github-actions bot closed this Jan 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants