Skip to content

Generalize pattern for planning hash joins. #418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

Generalize pattern for planning hash joins. #418

wants to merge 4 commits into from

Conversation

marmbrus
Copy link
Contributor

This will be helpful for SPARK-1495 and other cases where we want to have custom hash join implementations but don't want to repeat the logic for finding the join keys.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14152/

@adrian-wang
Copy link
Contributor

Thanks Michael! This will be really helpful.

@pwendell
Copy link
Contributor

Jenkins, retest this please (need to test our test harness on a SQL patch).

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14167/

@marmbrus
Copy link
Contributor Author

test this please

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14399/

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14402/

@marmbrus
Copy link
Contributor Author

@rxin PTAL

rxin
rxin reviewed Apr 24, 2014
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this used anywhere? (it is private def)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Removed as reduceLeft(And) is shorter and probably more clear anyway.

@rxin
Copy link
Contributor

rxin commented Apr 24, 2014

Overall looks pretty good. I left some comments mainly on readability.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14464/

@rxin
Copy link
Contributor

rxin commented Apr 25, 2014

Thanks. I've merged this.

asfgit pushed a commit that referenced this pull request Apr 25, 2014
@marmbrus @rxin
Generalize pattern for planning hash joins.
ab131ab 
This will be helpful for [SPARK-1495](https://issues.apache.org/jira/browse/SPARK-1495) and other cases where we want to have custom hash join implementations but don't want to repeat the logic for finding the join keys.

Author: Michael Armbrust <[email protected]>

Closes #418 from marmbrus/hashFilter and squashes the following commits:

d5cc79b [Michael Armbrust] Address @rxin 's comments.
366b6d9 [Michael Armbrust] style fixes
14560eb [Michael Armbrust] Generalize pattern for planning hash joins.
f4809c1 [Michael Armbrust] Move common functions to PredicateHelper.

(cherry picked from commit 86ff8b1)
Signed-off-by: Reynold Xin <[email protected]>
@asfgit asfgit closed this in 86ff8b1 Apr 25, 2014
@adrian-wang adrian-wang mentioned this pull request May 12, 2014
Closed
@adrian-wang adrian-wang mentioned this pull request May 20, 2014
Closed
@marmbrus marmbrus deleted the hashFilter branch June 6, 2014 05:25
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@marmbrus @pdeyhim
Generalize pattern for planning hash joins.
8cc2307 
This will be helpful for [SPARK-1495](https://issues.apache.org/jira/browse/SPARK-1495) and other cases where we want to have custom hash join implementations but don't want to repeat the logic for finding the join keys.

Author: Michael Armbrust <[email protected]>

Closes apache#418 from marmbrus/hashFilter and squashes the following commits:

d5cc79b [Michael Armbrust] Address @rxin 's comments.
366b6d9 [Michael Armbrust] style fixes
14560eb [Michael Armbrust] Generalize pattern for planning hash joins.
f4809c1 [Michael Armbrust] Move common functions to PredicateHelper.
andrewor14 pushed a commit to andrewor14/spark that referenced this pull request Jan 8, 2015
@pwendell
Merge pull request apache#418 from pwendell/0.9-versions
b60840e 
Version changes for release 0.9.0.
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
Merge pull request apache#418 from theopenlab/ansible-switch
ada4f0e 
Switch to official ansible project
@imback82 imback82 mentioned this pull request Apr 29, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@marmbrus @AmplabJenkins @adrian-wang @pwendell @rxin