[SPARK-23778][CORE] Avoid unneeded shuffle when union gets an empty RDD #21333
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #90647 has finished for PR 21333 at commit
|
varunvish910
approved these changes
May 30, 2018
jiangxb1987
reviewed
May 31, 2018
| } | ||
| } | ||
|
|
||
| test("SPARK-23778: empty RDD in union should not produce a UnionRDD") { |
There was a problem hiding this comment.
Have we tested when all input RDDs are empty?
There was a problem hiding this comment.
When all RDDs are empty we are returning a UnionRDD. Though in this case it is not a big issue, since a shuffle of an empty RDD is not an issue.
There was a problem hiding this comment.
can we add a test? just make sure we are safe when UnionRDD.rdds is Nil
|
@varuvish the Dataset API uses |
|
LGTM |
|
Test build #92098 has finished for PR 21333 at commit
|
|
thanks, merging to master! |
zzcclp
added a commit
to zzcclp/spark
that referenced
this pull request
Dec 6, 2018
apache#21333 When a union is invoked on several RDDs of which one is an empty RDD, the result of the operation is a UnionRDD. This causes an unneeded extra-shuffle when all the other RDDs have the same partitioning. The PR ignores incoming empty RDDs in the union method.
zzcclp
added a commit
to zzcclp/spark
that referenced
this pull request
Sep 20, 2019
…pty RDD apache#21333 When a union is invoked on several RDDs of which one is an empty RDD, the result of the operation is a UnionRDD. This causes an unneeded extra-shuffle when all the other RDDs have the same partitioning. The PR ignores incoming empty RDDs in the union method.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
When a
unionis invoked on several RDDs of which one is an empty RDD, the result of the operation is aUnionRDD. This causes an unneeded extra-shuffle when all the other RDDs have the same partitioning.The PR ignores incoming empty RDDs in the union method.
How was this patch tested?
added UT