Skip to content

[SPARK-26757][GraphX] Return 0 for count on empty Edge/Vertex RDDs #23681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[SPARK-26757][GraphX] Return 0 for count on empty Edge/Vertex RDDs #23681

wants to merge 3 commits into from

Conversation

@huonw
Copy link
Contributor

@huonw huonw commented Jan 29, 2019 •
edited
Loading

What changes were proposed in this pull request?

Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using reduce, rather
than fold or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.

How was this patch tested?

Two new unit tests.

@huonw
[SPARK-26757][GraphX] Return 0 for count on empty Edge/Vertex RDDs
a843a0b 
Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using `reduce`, rather
than `fold` or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.
@SparkQA
Copy link

SparkQA commented Jan 30, 2019

Test build #4537 has finished for PR 23681 at commit a843a0b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

maropu
maropu reviewed Jan 31, 2019
View reviewed changes
huonw added 2 commits January 31, 2019 13:41
@huonw
[SPARK-26757][GraphX] Remove unused variable
1f9dd8f
@huonw
[SPARK-26757][GraphX] Make SVD++ work with no edges
de47631 
Similar to `VertexRDD.count` and `EdgeRDD.count`, this used `reduce`
which fails on an empty RDD.
@SparkQA
Copy link

SparkQA commented Jan 31, 2019

Test build #4539 has finished for PR 23681 at commit de47631.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen pushed a commit that referenced this pull request Jan 31, 2019
@huonw @srowen
[SPARK-26757][GRAPHX] Return 0 for count on empty Edge/Vertex RDDs
2b5e033 
## What changes were proposed in this pull request?

Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using `reduce`, rather
than `fold` or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.

## How was this patch tested?

Two new unit tests.

Closes #23681 from huonw/empty-graphx.

Authored-by: Huon Wilson <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit da52698)
Signed-off-by: Sean Owen <[email protected]>
srowen pushed a commit that referenced this pull request Jan 31, 2019
@huonw @srowen
[SPARK-26757][GRAPHX] Return 0 for count on empty Edge/Vertex RDDs
537d15c 
## What changes were proposed in this pull request?

Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using `reduce`, rather
than `fold` or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.

## How was this patch tested?

Two new unit tests.

Closes #23681 from huonw/empty-graphx.

Authored-by: Huon Wilson <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit da52698)
Signed-off-by: Sean Owen <[email protected]>
@srowen
Copy link
Member

srowen commented Jan 31, 2019

Merged to master/2.4/2.3

@srowen srowen closed this in da52698 Jan 31, 2019
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
@huonw
[SPARK-26757][GRAPHX] Return 0 for count on empty Edge/Vertex RDDs
6893c30 
## What changes were proposed in this pull request?

Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using `reduce`, rather
than `fold` or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.

## How was this patch tested?

Two new unit tests.

Closes apache#23681 from huonw/empty-graphx.

Authored-by: Huon Wilson <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
@huonw huonw deleted the empty-graphx branch March 17, 2019 02:24
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 23, 2019
@huonw @kai-chi
[SPARK-26757][GRAPHX] Return 0 for count on empty Edge/Vertex RDDs
b9b2b3e 
## What changes were proposed in this pull request?

Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using `reduce`, rather
than `fold` or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.

## How was this patch tested?

Two new unit tests.

Closes apache#23681 from huonw/empty-graphx.

Authored-by: Huon Wilson <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit da52698)
Signed-off-by: Sean Owen <[email protected]>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 25, 2019
@huonw @kai-chi
[SPARK-26757][GRAPHX] Return 0 for count on empty Edge/Vertex RDDs
28895c9 
## What changes were proposed in this pull request?

Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using `reduce`, rather
than `fold` or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.

## How was this patch tested?

Two new unit tests.

Closes apache#23681 from huonw/empty-graphx.

Authored-by: Huon Wilson <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit da52698)
Signed-off-by: Sean Owen <[email protected]>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Aug 1, 2019
@huonw @kai-chi
[SPARK-26757][GRAPHX] Return 0 for count on empty Edge/Vertex RDDs
2a3ece5 
## What changes were proposed in this pull request?

Previously a "java.lang.UnsupportedOperationException: empty
collection" exception would be thrown due to using `reduce`, rather
than `fold` or similar that can tolerate empty RDDs.

This behaviour has existed for the Vertex RDDs since it was introduced
in b30e0ae. It seems this behaviour
was inherited by the Edge RDDs via copy-paste in
ee29ef3.

## How was this patch tested?

Two new unit tests.

Closes apache#23681 from huonw/empty-graphx.

Authored-by: Huon Wilson <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit da52698)
Signed-off-by: Sean Owen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowen srowen srowen approved these changes

@maropu maropu maropu approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@huonw @SparkQA @srowen @maropu