Skip to content

[SPARK-12470] [SQL] Fix size reduction calculation #10421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-12470] [SQL] Fix size reduction calculation #10421

wants to merge 1 commit into from

Conversation

@robbinspg
Copy link
Member

also only allocate required buffer size

@robbinspg
Copy link
Member Author

I believe this is uncovering a test failure in ExchangeCoordinatorSuite so this please hold this PR until I investigate further.

  • determining the number of reducers: aggregate operator *** FAILED ***
    3 did not equal 2 (ExchangeCoordinatorSuite.scala:316)

EDIT above tests now working with this PR merged into latest master

I'd appreciate it if someone could verify the patch as I'm pretty sure it is correct.

@robbinspg robbinspg changed the title [SPARK-12470] Fix size reduction calculation [SPARK-12470] [SQL] Fix size reduction calculation Dec 23, 2015
@robbinspg
Copy link
Member Author

@rxin as the original author of this code could you please review the PR?

@rxin
Copy link
Contributor

rxin commented Dec 29, 2015

I think @davies did some pretty big refactoring of this later. @davies can you take a quick look at this? Thanks.

davies
davies reviewed Dec 29, 2015
View reviewed changes
...c/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be better to use number of bytes for sizeReduction (also update the comments).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. sorted. Can this be run by Jenkins tests?

@davies
Copy link
Contributor

davies commented Dec 29, 2015

@rxin I had finished the refactoring long time ago.

@davies
Copy link
Contributor

davies commented Dec 30, 2015

LGTM, pending for tests.

@SparkQA
Copy link

SparkQA commented Dec 30, 2015

Test build #2268 has finished for PR 10421 at commit 12724b3.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@davies
Copy link
Contributor

davies commented Dec 30, 2015

Jenkins, OK to test.

@robbinspg
Copy link
Member Author

Fixed scala style check

Please retest

@SparkQA
Copy link

SparkQA commented Dec 30, 2015

Test build #2273 has finished for PR 10421 at commit 1a09c7a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

davies
davies reviewed Dec 30, 2015
View reviewed changes
...c/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$sizeReduction

@robbinspg
Copy link
Member Author

re-merged with latest master

Please retest

@SparkQA
Copy link

SparkQA commented Jan 1, 2016

Test build #2282 has finished for PR 10421 at commit e5a8d1e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Jan 1, 2016

I will let @davies merge it.

@asfgit asfgit closed this in b504b6a Jan 4, 2016
asfgit pushed a commit that referenced this pull request Jan 4, 2016
@robbinspg @davies
[SPARK-12470] [SQL] Fix size reduction calculation
b5a1f56 
also only allocate required buffer size

Author: Pete Robbins <[email protected]>

Closes #10421 from robbinspg/master.

(cherry picked from commit b504b6a)
Signed-off-by: Davies Liu <[email protected]>

Conflicts:
	sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala
@davies
Copy link
Contributor

davies commented Jan 4, 2016

Merged into master and 1.6 branch, thanks!

@robbinspg
Copy link
Member Author

Merging this into the 1.6 stream has caused a test failure in

org.apache.spark.sql.execution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

There was a change in master in the ExchangeCoordinatorSuite which set the expected partition sizes to a new value. I do not understand why the change in this PR affects the input partition sizes but it does.

I think this is a test issue rather than an issue with this PR. Should I raise a new Jira to fix the expected partition sizes?

@rxin
Copy link
Contributor

rxin commented Jan 5, 2016

cc @yhuai on this one

@robbinspg
Copy link
Member Author

I have a fix for the test failure. Should I create a new Jira and PR?

@robbinspg
Copy link
Member Author

created https://issues.apache.org/jira/browse/SPARK-12647 and associated PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@robbinspg @rxin @davies @SparkQA