SPARK-2630 Input data size of CoalescedRDD counts only one partition

[ash211]

No description provided.

[SparkQA]

QA tests have started for PR 2310 at commit df9ce44.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2310 at commit df9ce44.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ash211]

Failed tests seem to be Jenkins broken-ness

[pwendell]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 2310 at commit df9ce44.

• This patch merges cleanly.

[pwendell]

@ash211 this particular fix will only work for HadoopRDD but not other cases where we track input bytes read. A more general fix is pretty simple though, I think.

The way to fix this is to change the input bytes read from a Long to an AtomicLong and to incrementAndGet it whenever it is modified (currently in HadoopRDD, newHadoopRDD, and BlockManager) instead of just setting the value directly. I think that would fix the case where it's mutated multiple times in a single task.

[SparkQA]

QA tests have finished for PR 2310 at commit df9ce44.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ash211]

Switched to an AtomicLong. Also needed to move instantiation of metrics outside of compute() This now works in my testing for the HadoopRDD -> CoalescedRDD pipeline like this:

sc.textFile("pom.xml").coalesce(1).count()

Ready for review

[SparkQA]

QA tests have started for PR 2310 at commit 576483b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2310 at commit 576483b.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

This is part of public API, could we fix the issue without change the API?

Also, it looks different than others. I'd prefer to keep the type unchanged.

[davies]

@ash211 Hopefully this could be merged into 1.2, once you rebase it to master and keep the API unchanged, thanks!

[pwendell]

@ash211 I think we can close this issue now that we have merged #3120

[ash211]

Sounds good, I concur. Thanks!