SPARK-2791: Fix committing, reverting and state tracking in shuffle file consolidation

[aarondav]

All changes from this PR are by @mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle file consolidation that @mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1.

This patch is not intended as a replacement for #1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests.

If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable option.

[SparkQA]

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17535/consoleFull

[SparkQA]

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17535/consoleFull

[SparkQA]

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17537/consoleFull

[aarondav]

@mridulm Please take a look if possible.

@mateiz This now interacts with the ExternalSorter stuff, and it's possible it partially helps fix a serialization bug (since some serializers apparently write things during a close() after a flush()).

[witgo]

We can also merge it to 1.0 branch.

[mateiz]

That's true, this is small enough to add back into branch-1.0 too. We should see if we can do the same for the rest of #1609 too (ideally I would like to merge that whole PR into branch-1.0, but I'm not sure how easy that is).

[SparkQA]

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17537/consoleFull

[SparkQA]

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17553/consoleFull

[SparkQA]

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17553/consoleFull

[aarondav]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17565/consoleFull

[SparkQA]

QA results for PR 1678:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17565/consoleFull

[aarondav]

Both prior failures are due to being unable to bind to ports, which sounds like leftover state from other jenkins runs (especially since they were two entirely disjoint sets of failed tests related to port binding).

[aarondav]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17585/consoleFull

[SparkQA]

QA results for PR 1678:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17585/consoleFull

[mridulm]

revert can throw exception : which will cause other writers to not revert.
We need to wrap it in try/catch, log and continue

[aarondav]

Revert actually doesn't throw, per its (updated) comment.

[mateiz]

Jenkins, test this please

[SparkQA]

QA tests have started for PR 1678. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17689/consoleFull

[SparkQA]

QA results for PR 1678:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17689/consoleFull

[mateiz]

@aarondav can you update the name of this PR to reference this sub-task JIRA instead: https://issues.apache.org/jira/browse/SPARK-2791?

[aarondav]

Updated title.

[mateiz]

Alright, going to merge it then. The changes look good to me.

[JoshRosen]

This catches Exception but it could catch IOException instead.