[SPARK-4236] Cleanup removed applications' files in shuffle service

[aarondav]

This relies on a hook from whoever is hosting the shuffle service to invoke removeApplication() when the application is completed. Once invoked, we will clean up all the executors' shuffle directories we know about.

[aarondav]

The following 32 lines are the only material change in this entire PR. The rest is utility functions imported from core Utils, Java cruft around lambdas and classes, and updating/adding tests.

[SparkQA]

Test build #22972 has started for PR 3126 at commit 0028241.

• This patch merges cleanly.

[SparkQA]

Test build #22973 has started for PR 3126 at commit 382c97d.

• This patch merges cleanly.

[aarondav]

@andrewor14 @rxin PTAL. For the YARN auxiliary service, I think it is sufficient just to just call ExternalShuffleBlockHandler#applicationRemoved(appId) at the appropriate time.

[SparkQA]

Test build #22972 has finished for PR 3126 at commit 0028241.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AllocatedBlocks(streamIdToAllocatedBlocks: Map[Int, Seq[ReceivedBlockInfo]])
  • class ReceiverTracker(ssc: StreamingContext, skipReceiverLaunch: Boolean = false) extends Logging

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22972/
Test PASSed.

[SparkQA]

Test build #22973 has finished for PR 3126 at commit 382c97d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22973/
Test PASSed.

[andrewor14]

Hey was this already merged? I don't see it showing up on ASF.

[aarondav]

Whoops, accidentally deleted it locally.

[SparkQA]

Test build #23015 has started for PR 3126 at commit 382c97d.

• This patch merges cleanly.

[andrewor14]

can you just do cleanupLocalDirs = false? Or can we not because this is a Java method even though we're using it in Scala

[aarondav]

Right, not possible if the method is written in Java, unfortunately.

[SparkQA]

Test build #23015 has finished for PR 3126 at commit 382c97d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23015/
Test PASSed.

[aarondav]

Comments addressed.

[SparkQA]

Test build #23026 has started for PR 3126 at commit a1f73ab.

• This patch merges cleanly.

[SparkQA]

Test build #23026 has finished for PR 3126 at commit a1f73ab.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23026/
Test PASSed.

[andrewor14]

LGTM. Hey by the way is there anything we need to change about the way ContextCleaner cleans up shuffle files now when external shuffle service is enabled? The only difference in functionality here is that we will not clean shuffle files based on scope anymore (we currently go through executors but they're now dead). This could be a problem for long-running contexts. Maybe we should add some extra TTL-based cleanup and default the timeout to a week or something in SPARK-4287.

Also I think this conflicts with your other patch which I just merged.

[aarondav]

It was you -- JavaUtils changed. Updated.

[SparkQA]

Test build #23032 has started for PR 3126 at commit 33a64a9.

• This patch merges cleanly.

[andrewor14]

Weird, we both only added code to JavaUtils. I would think that it doesn't need to conflict there.

[SparkQA]

Test build #23032 has finished for PR 3126 at commit 33a64a9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StandaloneWorkerShuffleService(sparkConf: SparkConf, securityManager: SecurityManager)

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23032/
Test PASSed.

[andrewor14]

Ok merging into master and 1.2