[SPARK-14293] Improve shuffle load balancing and minimize network data transmission.

[peterpc0701]

What changes were proposed in this pull request?

Based on the map output sizes and locations tracked by MapOutputTracker, we can obtain a better load balancing

This patch proposes a strategy to set preferred locations for each reduce task, which could firstly keep each executor process almost the same amount of intermediate data and secondly minimize the network data transmission. This can benefit some conditions:

1．REDUCER_PREF_LOCS_FRACTION tries to place the reduce tasks close to the largest output. If there exists data skew in the map outputs. It could cause some executors that have large of map outputs become busy. Our method could avoid this case and minimize the network data transmission.

2．When there are large of reduce tasks in the job, it helps each executor processes almost the same data and keeps load balancing.

The special steps are following.
Step 1: For each task, calculate the amount of data and their distributions.

Step 2: Divide the tasks into n groups according to the number of nodes and data size, ensuring that the data size for each group is nearly equal.

Step 3: Determine the amount of local data if the tasks of every group are executed on every node. Thus, a n × n matrix is created.

Step 4: Choose the largest value in the matrix to identify which group is allocated to which node. Mark the row and column at which the selected group is located to ensure that the group is not chosen next time. Goto Step 4 until no group is available.

How was this patch tested?

Unit test suite
Author: Cheng Pei [email protected]

[holdenk]

This is interesting, there are some minor formatting things that might make sense to fix while waiting for review - but since this is designed to improve performance it probably makes sense to also do some performance testing so that we all can see what the benefit would look like. Maybe @rxin would like to take a look?

[peterpc0701]

@holdenk, thank you for your comments. I have fixed some minor formatting things about this issue. And some performance testings will be given in the last few days.

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

Hi @peterpc0701, how is the perf test going?