[SPARK-10790][YARN] Fix initial executor number not set issue and consolidate the codes

[jerryshao]

This bug is introduced in SPARK-9092, targetExecutorNumber should use minExecutors if initialExecutors is not set. Using 0 instead will meet the problem as mentioned in SPARK-10790.

Also consolidate and simplify some similar code snippets to keep the consistent semantics.

[SparkQA]

Test build #42987 has finished for PR 8910 at commit c912a2a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

The refactoring is nice. This is a reasonable place for this code now, though eventually it may not be YARN-specific?

[jerryshao]

Thanks @srowen for your review, I've updated the codes according to your comments.

[vanzin]

nit: "Getting the initial target number of executors depends on whether dynamic allocation is enabled."

[vanzin]

LGTM pending a couple of minor changes.

[SparkQA]

Test build #43027 has finished for PR 8910 at commit e6547fb.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43030 has finished for PR 8910 at commit d53d5c8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

retest this please

[vanzin]

not necessary anymore?

[SparkQA]

Test build #43032 has finished for PR 8910 at commit d53d5c8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43033 has finished for PR 8910 at commit 0e54fb1.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

Jenkins, retest this please.

[SparkQA]

Test build #43042 has finished for PR 8910 at commit 0e54fb1.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

Looks like some other patches introduce this mima failures.

[SparkQA]

Test build #1818 has finished for PR 8910 at commit 0e54fb1.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

Jenkins, retest this please.

[SparkQA]

Test build #43056 has finished for PR 8910 at commit 0e54fb1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

LGTM, merging to master and branch-1.5.