[SPARK-17979] Remove deprecated SPARK_YARN_USER_ENV and [SPARK-14453] SPARK_JAVA_OPTS

[yongtang]

What changes were proposed in this pull request?

This fix removes deprecated support for config SPARK_YARN_USER_ENV, as is mentioned in SPARK-17979.
This fix also removes deprecated support for the following:

SPARK_YARN_USER_ENV
SPARK_JAVA_OPTS
SPARK_CLASSPATH
SPARK_WORKER_INSTANCES

Related JIRA:
[SPARK-14453]: https://issues.apache.org/jira/browse/SPARK-14453
[SPARK-12344]: https://issues.apache.org/jira/browse/SPARK-12344
[SPARK-15781]: https://issues.apache.org/jira/browse/SPARK-15781

How was this patch tested?

Existing tests should pass.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

I think it's indeed OK to remove at this point because it's not documented, and because of the JIRA discussion. However it's ideal if you can look for other SPARK_* env variables that are long since gone but still checked in the code.

[yongtang]

Thanks @srowen Will take a look at other envs and update the PR if relevant.

[vanzin]

And please remove the template text from your PR description.

[yongtang]

@srowen @vanzin The PR has been updated with the following removed:

SPARK_YARN_USER_ENV
SPARK_JAVA_OPTS
SPARK_CLASSPATH
SPARK_WORKER_INSTANCES

All of them have already been labeled in the source code:

This is deprecated in Spark 1.0+.

(The PR description has been updated as well)

Please take a look and let me know if there are any issues.

[srowen]

This actually covers SPARK-14453 too then. It relates to SPARK-12344 but doesn't fully resolve it. We can link all these in the title though.

Yes, I support removing these props as they have been deprecated a long, long time.

There are more usages of SPARK_WORKER_INSTANCES and SPARK_CLASSPATH though.

[yongtang]

Thanks @srowen. The PR has been updated with additional usage of SPARK_CLASSPATH and SPARK_WORKER_INSTANCES removed.

There are SPARK_WORKER_INSTANCES used in start-slave.sh and stop-slave.sh but it looks like they are launching multiple processes so not sure if they should be removed as well.

[srowen]

You had a good point about SPARK_WORKER_INSTANCES being used in standalone mode. I think we might need to keep this doc as a result.

[yongtang]

@srowen The PR has been updated with the doc kept. Please take a look.

[vanzin]

LGTM. I also linked SPARK-17979 to the "parent" of all these cleanup tasks (SPARK-12344); before we completely remove things like SPARK_WORKER_INSTANCES we'll need to add proper support for configuration files in the different Spark daemons (master, worker, etc).

[vanzin]

Merging to master.