[SPARK-2925] [sql]fix spark-sql and start-thriftserver shell bugs when set --driver-java-options

[scwf]

https://issues.apache.org/jira/browse/SPARK-2925

Run cmd like this will get the error
bin/spark-sql --driver-java-options '-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8788,server=y,suspend=y'

Error: Unrecognized option '-Xnoagent'.
Run with --help for usage help or --verbose for debug output

[AmplabJenkins]

Can one of the admins verify this patch?

[scwf]

start-thriftserver also has the problem, fixed it

[scwf]

@marmbrus, can you verify this pr?

[marmbrus]

test this please

[marmbrus]

/cc @liancheng

[SparkQA]

QA tests have started for PR 1851. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18293/consoleFull

[SparkQA]

QA results for PR 1851:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18293/consoleFull

[liancheng]

@scwf Bash escaping is really annoying... Thanks for spotting and fixing this issue! This PR fixed the --driver-java-options option, but any other options that may accept quoted string with spaces still suffer this issue.

The root cause here is that variables in these two scripts are not properly quoted. We should quote them to fix all similar cases. Take bin/spark-sql as an example, all CLI_ARGS+=($1) and CLI_ARGS+=(\"$1\") should be CLI_ARGS+=("$1"), and the last line

eval exec "$FWDIR"/bin/spark-submit --class $CLASS ${SUBMISSION_ARGS[*]} spark-internal ${CLI_ARGS[*]}

should be

exec "$FWDIR"/bin/spark-submit --class $CLASS "${SUBMISSION_ARGS[@]}" spark-internal "${CLI_ARGS[@]}"

The same rules applies to sbin/start-thriftserver.sh.

[scwf]

@liancheng, using CLI_ARGS+=("$1") , "${SUBMISSION_ARGS[@]}" and "${CLI_ARGS[@]}" can not fix this issue, i think there are only two options may leads to the problem:

1. -e quoted-query-string , now the bash is ok for this option because it use CLI_ARGS+=("$1")
2. --driver-java-options java options, this pr fixed it
   so we only need handle --driver-java-options issue

[liancheng]

PR #1886 uses similar approach I mentioned above. I tested both -e and your --driver-java-options cases locally with #1886, and they seem working. My environment is Mac OS X 10.9 + Bash 4.3.18. (Please note that the eval is removed.)

[scwf]

@liancheng yeah, my mistake , we should also use SUBMISSION_ARGS+=("$1") to instead SUBMISSION_ARGS+=($1) , it will be ok

[scwf]

updated

[marmbrus]

ok to test

[SparkQA]

QA tests have started for PR 1851. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18354/consoleFull

[marmbrus]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1851. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18486/consoleFull

[scwf]

@marmbrus, the test failures but it seems error when running PySpark tests. And the jenkins seems not post tests result to Github. Can you suggest for that？

[liancheng]

retest this please

[SparkQA]

QA tests have started for PR 1851. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18540/consoleFull

[marmbrus]

Tests passed but failed to post to github.

I'm going to merge this to master and 1.1. Thanks!