[SPARK-2964] [SQL] Remove duplicated code from spark-sql and start-thriftserver.sh #1886

sarutak · 2014-08-11T07:08:44Z

No description provided.

SparkQA · 2014-08-11T07:14:42Z

QA tests have started for PR 1886. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18303/consoleFull

SparkQA · 2014-08-11T07:58:18Z

QA results for PR 1886:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18303/consoleFull

sarutak · 2014-08-11T09:47:48Z

I think, much better solution is modify spark-sql script to use utils.sh.

SparkQA · 2014-08-11T09:49:41Z

QA tests have started for PR 1886. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18305/consoleFull

liancheng · 2014-08-11T15:49:39Z

bin/spark-sql

Please remove the extra newline.

liancheng · 2014-08-11T15:57:12Z

bin/spark-sql and sbin/start-thriftserver.sh were both added earlier and didn't get updated when bin/utils.sh was added. Thanks for fixing both this issue and the -S one. Left some comments about coding style, otherwise LGTM.

sarutak · 2014-08-11T17:48:28Z

@liancheng Thanks reviewing!
Now I've just modified coding styles and removed useless code.

SparkQA · 2014-08-11T17:49:45Z

QA tests have started for PR 1886. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18312/consoleFull

liancheng · 2014-08-11T18:29:41Z

bin/spark-sql

Actually the eval here has to be removed since we've already quoted things. You may try the following to verify:

./bin/spark-sql --name "a b"

SparkQA · 2014-08-11T18:59:51Z

QA tests have started for PR 1886. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18320/consoleFull

marmbrus · 2014-08-13T21:42:04Z

@liancheng is this good to go?

marmbrus · 2014-08-13T22:30:29Z

Jenkins, test this please.

SparkQA · 2014-08-13T22:35:06Z

QA tests have started for PR 1886. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18481/consoleFull

liancheng · 2014-08-14T13:34:04Z

@marmbrus Yes, this is ready to be merged.

marmbrus · 2014-08-14T17:45:45Z

Hmmm... could me more transient failures (/cc @davies @JoshRosen), but since you modified pyspark launching scripts i'm hesitant to merge this until it passes jenkins.

marmbrus · 2014-08-14T17:45:54Z

Jenkins, test this please.

sarutak · 2014-08-16T07:43:09Z

Oh... Jenkins doesn't like me? :(

liancheng · 2014-08-16T08:00:02Z

Jenkins is indeed a little overloaded these days because of all those passionate contributors like you :) But this time I think it's because this branch can no longer be merged cleanly. Would you mind to rebase this PR? BTW, we can make sbin/start-thriftserver.sh to leverage bin/utils.sh if you'd like to. It should be straightforward. Thanks!

sarutak · 2014-08-16T08:07:00Z

@liancheng Thanks!
I'll rebase this PR and I'll try to make start-thriftserver.sh to leverage bin/utils.sh.

SparkQA · 2014-08-16T08:30:05Z

QA tests have started for PR 1886 at commit 6d8e802.

This patch merges cleanly.

SparkQA · 2014-08-16T09:21:41Z

QA tests have finished for PR 1886 at commit 6d8e802.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- $FWDIR/bin/spark-submit --class org.apache.spark.repl.Main "$
- $FWDIR/bin/spark-submit --class org.apache.spark.repl.Main "$
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- "$FWDIR"/bin/spark-submit --class $CLASS "$

SparkQA · 2014-08-18T17:30:46Z

QA tests have started for PR 1886 at commit f63f897.

This patch merges cleanly.

SparkQA · 2014-08-18T18:33:00Z

QA tests have finished for PR 1886 at commit f63f897.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2014-08-19T08:20:13Z

@sarutak Would you mind to add a description of fixing the -S | --silent option issue in the PR description? This issue is actually more important than leveraging bin/utils.sh.

SparkQA · 2014-08-23T09:15:44Z

QA tests have started for PR 1886 at commit ffb68fa.

This patch merges cleanly.

sarutak · 2014-08-23T09:17:37Z

I rebased #1994 to this PR for now, and rename the title of this PR to proper one.

SparkQA · 2014-08-23T10:32:34Z

QA tests have finished for PR 1886 at commit ffb68fa.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- "$FWDIR"/bin/spark-submit --class $CLASS "$

liancheng · 2014-08-24T06:16:29Z

LGTM.

@sarutak Did you merge my branch to yours? 'Cause I saw my commits in the history of this branch. Actually you can first update your local master branch, then execute git rebase master to rebase your branch to master. For now, this branch is OK anyway.

sarutak · 2014-08-24T09:43:53Z

@liancheng Yes, I merged your branch to mine. I'll update my branch if your #1994 is updated, thanks!

marmbrus · 2014-08-26T01:22:15Z

Can you please update this to resolve the merge conflicts?

This reverts commit 81b43a8.

sarutak · 2014-08-26T01:38:08Z

@marmbrus I resolved merge conflicts.
@liancheng this PR includes your commit in #1944 without 81b43a8 , you need resolve merge issue only about 81b43a8 in #1944 .

SparkQA · 2014-08-26T01:41:01Z

QA tests have started for PR 1886 at commit 8ef8751.

This patch merges cleanly.

SparkQA · 2014-08-26T02:51:12Z

QA tests have finished for PR 1886 at commit 8ef8751.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- case class ExplainCommand(plan: LogicalPlan, extended: Boolean = false) extends Command

sarutak · 2014-08-26T02:59:33Z

Jenkins, retest this please.

SparkQA · 2014-08-26T03:05:52Z

QA tests have started for PR 1886 at commit 8ef8751.

This patch merges cleanly.

SparkQA · 2014-08-26T04:21:00Z

QA tests have finished for PR 1886 at commit 8ef8751.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- "$FWDIR"/bin/spark-submit --class $CLASS "$

sarutak · 2014-08-26T04:48:20Z

I believe the failure is not caused by my changes.
Jenkins, retest this please.

sarutak · 2014-08-26T07:14:23Z

Jenkins, retest this please.

SparkQA · 2014-08-26T07:21:09Z

QA tests have started for PR 1886 at commit 8ef8751.

This patch merges cleanly.

SparkQA · 2014-08-26T08:39:34Z

QA tests have finished for PR 1886 at commit 8ef8751.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- "$FWDIR"/bin/spark-submit --class $CLASS "$

marmbrus · 2014-08-27T00:34:11Z

Thanks! I've merged this to master and 1.1.

…riftserver.sh Author: Cheng Lian <[email protected]> Author: Kousuke Saruta <[email protected]> Closes #1886 from sarutak/SPARK-2964 and squashes the following commits: 8ef8751 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2964 26e7c95 [Kousuke Saruta] Revert "Shorten timeout to more reasonable value" ffb68fa [Kousuke Saruta] Modified spark-sql and start-thriftserver.sh to use bin/utils.sh 8c6f658 [Kousuke Saruta] Merge branch 'spark-3026' of https://github.com/liancheng/spark into SPARK-2964 81b43a8 [Cheng Lian] Shorten timeout to more reasonable value a89e66d [Cheng Lian] Fixed command line options quotation in scripts 9c894d3 [Cheng Lian] Fixed bin/spark-sql -S option typo be4736b [Cheng Lian] Report better error message when running JDBC/CLI without hive-thriftserver profile enabled (cherry picked from commit faeb9c0) Signed-off-by: Michael Armbrust <[email protected]>

…riftserver.sh Author: Cheng Lian <[email protected]> Author: Kousuke Saruta <[email protected]> Closes apache#1886 from sarutak/SPARK-2964 and squashes the following commits: 8ef8751 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-2964 26e7c95 [Kousuke Saruta] Revert "Shorten timeout to more reasonable value" ffb68fa [Kousuke Saruta] Modified spark-sql and start-thriftserver.sh to use bin/utils.sh 8c6f658 [Kousuke Saruta] Merge branch 'spark-3026' of https://github.com/liancheng/spark into SPARK-2964 81b43a8 [Cheng Lian] Shorten timeout to more reasonable value a89e66d [Cheng Lian] Fixed command line options quotation in scripts 9c894d3 [Cheng Lian] Fixed bin/spark-sql -S option typo be4736b [Cheng Lian] Report better error message when running JDBC/CLI without hive-thriftserver profile enabled

liancheng reviewed Aug 11, 2014
View reviewed changes

bin/spark-sql Outdated

Copy link

Contributor

liancheng Aug 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the extra newline.

liancheng reviewed Aug 11, 2014
View reviewed changes

liancheng mentioned this pull request Aug 12, 2014

[SPARK-2925] [sql]fix spark-sql and start-thriftserver shell bugs when set --driver-java-options #1851

Closed

liancheng mentioned this pull request Aug 14, 2014

[SPARK-3026][SQL] Report better error message when running JDBC/CLI without hive profile enabled #1944

Closed

sarutak changed the title ~~[SPARK-2964] [SQL] Wrong silent option in spark-sql script~~ [SPARK-2964] [SQL] Improve spark-sql and start-thriftserver to leverage bin/util.sh Aug 16, 2014

sarutak force-pushed the SPARK-2964 branch from 56dbdb8 to ffb68fa Compare August 23, 2014 09:13

sarutak changed the title ~~[SPARK-2964] [SQL] Fix the -S and --silent option of spark-sql, and improve spark-sql and start-thriftserver to leverage bin/util.sh~~ [SPARK-2964] [SQL] Remove duplicated code from spark-sql and start-thriftserver.sh Aug 23, 2014

sarutak added 2 commits August 26, 2014 10:30

Revert "Shorten timeout to more reasonable value"

26e7c95

This reverts commit 81b43a8.

Merge branch 'master' of git://git.apache.org/spark into SPARK-2964

8ef8751

asfgit closed this in faeb9c0 Aug 27, 2014

sarutak deleted the SPARK-2964 branch April 11, 2015 05:21

[SPARK-2964] [SQL] Remove duplicated code from spark-sql and start-thriftserver.sh #1886

[SPARK-2964] [SQL] Remove duplicated code from spark-sql and start-thriftserver.sh #1886

Uh oh!

Conversation

sarutak commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

sarutak commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

liancheng Aug 11, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng commented Aug 11, 2014

Uh oh!

sarutak commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

liancheng Aug 11, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

marmbrus commented Aug 13, 2014

Uh oh!

marmbrus commented Aug 13, 2014

Uh oh!

SparkQA commented Aug 13, 2014

Uh oh!

liancheng commented Aug 14, 2014

Uh oh!

marmbrus commented Aug 14, 2014

Uh oh!

marmbrus commented Aug 14, 2014

Uh oh!

sarutak commented Aug 16, 2014

Uh oh!

liancheng commented Aug 16, 2014

Uh oh!

sarutak commented Aug 16, 2014

Uh oh!

SparkQA commented Aug 16, 2014

Uh oh!

SparkQA commented Aug 16, 2014

Uh oh!

SparkQA commented Aug 18, 2014

Uh oh!

SparkQA commented Aug 18, 2014

Uh oh!

liancheng commented Aug 19, 2014

Uh oh!

SparkQA commented Aug 23, 2014

Uh oh!

sarutak commented Aug 23, 2014

Uh oh!

SparkQA commented Aug 23, 2014

Uh oh!

liancheng commented Aug 24, 2014

Uh oh!

sarutak commented Aug 24, 2014

Uh oh!

marmbrus commented Aug 26, 2014

Uh oh!

sarutak commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

sarutak commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!

SparkQA commented Aug 26, 2014

Uh oh!