Skip to content

[SPARK-6136] [SQL] Removed JDBC integration tests which depends on docker-client #4872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-6136] [SQL] Removed JDBC integration tests which depends on docker-client #4872

wants to merge 2 commits into from

Conversation

@liancheng
Copy link
Contributor

Integration test suites in the JDBC data source (MySQLIntegration and PostgresIntegration) depend on docker-client 2.7.5, which transitively depends on Guava 17.0. Unfortunately, Guava 17.0 is causing test runtime binary compatibility issues when Spark is compiled against Hive 0.12.0, or Hadoop 2.4.

Considering MySQLIntegration and PostgresIntegration are ignored right now, I'd suggest moving them from the Spark project to the Spark integration tests project. This PR removes both the JDBC data source integration tests and the docker-client test dependency.

Review on Reviewable

@SparkQA
Copy link

SparkQA commented Mar 3, 2015

Test build #28229 has started for PR 4872 at commit 159b24a.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Mar 3, 2015

Test build #28229 has finished for PR 4872 at commit 159b24a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28229/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Mar 3, 2015

Test build #28230 has started for PR 4872 at commit 1f4169e.

  • This patch merges cleanly.

@srowen
Copy link
Member

srowen commented Mar 3, 2015

Isn't it a little extreme to remove the tests? what about just excluding the Guava dep so that 14.0 is used? it may just work

@SparkQA
Copy link

SparkQA commented Mar 3, 2015

Test build #28230 has finished for PR 4872 at commit 1f4169e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28230/
Test PASSed.

@liancheng
Copy link
Contributor Author

@srowen These two test suites do a bunch of time consuming jobs like setting up docker containers, pulling in MySQL/PostgreSQL, etc., and are thus ignored from the very beginning. (You may see the @Ignore annotations.) I've put moving them to the Spark integration tests project on my todo list. I believe that's a better place for us to run these complex integration tests, especially because that project has already been well dockerized.

@liancheng
Copy link
Contributor Author

/cc @marmbrus Just to double check, is it OK to remove them?

@srowen
Copy link
Member

srowen commented Mar 3, 2015

@liancheng Ah I looked right past the @Ignore. Well, I tend to agree with your logic. You guys know better than I would.

@marmbrus
Copy link
Contributor

marmbrus commented Mar 3, 2015

+1 to moving these to integration tests, especially if they are causing build problems. Can you make sure there is a JIRA somewhere (where?) to make sure we don't forget though. These tests actually did find all kinds of quirks with the various drivers.

@liancheng
Copy link
Contributor Author

@marmbrus @srowen Filed SPARK-6147 to track this. Thanks for the comments!

asfgit pushed a commit that referenced this pull request Mar 4, 2015
@liancheng
[SPARK-6136] [SQL] Removed JDBC integration tests which depends on do…
035243d 
…cker-client

Integration test suites in the JDBC data source (`MySQLIntegration` and `PostgresIntegration`) depend on docker-client 2.7.5, which transitively depends on Guava 17.0. Unfortunately, Guava 17.0 is causing test runtime binary compatibility issues when Spark is compiled against Hive 0.12.0, or Hadoop 2.4.

Considering `MySQLIntegration` and `PostgresIntegration` are ignored right now, I'd suggest moving them from the Spark project to the [Spark integration tests] [1] project. This PR removes both the JDBC data source integration tests and the docker-client test dependency.

[1]: |https://github.com/databricks/spark-integration-tests

<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/4872)
<!-- Reviewable:end -->

Author: Cheng Lian <[email protected]>

Closes #4872 from liancheng/remove-docker-client and squashes the following commits:

1f4169e [Cheng Lian] Removes DockerHacks
159b24a [Cheng Lian] Removed JDBC integration tests which depends on docker-client

(cherry picked from commit 76b472f)
Signed-off-by: Cheng Lian <[email protected]>
@asfgit asfgit closed this in 76b472f Mar 4, 2015
@liancheng liancheng deleted the remove-docker-client branch March 4, 2015 12:27
@liancheng liancheng mentioned this pull request Mar 5, 2015
Closed
This was referenced Mar 14, 2015
Closed
Closed
@JoshRosen JoshRosen mentioned this pull request Nov 5, 2015
Closed
asfgit pushed a commit that referenced this pull request Nov 10, 2015
@JoshRosen @rxin
[SPARK-9818] Re-enable Docker tests for JDBC data source
d0828d2 
This patch re-enables tests for the Docker JDBC data source. These tests were reverted in #4872 due to transitive dependency conflicts introduced by the `docker-client` library. This patch should avoid those problems by using a version of `docker-client` which shades its transitive dependencies and by performing some build-magic to work around problems with that shaded JAR.

In addition, I significantly refactored the tests to simplify the setup and teardown code and to fix several Docker networking issues which caused problems when running in `boot2docker`.

Closes #8101.

Author: Josh Rosen <[email protected]>
Author: Yijie Shen <[email protected]>

Closes #9503 from JoshRosen/docker-jdbc-tests.

(cherry picked from commit 1dde39d)
Signed-off-by: Reynold Xin <[email protected]>
asfgit pushed a commit that referenced this pull request Nov 11, 2015
@JoshRosen @rxin
[SPARK-9818] Re-enable Docker tests for JDBC data source
1dde39d 
This patch re-enables tests for the Docker JDBC data source. These tests were reverted in #4872 due to transitive dependency conflicts introduced by the `docker-client` library. This patch should avoid those problems by using a version of `docker-client` which shades its transitive dependencies and by performing some build-magic to work around problems with that shaded JAR.

In addition, I significantly refactored the tests to simplify the setup and teardown code and to fix several Docker networking issues which caused problems when running in `boot2docker`.

Closes #8101.

Author: Josh Rosen <[email protected]>
Author: Yijie Shen <[email protected]>

Closes #9503 from JoshRosen/docker-jdbc-tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@liancheng @SparkQA @AmplabJenkins @srowen @marmbrus