[SPARK-11796] [test-maven] [WIP] Fixing httpclient and httpcore depedency issues related to docker-client #9876

markgrover · 2015-11-21T00:10:53Z

I was able to get past this error:

*** RUN ABORTED ***
  java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
  at org.glassfish.jersey.apache.connector.ApacheConnector.<init>(ApacheConnector.java:240)
  at org.glassfish.jersey.apache.connector.ApacheConnectorProvider.getConnector(ApacheConnectorProvider.java:115)
  at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:418)
  at org.glassfish.jersey.client.ClientConfig$State.access$000(ClientConfig.java:88)
  at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:120)
  at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:117)
  at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:340)
  at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:726)
  at org.glassfish.jersey.client.ClientRequest.getConfiguration(ClientRequest.java:285)
  at org.glassfish.jersey.client.JerseyInvocation.validateHttpMethodAndEntity(JerseyInvocation.java:126)
  ...

However, then I ran into some class not found errors like these:

*** RUN ABORTED ***
  java.lang.NoClassDefFoundError: org/apache/http/ssl/SSLContexts
  at org.apache.http.conn.ssl.SSLConnectionSocketFactory.getSocketFactory(SSLConnectionSocketFactory.java:172)
  at com.spotify.docker.client.DefaultDockerClient.getSchemeRegistry(DefaultDockerClient.java:254)
  at com.spotify.docker.client.DefaultDockerClient.getConnectionManager(DefaultDockerClient.java:242)
  at com.spotify.docker.client.DefaultDockerClient.<init>(DefaultDockerClient.java:205)
  at com.spotify.docker.client.DefaultDockerClient$Builder.build(DefaultDockerClient.java:1370)
  at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:73)
  at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
  at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:58)
  at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
  at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.run(DockerJDBCIntegrationSuite.scala:58)
  ...
  Cause: java.lang.ClassNotFoundException: org.apache.http.ssl.SSLContexts
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  at org.apache.http.conn.ssl.SSLConnectionSocketFactory.getSocketFactory(SSLConnectionSocketFactory.java:172)
  at com.spotify.docker.client.DefaultDockerClient.getSchemeRegistry(DefaultDockerClient.java:254)
  at com.spotify.docker.client.DefaultDockerClient.getConnectionManager(DefaultDockerClient.java:242)

Anyways, finally after trying a few more things and going through @JoshRosen 's previous discussion with docker-client folks, this patch is what I finally came up with.

It goes past all the ClassNotFound errors but on my mac, it gives 404 errors. So, I am trying to post the patch here, in case it doesn't pass on my local environment.

markgrover · 2015-11-21T00:11:04Z

Jenkins, test this please.

markgrover · 2015-11-21T00:11:57Z

Josh, it would be good test this after removing the disabling of the test in Jenkins. I am happy to try that out but I am not sure I have the right privileges on Jenkins. Feedback/thoughts welcome.

JoshRosen · 2015-11-21T00:20:04Z

@mark, put [test-maven] into your PR title and it'll run through the Maven PRB where that disabling is not in effect.

JoshRosen · 2015-11-21T00:28:53Z

Jenkins, this is ok to test.

SparkQA · 2015-11-21T02:28:51Z

Test build #46462 has finished for PR 9876 at commit 6dfff89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-11-21T02:30:31Z

Jenkins, retest this please.

SparkQA · 2015-11-21T04:15:59Z

Test build #46467 has finished for PR 9876 at commit 6dfff89.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

markgrover · 2015-11-21T04:29:28Z

Thanks Josh. So, I have a good news and bad news.

The good news is that the maven/dependency related issues have been resolved. The bad news is that the tests are still failing, albeit due to MySQL and PostgreSQL drivers not being accessible.

The error is pasted here for easier access:

MySQLIntegrationSuite:
Exception encountered when invoking run on a nested suite - The code passed to eventually never returned normally. Attempted 70 times over 1.0002781788666666 minutes. Last failure message: No suitable driver found for jdbc:mysql://192.168.10.28:52440/mysql?user=root&password=rootpass. *** ABORTED ***
  Exception encountered when invoking run on a nested suite - The code passed to eventually never returned normally. Attempted 70 times over 1.0002781788666666 minutes. Last failure message: No suitable driver found for jdbc:mysql://192.168.10.28:52440/mysql?user=root&password=rootpass. (DockerJDBCIntegrationSuite.scala:115)
PostgresIntegrationSuite:
Exception encountered when invoking run on a nested suite - The code passed to eventually never returned normally. Attempted 70 times over 1.00021882435 minutes. Last failure message: No suitable driver found for jdbc:postgresql://192.168.10.28:39460/postgres?user=postgres&password=rootpass. *** ABORTED ***
  Exception encountered when invoking run on a nested suite - The code passed to eventually never returned normally. Attempted 70 times over 1.00021882435 minutes. Last failure message: No suitable driver found for jdbc:postgresql://192.168.10.28:39460/postgres?user=postgres&password=rootpass. (DockerJDBCIntegrationSuite.scala:115)
Run completed in 2 minutes, 46 seconds.
Total number of tests run: 0
Suites: completed 1, aborted 2
Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
*** 2 SUITES ABORTED ***

JoshRosen · 2015-11-21T06:12:01Z

My hunch is that you'll have to declare an explicit dependency on those drivers, since I think that there's a difference between Maven and SBT handling of test-jars' transitive dependencies. See sql/pom.xml for the drivers; if the versions are just hardcoded there, you might want to lift them up to the root pom's dependencyManagement when also declaring the dep. in the docker subproject.

markgrover · 2015-11-21T15:15:00Z

OK, I will do that. I am afk for some time but will take care of this when
I am back. Thanks.
On Nov 20, 2015 10:12 PM, "Josh Rosen" [email protected] wrote:

My hunch is that you'll have to declare an explicit dependency on those
drivers, since I think that there's a difference between Maven and SBT
handling of test-jars' transitive dependencies. See sql/pom.xml for the
drivers; if the versions are just hardcoded there, you might want to lift
them up to the root pom's dependencyManagement when also declaring the dep.
in the docker subproject.

—
Reply to this email directly or view it on GitHub
#9876 (comment).

srowen · 2015-11-22T13:01:04Z

Should the client/core versions match up? and be declared as one global version?
I think using the latest of these is generally a Good Thing, but I know so many components rely on different old versions of the library. If it works, great. We probably still need to audit mvn -Dverbose dependency:tree to see what other versions still come in, and what components have had their version bumped, and sanity-check that.

JoshRosen · 2015-11-23T21:10:37Z

FWIW it would probably be easier to produce a new release of the Docker client which properly relocates and shades dependencies; see my comments + discussion at spotify/docker-client#272.

In the meantime, though, doing dependency hacks that are isolated to this Maven subproject seems fine to me.

… tests. Also, adding the same driver depedencies in dependency management section of root pom so they can be shared amongst various modules (e.g. sql)

markgrover · 2015-12-09T03:03:36Z

Thanks @JoshRosen For now, I have added the mysql and postgresql drivers as dependencies and the tests pass now. We just have to make sure that the CI environment has the docker engine and environment set up properly.

Medium term, I agree about new release of Docker client being the right solution.

markgrover · 2015-12-09T03:03:46Z

Jenkins, retest this please.

SparkQA · 2015-12-09T03:41:37Z

Test build #47399 has finished for PR 9876 at commit 73a3182.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * public class JavaIndexToStringExample\n

markgrover · 2015-12-09T23:18:42Z

The failure seems unrelated to my change, so I will ask Jenkins to retest. Also, I am not sure if I have the right permissions to trigger a new build and I'd appreciate if someone could add me to the whitelist.

markgrover · 2015-12-09T23:18:47Z

Jenkins, retest this please.

srowen · 2015-12-09T23:26:07Z

Jenkins add to whitelist

markgrover · 2015-12-10T00:24:39Z

Thanks, @srowen!

SparkQA · 2015-12-10T01:52:57Z

Test build #47456 has finished for PR 9876 at commit 73a3182.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-10T01:58:16Z

LGTM. Thanks @markgrover!

JoshRosen · 2015-12-10T02:36:14Z

Tested locally and confirmed that these tests still run fine in SBT, so I'm going to merge this to master and branch-1.6.

JoshRosen · 2015-12-10T02:37:18Z

I'm going to update the PR title and description in the final commit so that they're no longer WIP / so verbose.

… docker-client This commit fixes dependency issues which prevented the Docker-based JDBC integration tests from running in the Maven build. Author: Mark Grover <[email protected]> Closes #9876 from markgrover/master_docker. (cherry picked from commit 2166c2a) Signed-off-by: Josh Rosen <[email protected]>

markgrover · 2015-12-10T02:48:17Z

Thanks so much, Josh!

blbradley · 2016-01-24T04:17:59Z

@srowen httpclient 4.5.* depends on versions of httpcore 4.4.*

Fixing httpclient and httpcore depedency issues related to docker-client

6dfff89

markgrover changed the title ~~[SPARK-11796] [WIP] Fixing httpclient and httpcore depedency issues related to docker-client~~ [SPARK-11796] [test-maven] [WIP] Fixing httpclient and httpcore depedency issues related to docker-client Nov 21, 2015

Mark Grover added 2 commits December 8, 2015 18:59

Merge branch 'master' into master_docker

bc9f4c7

Adding MySQL and PostgreSQL drivers as dependencies for use in docker…

73a3182

… tests. Also, adding the same driver depedencies in dependency management section of root pom so they can be shared amongst various modules (e.g. sql)

asfgit closed this in 2166c2a Dec 10, 2015

markgrover deleted the master_docker branch December 10, 2015 07:16

blbradley mentioned this pull request Jan 24, 2016

[SPARK-12972][CORE] update httpclient to 4.5.1 #10887

Closed

[SPARK-11796] [test-maven] [WIP] Fixing httpclient and httpcore depedency issues related to docker-client #9876

[SPARK-11796] [test-maven] [WIP] Fixing httpclient and httpcore depedency issues related to docker-client #9876

Uh oh!

Conversation

markgrover commented Nov 21, 2015

Uh oh!

markgrover commented Nov 21, 2015

Uh oh!

markgrover commented Nov 21, 2015

Uh oh!

JoshRosen commented Nov 21, 2015

Uh oh!

JoshRosen commented Nov 21, 2015

Uh oh!

SparkQA commented Nov 21, 2015

Uh oh!

JoshRosen commented Nov 21, 2015

Uh oh!

SparkQA commented Nov 21, 2015

Uh oh!

markgrover commented Nov 21, 2015

Uh oh!

JoshRosen commented Nov 21, 2015

Uh oh!

markgrover commented Nov 21, 2015

Uh oh!

srowen commented Nov 22, 2015

Uh oh!

JoshRosen commented Nov 23, 2015

Uh oh!

markgrover commented Dec 9, 2015

Uh oh!

markgrover commented Dec 9, 2015

Uh oh!

SparkQA commented Dec 9, 2015

Uh oh!

markgrover commented Dec 9, 2015

Uh oh!

markgrover commented Dec 9, 2015

Uh oh!

srowen commented Dec 9, 2015

Uh oh!

markgrover commented Dec 10, 2015

Uh oh!

SparkQA commented Dec 10, 2015

Uh oh!

JoshRosen commented Dec 10, 2015

Uh oh!

JoshRosen commented Dec 10, 2015

Uh oh!

JoshRosen commented Dec 10, 2015

Uh oh!

markgrover commented Dec 10, 2015

Uh oh!

blbradley commented Jan 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants