[SPARK-9818] Re-enable Docker tests for JDBC data source #9503

JoshRosen · 2015-11-05T21:17:18Z

This patch re-enables tests for the Docker JDBC data source. These tests were reverted in #4872 due to transitive dependency conflicts introduced by the docker-client library. This patch should avoid those problems by using a version of docker-client which shades its transitive dependencies and by performing some build-magic to work around problems with that shaded JAR.

In addition, I significantly refactored the tests to simplify the setup and teardown code and to fix several Docker networking issues which caused problems when running in boot2docker.

Closes #8101.

JoshRosen · 2015-11-05T21:17:25Z

/cc @marmbrus for review.

JoshRosen · 2015-11-05T21:19:04Z

Although the tests should automatically pull any missing Docker images, I went ahead and manually pulled them on all of the AMP Jenkins workers. This uncovered an issue where two of the workers' Docker filesystems had entered a safe mode that prevented image pulling, so I logged in and fixed that.

marmbrus · 2015-11-05T21:26:10Z

sql/core/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala

I always forget your can do this...

marmbrus · 2015-11-05T21:38:11Z

For posterity, is there any easy way to do the parallel pull if we upgrade one of the images or add a new one?

JoshRosen · 2015-11-05T21:39:42Z

Here's the command that I used:

pssh -O StrictHostKeyChecking=no -h /root/jenkins_workers.txt -i -x '-t -t' "sudo docker pull mysql:5.7.9"

marmbrus · 2015-11-05T21:45:07Z

Tested this locally and other than needing to set DOCKER_IP everything worked great. Talked ot @JoshRosen offline and he's going to try and automate this.

SparkQA · 2015-11-05T21:57:11Z

Test build #1990 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-05T21:58:05Z

Test build #45145 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-05T22:01:15Z

Test build #1992 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-05T22:04:09Z

Test build #1991 has finished for PR 9503 at commit bf98ae0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

JoshRosen · 2015-11-06T00:24:33Z

sql/core/pom.xml

Woah, this is an ancient version of docker-client. I'll try using a newer release to see if that magically fixes the ASM issues.

SparkQA · 2015-11-06T00:39:13Z

Test build #45172 has finished for PR 9503 at commit 95f00e1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

SparkQA · 2015-11-06T03:12:23Z

Test build #45176 has finished for PR 9503 at commit fcf5dc4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

…sues.

SparkQA · 2015-11-06T09:23:10Z

Test build #45214 has finished for PR 9503 at commit 2273f87.

This patch fails to build.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- abstract class DatabaseOnDocker
- class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

SparkQA · 2015-11-06T09:35:30Z

Test build #45215 has finished for PR 9503 at commit 65315c4.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * abstract class DatabaseOnDocker\n * class MySQLIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.Integer\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(0).equals(\"class java.lang.Boolean\"))\n * assert(types(1).equals(\"class java.lang.Long\"))\n * assert(types(2).equals(\"class java.lang.Integer\"))\n * assert(types(3).equals(\"class java.lang.Integer\"))\n * assert(types(4).equals(\"class java.lang.Integer\"))\n * assert(types(5).equals(\"class java.lang.Long\"))\n * assert(types(6).equals(\"class java.math.BigDecimal\"))\n * assert(types(7).equals(\"class java.lang.Double\"))\n * assert(types(8).equals(\"class java.lang.Double\"))\n * assert(types(0).equals(\"class java.sql.Date\"))\n * assert(types(1).equals(\"class java.sql.Timestamp\"))\n * assert(types(2).equals(\"class java.sql.Timestamp\"))\n * assert(types(3).equals(\"class java.sql.Timestamp\"))\n * assert(types(4).equals(\"class java.sql.Date\"))\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.String\"))\n * assert(types(2).equals(\"class java.lang.String\"))\n * assert(types(3).equals(\"class java.lang.String\"))\n * assert(types(4).equals(\"class java.lang.String\"))\n * assert(types(5).equals(\"class java.lang.String\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class [B\"))\n * assert(types(8).equals(\"class [B\"))\n * class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite\n * assert(types(0).equals(\"class java.lang.String\"))\n * assert(types(1).equals(\"class java.lang.Integer\"))\n * assert(types(2).equals(\"class java.lang.Double\"))\n * assert(types(3).equals(\"class java.lang.Long\"))\n * assert(types(4).equals(\"class java.lang.Boolean\"))\n * assert(types(5).equals(\"class [B\"))\n * assert(types(6).equals(\"class [B\"))\n * assert(types(7).equals(\"class java.lang.Boolean\"))\n * assert(types(8).equals(\"class java.lang.String\"))\n * assert(types(9).equals(\"class java.lang.String\"))\n

JoshRosen · 2015-11-10T06:32:51Z

Something really weird is going on here.

Following the advice at http://stackoverflow.com/a/21876380/590203, I added the following line to DockerJDBCIntegrationSuite in order to figure out where BasicLineFormatter was being loaded from:

println(this.getClass.getClassLoader.getResource("org/apache/http/message/BasicLineFormatter.class"))

When I run this locally with

 sbt/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive-thriftserver -Phive

I get

jar:file:/Users/joshrosen/Documents/spark/lib_managed/jars/httpcore-4.2.4.jar!/org/apache/http/message/BasicLineFormatter.class

This doesn't make sense, though: show test:dependencyClasspath does not include that JAR, so I'm not quite sure how it's winding up on the classpath. If I toss in a

  this.getClass.getClassLoader.asInstanceOf[URLClassLoader].getURLs.foreach(println)

I still don't see where that JAR is coming from.

JoshRosen · 2015-11-10T07:10:29Z

In fact, httpcore-4.2.4 doesn't even show up in the -classpath argument passed to ForkMain.

I used some of the handy classpath-debugging tools in JHades to dump more information and it looks like httpcore-4.2.4 is included in the Tachyon JARs' MANIFEST.MF files. Here's the manfest from tachyon-client:

Manifest-Version: 1.0
Built-By: DreamSweet
Build-Jdk: 1.7.0_75
Class-Path: guava-14.0.1.jar commons-lang-2.4.jar commons-io-2.4.jar c
 ommons-lang3-3.0.jar curator-framework-2.1.0-incubating.jar zookeeper
 -3.4.5.jar slf4j-log4j12-1.7.2.jar log4j-1.2.15.jar mail-1.4.jar acti
 vation-1.1.jar jline-0.9.94.jar netty-3.2.2.Final.jar curator-client-
 2.1.0-incubating.jar curator-recipes-2.1.0-incubating.jar slf4j-api-1
 .7.2.jar hadoop-client-1.0.4.jar hadoop-core-1.0.4.jar xmlenc-0.52.ja
 r commons-codec-1.4.jar commons-math-2.1.jar commons-configuration-1.
 6.jar commons-collections-3.2.1.jar commons-digester-1.8.jar commons-
 beanutils-1.7.0.jar commons-beanutils-core-1.8.0.jar commons-net-1.4.
 1.jar commons-el-1.0.jar hsqldb-1.8.0.10.jar oro-2.0.8.jar jackson-ma
 pper-asl-1.0.1.jar jackson-core-asl-1.0.1.jar tachyon-common-0.8.1.ja
 r netty-all-4.0.28.Final.jar libthrift-0.9.2.jar httpclient-4.2.5.jar
  httpcore-4.2.4.jar tachyon-client-unshaded-0.8.1.jar tachyon-underfs
 -hdfs-0.8.1.jar tachyon-underfs-glusterfs-0.8.1.jar glusterfs-hadoop-
 2.3.13.jar hadoop-yarn-server-nodemanager-2.3.0.jar hadoop-yarn-commo
 n-2.3.0.jar hadoop-yarn-api-2.3.0.jar jaxb-api-2.2.2.jar stax-api-1.0
 -2.jar jettison-1.1.jar servlet-api-2.5.jar jersey-core-1.9.jar jerse
 y-client-1.9.jar jetty-util-6.1.26.jar hadoop-annotations-2.3.0.jar g
 uice-servlet-3.0.jar protobuf-java-2.5.0.jar guice-3.0.jar javax.inje
 ct-1.jar aopalliance-1.0.jar jersey-json-1.9.jar jaxb-impl-2.2.3-1.ja
 r jackson-jaxrs-1.8.3.jar jackson-xc-1.8.3.jar jersey-guice-1.9.jar h
 adoop-yarn-server-common-2.3.0.jar hadoop-common-2.3.0.jar commons-cl
 i-1.2.jar commons-math3-3.1.1.jar jetty-6.1.26.jar jersey-server-1.9.
 jar asm-3.1.jar jasper-compiler-5.5.23.jar jasper-runtime-5.5.23.jar 
 jsp-api-2.1.jar avro-1.7.4.jar paranamer-2.3.jar snappy-java-1.0.4.1.
 jar hadoop-auth-2.3.0.jar jsch-0.1.42.jar jsr305-1.3.9.jar commons-co
 mpress-1.4.1.jar xz-1.0.jar tachyon-underfs-s3-0.8.1.jar jets3t-0.8.1
 .jar commons-logging-1.1.1.jar commons-httpclient-3.1.jar java-xmlbui
 lder-0.4.jar tachyon-underfs-local-0.8.1.jar
Created-By: Apache Maven 3.3.3
Archiver-Version: Plexus Archiver

According to the Java 7 JAR specification (emphasis mine):

Class-Path: The value of this attribute specifies the relative URLs of the extensions or libraries that this application or extension needs. URLs are separated by one or more spaces. The application or extension class loader uses the value of this attribute to construct its internal search path.

Assuming that "relative" means "relative to the JAR containing the manifest", I think that what's happening is that this is causing the httpcore-4.2.4 JAR from lib_managed to be added to the classpath, which explains why testing this patch after rebasing on top of #9575 (which minimizes our use of lib_managed/jars to only include the Datanucleus JARs) seemed to fix things.

What a giant mess. It looks like the <addClassPath> manifest configuration ended up being added to Tachyon's root POM as part of Alluxio/alluxio#939.

pwendell · 2015-11-10T07:22:14Z

I only reviewed the build changes, but they look good to me.

JoshRosen · 2015-11-10T07:31:14Z

I found this Manifest class-path issue to be extremely puzzling, but it looks like other people have hit it as well: https://dzone.com/articles/jar-manifest-class-path-is-not-for-java-applicatio

Quote:

A JAR with a Class-Path manifest header will make those classpath entries available to the Java classloader regardless of whether that JAR is executed with java -jar ... or is simply placed on the classpath of a larger Java application.

This puzzling behavior is described a bit more explicitly in Java 8's JAR specification.

SparkQA · 2015-11-10T07:39:10Z

Test build #45495 has finished for PR 9503 at commit 9011bc5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-11-10T07:43:44Z

Created https://tachyon.atlassian.net/browse/TACHYON-1254 for this; this will be worked around in Spark via my lib_managed patch, but I still feel that it should be fixed in Tachyon as well so that it doesn't bite anyone in the future.

JoshRosen · 2015-11-10T08:10:30Z

And just to be 100%-sure that it was the Tachyon manifests's fault, I used the following command to delete them and verified that doing so fixed the tests:

cd $SPARK_HOME
ls -d -1 lib_managed/jars/* | grep tachyon | xargs -n1 -I{} zip {} --delete META-INF/MANIFEST.MF

JoshRosen · 2015-11-10T18:21:37Z

Jenkins, retest this please.

SparkQA · 2015-11-10T20:23:33Z

Test build #45536 has finished for PR 9503 at commit 9011bc5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-11-10T20:24:26Z

Looks like a flaky streaming test?

Jenkins, retest this please.

SparkQA · 2015-11-10T23:14:02Z

Test build #2030 has finished for PR 9503 at commit 9011bc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-11-10T23:58:21Z

I'm merging this. Thanks.

This patch re-enables tests for the Docker JDBC data source. These tests were reverted in #4872 due to transitive dependency conflicts introduced by the `docker-client` library. This patch should avoid those problems by using a version of `docker-client` which shades its transitive dependencies and by performing some build-magic to work around problems with that shaded JAR. In addition, I significantly refactored the tests to simplify the setup and teardown code and to fix several Docker networking issues which caused problems when running in `boot2docker`. Closes #8101. Author: Josh Rosen <[email protected]> Author: Yijie Shen <[email protected]> Closes #9503 from JoshRosen/docker-jdbc-tests. (cherry picked from commit 1dde39d) Signed-off-by: Reynold Xin <[email protected]>

tedyu · 2015-11-13T10:44:16Z

From https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/4095/consoleFull :

MySQLIntegrationSuite:
*** RUN ABORTED ***
  java.lang.NoSuchMethodError: org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
  at org.glassfish.jersey.apache.connector.ApacheConnector.<init>(ApacheConnector.java:240)
  at org.glassfish.jersey.apache.connector.ApacheConnectorProvider.getConnector(ApacheConnectorProvider.java:115)
  at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:418)
  at org.glassfish.jersey.client.ClientConfig$State.access$000(ClientConfig.java:88)
  at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:120)
  at org.glassfish.jersey.client.ClientConfig$State$3.get(ClientConfig.java:117)
  at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:340)
  at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:726)
  at org.glassfish.jersey.client.ClientRequest.getConfiguration(ClientRequest.java:285)
  at org.glassfish.jersey.client.JerseyInvocation.validateHttpMethodAndEntity(JerseyInvocation.java:126)

markhamstra · 2015-11-13T19:04:25Z

Similar failure with both PostgresIntegrationSuite and MySQLIntegrationSuite from dev/run-tests on OSX 10.11.1:

[info] Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.jdbc.PostgresIntegrationSuite *** ABORTED *** (3 seconds, 40 milliseconds)
[info]   com.spotify.docker.client.DockerException: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:2375 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1, localhost/fe80:0:0:0:0:0:0:1%1] failed: Connection refused
[info]   at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1141)
[info]   at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1082)
[info]   at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:281)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:76)
[info]   at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
...

JoshRosen · 2015-11-13T19:27:34Z

Maybe the docker daemon didn't get started / restarted on a subset of the machines?

tedyu · 2015-11-14T00:08:47Z

Should MySQLIntegrationSuite be disabled since the test failure is reproducible ?

JoshRosen · 2015-11-14T02:09:19Z

If you don't have Docker set up locally, you can use the DockerTest test tag to exclude those tests.

lvc · 2016-10-03T14:04:45Z

@JoshRosen

The backward compatibility reports for Jersey* libraries are now available in the API Tracker project:

https://abi-laboratory.pro/java/tracker/timeline/jersey-core/
https://abi-laboratory.pro/java/tracker/timeline/jersey-server/

The JAPICC tool detected some servlet related changes in jersey-core 1.10:
https://abi-laboratory.pro/java/tracker/compat_report/jersey-server/1.9.1/1.10/2ae0a/bin_compat_report.html

nonsleepr · 2018-05-22T21:18:41Z

Adding following for better visibility.

The problem while using com.whisk:docker-testkit-impl-spotify along with Spark:

com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
  at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2657)
  at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2588)
  at com.spotify.docker.client.DefaultDockerClient.info(DefaultDockerClient.java:522)
  ... 40 elided
Caused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
  at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
  at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
  at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
  at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2586)
  ... 41 more
Caused by: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
  at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:202)
  at org.glassfish.jersey.client.ClientRuntime.access$400(ClientRuntime.java:79)
  at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:182)
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
  at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
  at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  ... 1 more
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
  at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:74)
  at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:59)
  at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:43)
  at jnr.ffi.LibraryLoader.load(LibraryLoader.java:287)
  at jnr.unixsocket.Native.<clinit>(Native.java:76)
  at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:68)
  at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:49)
  at com.spotify.docker.client.ApacheUnixSocket.<init>(ApacheUnixSocket.java:59)
  at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:67)
  at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:118)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
  at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
  at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
  at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
  at org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
  at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
  at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
  at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
  at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
  at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
  ... 12 more

The solution is to use shaded package:

libraryDependencies += "com.spotify" % "docker-client" % "8.11.3" classifier "shaded"

yjshen and others added 8 commits October 18, 2015 18:00

Import yjshen's PR.

4b82a2c

Use newer shaded docker-client artifact

eddcc47

Use map for setting environment variables.

8bb62ea

Ensure that Docker container is cleaned up after error.

23443a5

Remove a bunch of layers of indirection / abstraction.

e6c7709

Fix networking for boot2docker.

d06847b

Log warning in case container cleanup fails

93bcf45

Rename class and freeze image versions

bf98ae0

JoshRosen mentioned this pull request Nov 5, 2015

[SPARK-9818][SQL][WIP]Revert SPARK-6136 to enable jdbc tests using docker #8101

Closed

marmbrus reviewed Nov 5, 2015
View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala Outdated

Copy link

Contributor

marmbrus Nov 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I always forget your can do this...

Automatically get docker IP from docker-machine and boot2docker.

1e389e2

Print nicer message if Docker connection fails.

d9c37df

Add test tag for excluding Docker tests in SBT.

95f00e1

JoshRosen reviewed Nov 6, 2015
View reviewed changes

Upgrade to non-ancient version of docker-client.

fcf5dc4

Upgrade Jersey to avoid ASM incompatibilities / classpath ordering is…

2273f87

…sues.

Merge remote-tracking branch 'origin/master' into docker-jdbc-tests

65315c4

Merge remote-tracking branch 'origin/master' into docker-jdbc-tests

9011bc5

JoshRosen mentioned this pull request Nov 10, 2015

Migrate Spark SQL JDBC data source integration tests here databricks/spark-integration-tests#6

Closed

asfgit closed this in 1dde39d Nov 11, 2015

This was referenced Nov 11, 2015

[SPARK-10186][SQL][WIP] Array types using JDBCRDD and postgres #9137

Closed

[SPARK-10170] [SQL] Add DB2 JDBC dialect support. #8393

Closed

[SPARK-10655][SQL] Adding additional data type mappings to jdbc DB2dialect. #9162

Closed

JoshRosen deleted the docker-jdbc-tests branch April 20, 2016 23:58

[SPARK-9818] Re-enable Docker tests for JDBC data source #9503

[SPARK-9818] Re-enable Docker tests for JDBC data source #9503

Uh oh!

Conversation

JoshRosen commented Nov 5, 2015

Uh oh!

JoshRosen commented Nov 5, 2015

Uh oh!

JoshRosen commented Nov 5, 2015

Uh oh!

marmbrus Nov 5, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Nov 5, 2015

Uh oh!

JoshRosen commented Nov 5, 2015

Uh oh!

marmbrus commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

JoshRosen Nov 6, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

pwendell commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

JoshRosen commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

rxin commented Nov 10, 2015

Uh oh!

tedyu commented Nov 13, 2015

Uh oh!

markhamstra commented Nov 13, 2015

Uh oh!

JoshRosen commented Nov 13, 2015

Uh oh!

tedyu commented Nov 14, 2015

Uh oh!

JoshRosen commented Nov 14, 2015

Uh oh!

lvc commented Oct 3, 2016

Uh oh!

nonsleepr commented May 22, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects