[SPARK-9818][SQL][WIP]Revert SPARK-6136 to enable jdbc tests using docker #8101

yjshen · 2015-08-11T10:03:26Z

It's just working in progress, not ready to be reviewed.

I'd like to see if this breaks something.

SparkQA · 2015-08-11T10:12:40Z

Test build #40441 has finished for PR 8101 at commit 9906e31.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MySQLDatabase
- class MySQLIntegration extends FunSuite with BeforeAndAfterAll
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresDatabase
- class PostgresIntegration extends FunSuite with BeforeAndAfterAll
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

SparkQA · 2015-08-11T11:11:25Z

Test build #40444 has finished for PR 8101 at commit a122c6d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MySQLDatabase extends Logging
- class MySQLIntegration extends SparkFunSuite with BeforeAndAfterAll
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresDatabase extends Logging
- class PostgresIntegration extends SparkFunSuite with BeforeAndAfterAll
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

yjshen · 2015-08-11T11:38:17Z

Jenkins, retest this please.

SparkQA · 2015-08-11T12:19:23Z

Test build #40448 has finished for PR 8101 at commit a122c6d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MySQLDatabase extends Logging
- class MySQLIntegration extends SparkFunSuite with BeforeAndAfterAll
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresDatabase extends Logging
- class PostgresIntegration extends SparkFunSuite with BeforeAndAfterAll
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

JoshRosen · 2015-08-11T16:28:43Z

I think that our Jenkins actually has Docker installed now, so in principle there should be a way to automate these tests once we get them working.

SparkQA · 2015-08-12T16:05:24Z

Test build #40632 has finished for PR 8101 at commit 8c91d7c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2015-08-12T17:31:17Z

sql/core/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegration.scala

nit: MySQLIntegrationSuite

marmbrus · 2015-08-12T17:35:18Z

Thanks for working on this! I'd love to get these tests running again.

SparkQA · 2015-08-12T18:34:57Z

Test build #40649 has finished for PR 8101 at commit aed4162.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yjshen · 2015-08-13T01:48:57Z

Image pull failed: mysql: ProgressMessage{id=null, status=null, stream=null, error=Error pulling image (5.6.17) from mysql,
Get https://registry-1.docker.io/v1/images/36e732ca26106f3e286b9f9500ef17803430d3b5a044d25c64b66988fd198d59/ancestry:
read tcp 54.208.162.63:443: connection reset by peer, progress=null, progressDetail=null}

SparkQA · 2015-08-13T03:33:21Z

Test build #40725 has finished for PR 8101 at commit aed4162.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

yjshen · 2015-08-13T03:35:08Z

Jenkins, retest this please.

SparkQA · 2015-08-13T05:50:54Z

Test build #40727 has finished for PR 8101 at commit aed4162.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yjshen · 2015-08-13T09:06:27Z

Jenkins, retest this please.

SparkQA · 2015-08-13T09:42:11Z

Test build #40751 has finished for PR 8101 at commit aed4162.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yjshen · 2015-08-13T09:44:48Z

pull mysql docker image would sometimes fail the build:

Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.jdbc.MySQLIntegrationSuite *** ABORTED *** (2 minutes, 33 seconds)
[info]   com.spotify.docker.client.ImagePullFailedException: Image pull failed: mysql: ProgressMessage{id=null, status=null, stream=null, error=Error pulling image (5.6.22) from mysql, Mktemp failed: mkdir /var/lib/docker/graph/_tmp/aecd3ebbe9b9645d4ed3c6420348d11ef4e2c166558513178336844f607a027f: no space left on device, progress=null, progressDetail=null}

marmbrus · 2015-08-13T16:41:54Z

Can we retry? Only do it if it doesn't exist?
On Aug 13, 2015 2:45 AM, "Yijie Shen" [email protected] wrote:

pull mysql docker image would sometimes fail:

Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.jdbc.MySQLIntegrationSuite *** ABORTED *** (2 minutes, 33 seconds)
[info] com.spotify.docker.client.ImagePullFailedException: Image pull failed: mysql: ProgressMessage{id=null, status=null, stream=null, error=Error pulling image (5.6.22) from mysql, Mktemp failed: mkdir /var/lib/docker/graph/_tmp/aecd3ebbe9b9645d4ed3c6420348d11ef4e2c166558513178336844f607a027f: no space left on device, progress=null, progressDetail=null}

—
Reply to this email directly or view it on GitHub
#8101 (comment).

SparkQA · 2015-08-14T10:24:35Z

Test build #40861 has finished for PR 8101 at commit 8f1bd84.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-14T11:35:07Z

Test build #40865 has finished for PR 8101 at commit 50e0e84.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-14T15:08:59Z

Test build #40878 has finished for PR 8101 at commit 20f65a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-15T06:17:43Z

Test build #40945 has finished for PR 8101 at commit 20f65a1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-15T06:44:45Z

Test build #40943 has finished for PR 8101 at commit 20f65a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-15T13:47:39Z

Test build #40957 has finished for PR 8101 at commit de10e94.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-02T09:40:07Z

Test build #41921 has finished for PR 8101 at commit fbc471b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class DatabaseOnDocker
- class MySQLIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

SparkQA · 2015-09-02T09:45:20Z

Test build #41926 has finished for PR 8101 at commit fbc471b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DCT(JavaTransformer, HasInputCol, HasOutputCol):
- class SQLTransformer(JavaTransformer):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

yjshen · 2015-09-02T10:27:38Z

Pull mysql image would sometimes fail even with 5 retries...

rxin · 2015-09-03T22:56:24Z

cc @marmbrus what do you think? Is there a way we can preload the images on Jenkins?

marmbrus · 2015-09-03T22:59:26Z

Where are the errors with pulling the image? com.spotify.docker.client.DockerRequestException: Request error: POST unix://localhost:80/v1.12/containers/create: 500 usually indicates a port conflict, not a problem pulling the image.

yjshen · 2015-09-08T09:43:56Z

Sorry for my late reply.
MySQLIntegrationSuite would fail with:

Error Message

Image pull failed: mysql:latest: ProgressMessage{id=null, status=null, stream=null, error=Error pulling image (latest) from mysql, Driver devicemapper failed to create image rootfs 9d3ceacde91b4c7d6c1275032adb558d668afd5489c007f3512b39793ddf992d: Error running DeviceCreate (createSnapDevice) dm_task_run failed, progress=null, progressDetail=null}
Stacktrace

sbt.ForkMain$ForkError: Image pull failed: mysql:latest: ProgressMessage{id=null, status=null, stream=null, error=Error pulling image (latest) from mysql, Driver devicemapper failed to create image rootfs 9d3ceacde91b4c7d6c1275032adb558d668afd5489c007f3512b39793ddf992d: Error running DeviceCreate (createSnapDevice) dm_task_run failed, progress=null, progressDetail=null}
    at com.spotify.docker.client.LoggingPullHandler.progress(LoggingPullHandler.java:45)
...

PostgresIntegrationSuite would fail with:

Error Message

Request error: POST unix://localhost:80/v1.12/containers/create: 500
Stacktrace

sbt.ForkMain$ForkError: Request error: POST unix://localhost:80/v1.12/containers/create: 500
    at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:907)
...

yjshen · 2015-09-15T07:21:35Z

After exploring all the build failure above, Mysql test only fails with Image pull failed, Postgres test would fail due to create container or pull image.

yjshen · 2015-09-15T07:22:39Z

I'll trigger more test build now.

yjshen · 2015-09-15T07:22:59Z

Jenkins, retest this please.

yjshen · 2015-09-15T07:36:04Z

Jenkins, retest this please.

yjshen · 2015-09-15T08:06:11Z

Jenkins, retest this please.

yjshen · 2015-09-15T08:24:27Z

Jenkins, retest this please.

SparkQA · 2015-09-15T08:35:13Z

Test build #42476 has finished for PR 8101 at commit fbc471b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class DatabaseOnDocker
- class MySQLIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

SparkQA · 2015-09-15T09:21:01Z

Test build #42473 has finished for PR 8101 at commit fbc471b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class DatabaseOnDocker
- class MySQLIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

SparkQA · 2015-09-15T09:27:17Z

Test build #42474 has finished for PR 8101 at commit fbc471b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class DatabaseOnDocker
- class MySQLIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

SparkQA · 2015-09-15T10:21:43Z

Test build #42480 has finished for PR 8101 at commit fbc471b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class DatabaseOnDocker
- class MySQLIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.Integer"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(0).equals("class java.lang.Boolean"))
- assert(types(1).equals("class java.lang.Long"))
- assert(types(2).equals("class java.lang.Integer"))
- assert(types(3).equals("class java.lang.Integer"))
- assert(types(4).equals("class java.lang.Integer"))
- assert(types(5).equals("class java.lang.Long"))
- assert(types(6).equals("class java.math.BigDecimal"))
- assert(types(7).equals("class java.lang.Double"))
- assert(types(8).equals("class java.lang.Double"))
- assert(types(0).equals("class java.sql.Date"))
- assert(types(1).equals("class java.sql.Timestamp"))
- assert(types(2).equals("class java.sql.Timestamp"))
- assert(types(3).equals("class java.sql.Timestamp"))
- assert(types(4).equals("class java.sql.Date"))
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.String"))
- assert(types(2).equals("class java.lang.String"))
- assert(types(3).equals("class java.lang.String"))
- assert(types(4).equals("class java.lang.String"))
- assert(types(5).equals("class java.lang.String"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class [B"))
- assert(types(8).equals("class [B"))
- class PostgresIntegrationSuite extends DatabaseIntegrationSuite
- assert(types(0).equals("class java.lang.String"))
- assert(types(1).equals("class java.lang.Integer"))
- assert(types(2).equals("class java.lang.Double"))
- assert(types(3).equals("class java.lang.Long"))
- assert(types(4).equals("class java.lang.Boolean"))
- assert(types(5).equals("class [B"))
- assert(types(6).equals("class [B"))
- assert(types(7).equals("class java.lang.Boolean"))
- assert(types(8).equals("class java.lang.String"))
- assert(types(9).equals("class java.lang.String"))

yjshen · 2015-09-15T13:39:29Z

It seems when test build triggered for multiple times in a row, only the first test build would fail and all subsequent builds success.

JoshRosen · 2015-10-01T00:04:32Z

I'd like to help get this patch merged so that we can re-run these tests, but before we put a lot more work into polishing it I would like to figure out whether our regular Jenkins build is the right place to run these tests.

These tests are fairly heavyweight to run (since they have to download a bunch of Docker images) and might add significant amounts of time to the test runs. Therefore, I think we should skip them for the majority of pull requests and should only run them when JDBC-related code is changed (maybe just code under the jdbc/ directory).

Similarly, we may not want to run these tests as part of every master build. Over time, I imagine that we'll accumulate larger-and-larger regression test suites here and they're going to add a huge cost for most builds. Therefore, I think that we should try to tag these suites so that they're not run as part of most master builds but are only run once per day or something like that.

JoshRosen · 2015-10-16T19:55:59Z

sql/core/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala

One concern: this has the potential to become flaky if the image is updated. I'd prefer to freeze a particular version here, for the same reason that I like things like Maven and pip --freeze.

+1, at least we should keep on the same version of the rdbms in question (as sometimes there are tweaks, enhancements on the docker image itself)

JoshRosen · 2015-10-21T20:25:57Z

sql/core/src/test/scala/org/apache/spark/sql/jdbc/DockerHacks.scala

Hey, so what's the deal with this? Is this factory still necessary or was it an artifact of legacy Docker bugs or docker-client issues?

JoshRosen · 2015-11-05T21:19:41Z

Hi @yjshen,

I've opened #9503 to re-enable these Docker tests.

This patch re-enables tests for the Docker JDBC data source. These tests were reverted in #4872 due to transitive dependency conflicts introduced by the `docker-client` library. This patch should avoid those problems by using a version of `docker-client` which shades its transitive dependencies and by performing some build-magic to work around problems with that shaded JAR. In addition, I significantly refactored the tests to simplify the setup and teardown code and to fix several Docker networking issues which caused problems when running in `boot2docker`. Closes #8101. Author: Josh Rosen <[email protected]> Author: Yijie Shen <[email protected]> Closes #9503 from JoshRosen/docker-jdbc-tests. (cherry picked from commit 1dde39d) Signed-off-by: Reynold Xin <[email protected]>

yjshen force-pushed the revert-6136 branch from 5a77040 to 9906e31 Compare August 11, 2015 10:04

yjshen force-pushed the revert-6136 branch from a122c6d to 8c91d7c Compare August 12, 2015 14:06

marmbrus reviewed Aug 12, 2015
View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegration.scala Outdated

Copy link

Contributor

marmbrus Aug 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: MySQLIntegrationSuite

yjshen force-pushed the revert-6136 branch from 92f1b74 to ab30dfd Compare August 12, 2015 17:49

yjshen force-pushed the revert-6136 branch from 8f1bd84 to 50e0e84 Compare August 14, 2015 11:01

yjshen force-pushed the revert-6136 branch from 50e0e84 to 20f65a1 Compare August 14, 2015 12:10

JoshRosen reviewed Oct 16, 2015
View reviewed changes

mariusvniekerk mentioned this pull request Oct 21, 2015

[SPARK-10186][SQL][WIP] Array types using JDBCRDD and postgres #9137

Closed

JoshRosen reviewed Oct 21, 2015
View reviewed changes

JoshRosen mentioned this pull request Nov 5, 2015

[SPARK-9818] Re-enable Docker tests for JDBC data source #9503

Closed

asfgit closed this in 1dde39d Nov 11, 2015

[SPARK-9818][SQL][WIP]Revert SPARK-6136 to enable jdbc tests using docker #8101

[SPARK-9818][SQL][WIP]Revert SPARK-6136 to enable jdbc tests using docker #8101

Uh oh!

Conversation

yjshen commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

yjshen commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

JoshRosen commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 12, 2015

Uh oh!

marmbrus Aug 12, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Aug 12, 2015

Uh oh!

SparkQA commented Aug 12, 2015

Uh oh!

yjshen commented Aug 13, 2015

Uh oh!

SparkQA commented Aug 13, 2015

Uh oh!

yjshen commented Aug 13, 2015

Uh oh!

SparkQA commented Aug 13, 2015

Uh oh!

yjshen commented Aug 13, 2015

Uh oh!

SparkQA commented Aug 13, 2015

Uh oh!

yjshen commented Aug 13, 2015

Uh oh!

marmbrus commented Aug 13, 2015

Uh oh!

SparkQA commented Aug 14, 2015

Uh oh!

SparkQA commented Aug 14, 2015

Uh oh!

SparkQA commented Aug 14, 2015

Uh oh!

SparkQA commented Aug 15, 2015

Uh oh!

SparkQA commented Aug 15, 2015

Uh oh!

SparkQA commented Aug 15, 2015

Uh oh!

SparkQA commented Sep 2, 2015

Uh oh!

SparkQA commented Sep 2, 2015

Uh oh!

yjshen commented Sep 2, 2015

Uh oh!

rxin commented Sep 3, 2015

Uh oh!

marmbrus commented Sep 3, 2015

Uh oh!

yjshen commented Sep 8, 2015

Uh oh!

yjshen commented Sep 15, 2015

Uh oh!

yjshen commented Sep 15, 2015

Uh oh!

yjshen commented Sep 15, 2015

Uh oh!

yjshen commented Sep 15, 2015

Uh oh!

yjshen commented Sep 15, 2015

Uh oh!

yjshen commented Sep 15, 2015

Uh oh!

SparkQA commented Sep 15, 2015

Uh oh!

SparkQA commented Sep 15, 2015