Test aut with Apache Spark 2.4.0

[ruebot]

Spark 2.4.0 came out in November 2018. We should put aut a few tests with it.

1. Update pom.xml to use Apache Spark 2.4.0
2. Update pom.xml to use Scala 2.11.12
3. Test all examples here
4. Test examples slated for 0.18.0 release
5. Test with AUK

Did I miss anything?

[ruebot]

Branch: https://github.com/archivesunleashed/aut/tree/spark-2.4.0

[ianmilligan1]

Tested with Spark 2.4.0 on all the examples both currently live on the docs as well as those on the 0.18.0 branch. All checked out.

[ruebot]

That's great! Thank you!!

I'll test again with AUK. I did it on Monday, and didn't run into this. Trying to figure out if that made it into the release or not. Because the release notes have it as a known issue, but it worked on Monday. Maybe it's more of an FYI 🤷‍♂️

...anyway. Testing again!

[ruebot]

Apache Spark 2.4.0 + aut 0.17.0 in AUK works:

/home/nruest/bin/spark-2.4.0-bin-hadoop2.7/bin/spark-shell --master local\[10\] --driver-memory 30G --conf spark.network.timeout=10000000 --conf spark.executor.heartbeatInterval=600s --conf spark.driver.maxResultSize=4G --packages "io.archivesunleashed:aut:0.17.0" -i /home/nruest/tmp/auk/75/3552/2/spark_jobs/3552.scala | tee /home/nruest/tmp/auk/75/3552/2/spark_jobs/3552.scala.log

...

$ tree /home/nruest/tmp/auk/75/3552/2
/home/nruest/tmp/auk/75/3552/2
├── derivatives
│   ├── all-domains
│   │   └── output
│   │       ├── part-00000
│   │       ├── part-00001
│   │       ├── part-00002
│   │       ├── part-00003
│   │       ├── part-00004
│   │       └── _SUCCESS
│   ├── all-text
│   │   └── output
│   │       ├── part-00000
│   │       ├── part-00001
│   │       ├── part-00002
│   │       ├── part-00003
│   │       ├── part-00004
│   │       ├── part-00005
│   │       ├── part-00006
│   │       ├── part-00007
│   │       ├── part-00008
│   │       ├── part-00009
│   │       └── _SUCCESS
│   └── gephi
│       └── 3552-gephi.graphml
└── spark_jobs
    ├── 3552.scala
    └── 3552.scala.log

7 directories, 20 files

[ruebot]

Apache Spark 2.4.0 + aut 0.17.1-SNAPSHOT in AUK fails:

/home/nruest/bin/spark-2.4.0-bin-hadoop2.7/bin/spark-shell --master local\[10\] --driver-memory 30G --conf spark.network.timeout=10000000 --conf spark.executor.heartbeatInterval=600s --conf spark.driver.maxResultSize=4G --packages "io.archivesunleashed:aut:0.17.1-SNAPSHOT" -i /home/nruest/tmp/auk/75/3552/2/spark_jobs/3552.scala | tee /home/nruest/tmp/auk/75/3552/2/spark_jobs/3552.scala.log

:: problems summary ::
:::: WARNINGS
                [NOT FOUND  ] commons-configuration#commons-configuration;1.8!commons-configuration.jar (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/commons-configuration/commons-configuration/1.8/commons-configuration-1.8.jar

                [NOT FOUND  ] org.apache.commons#commons-lang3;3.3.1!commons-lang3.jar (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/org/apache/commons/commons-lang3/3.3.1/commons-lang3-3.3.1.jar

                [NOT FOUND  ] com.google.protobuf#protobuf-java;3.2.0!protobuf-java.jar(bundle) (1ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar

                [NOT FOUND  ] commons-codec#commons-codec;1.11!commons-codec.jar (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/commons-codec/commons-codec/1.11/commons-codec-1.11.jar

                [NOT FOUND  ] commons-io#commons-io;2.6!commons-io.jar (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/commons-io/commons-io/2.6/commons-io-2.6.jar

                [NOT FOUND  ] com.fasterxml.jackson.core#jackson-core;2.9.6!jackson-core.jar(bundle) (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.9.6/jackson-core-2.9.6.jar

                [NOT FOUND  ] com.fasterxml.jackson.core#jackson-annotations;2.9.6!jackson-annotations.jar(bundle) (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.9.6/jackson-annotations-2.9.6.jar

                [NOT FOUND  ] javax.xml.bind#jaxb-api;2.3.0!jaxb-api.jar (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/javax/xml/bind/jaxb-api/2.3.0/jaxb-api-2.3.0.jar

                [NOT FOUND  ] javax.ws.rs#javax.ws.rs-api;2.1!javax.ws.rs-api.${packaging.type} (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.1/javax.ws.rs-api-2.1.${packaging.type}

                [NOT FOUND  ] javax.annotation#javax.annotation-api;1.3!javax.annotation-api.jar (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/javax/annotation/javax.annotation-api/1.3/javax.annotation-api-1.3.jar

                [NOT FOUND  ] org.apache.httpcomponents#httpcore;4.4.10!httpcore.jar (0ms)

        ==== local-m2-cache: tried

          file:/home/nruest/.m2/repository/org/apache/httpcomponents/httpcore/4.4.10/httpcore-4.4.10.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::              FAILED DOWNLOADS            ::

                :: ^ see resolution messages for details  ^ ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: commons-configuration#commons-configuration;1.8!commons-configuration.jar

                :: org.apache.commons#commons-lang3;3.3.1!commons-lang3.jar

                :: com.google.protobuf#protobuf-java;3.2.0!protobuf-java.jar(bundle)

                :: javax.xml.bind#jaxb-api;2.3.0!jaxb-api.jar

                :: commons-codec#commons-codec;1.11!commons-codec.jar

                :: javax.ws.rs#javax.ws.rs-api;2.1!javax.ws.rs-api.${packaging.type}

                :: javax.annotation#javax.annotation-api;1.3!javax.annotation-api.jar

                :: commons-io#commons-io;2.6!commons-io.jar

                :: org.apache.httpcomponents#httpcore;4.4.10!httpcore.jar

                :: com.fasterxml.jackson.core#jackson-core;2.9.6!jackson-core.jar(bundle)

                :: com.fasterxml.jackson.core#jackson-annotations;2.9.6!jackson-annotations.jar(bundle)

                ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: commons-configuration#commons-configuration;1.8!commons-configuration.jar, download failed: org.apache.commons#commons-lang3;3.3.1!commons-lang3.jar, download failed: com.google.protobuf#protobuf-java;3.2.0!pr
otobuf-java.jar(bundle), download failed: javax.xml.bind#jaxb-api;2.3.0!jaxb-api.jar, download failed: commons-codec#commons-codec;1.11!commons-codec.jar, download failed: javax.ws.rs#javax.ws.rs-api;2.1!javax.ws.rs-api.${packaging.type}, download failed: javax.annotation#javax.ann
otation-api;1.3!javax.annotation-api.jar, download failed: commons-io#commons-io;2.6!commons-io.jar, download failed: org.apache.httpcomponents#httpcore;4.4.10!httpcore.jar, download failed: com.fasterxml.jackson.core#jackson-core;2.9.6!jackson-core.jar(bundle), download failed: co
m.fasterxml.jackson.core#jackson-annotations;2.9.6!jackson-annotations.jar(bundle)]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1306)
        at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:315)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I'll dig into the dependency issues more.

[ruebot]

Working on getting things to build with --packages on this branch. Things are starting to get ugly. We don't build now because we hit something like this with a test. I can keep chipping away on this, but I'd say let's make it backburner for now since I don't think it is much of a priority.

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.338 sec <<< FAILURE!
command line app tests(io.archivesunleashed.CommandLineAppTest)  Time elapsed: 5.118 sec  <<< ERROR!
java.lang.IllegalArgumentException: Illegal pattern component: XXX
	at org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
	at org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
	at org.apache.commons.lang3.time.FastDatePrinter.<init>(FastDatePrinter.java:142)
	at org.apache.commons.lang3.time.FastDateFormat.<init>(FastDateFormat.java:384)
	at org.apache.commons.lang3.time.FastDateFormat.<init>(FastDateFormat.java:369)
	at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
	at org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
	at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82)
	at org.apache.commons.lang3.time.FastDateFormat.getInstance(FastDateFormat.java:165)
	at org.apache.spark.sql.execution.datasources.csv.CSVOptions.<init>(CSVOptions.scala:139)
	at org.apache.spark.sql.execution.datasources.csv.CSVOptions.<init>(CSVOptions.scala:41)
	at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.prepareWrite(CSVFileFormat.scala:72)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:103)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228)
	at org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:656)
	at io.archivesunleashed.app.CommandLineApp.save(CommandLineApp.scala:183)
	at io.archivesunleashed.app.CommandLineApp$$anonfun$12.apply(CommandLineApp.scala:147)
	at io.archivesunleashed.app.CommandLineApp$$anonfun$12.apply(CommandLineApp.scala:142)
	at io.archivesunleashed.app.CommandLineApp.dfHandler(CommandLineApp.scala:250)
	at io.archivesunleashed.app.CommandLineApp.process(CommandLineApp.scala:298)
	at io.archivesunleashed.app.CommandLineAppRunner$.test(CommandLineApp.scala:346)
	at io.archivesunleashed.CommandLineAppTest$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(CommandLineAppTest.scala:76)
	at io.archivesunleashed.CommandLineAppTest$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(CommandLineAppTest.scala:75)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at io.archivesunleashed.CommandLineAppTest$$anonfun$2.apply$mcV$sp(CommandLineAppTest.scala:75)
	at io.archivesunleashed.CommandLineAppTest$$anonfun$2.apply(CommandLineAppTest.scala:74)
	at io.archivesunleashed.CommandLineAppTest$$anonfun$2.apply(CommandLineAppTest.scala:74)
	at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
	at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
	at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560)
	at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
	at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
	at io.archivesunleashed.CommandLineAppTest.org$scalatest$BeforeAndAfter$$super$runTest(CommandLineAppTest.scala:32)
	at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:203)
	at io.archivesunleashed.CommandLineAppTest.runTest(CommandLineAppTest.scala:32)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
	at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
	at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
	at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
	at org.scalatest.Suite$class.run(Suite.scala:1147)
	at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
	at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
	at io.archivesunleashed.CommandLineAppTest.org$scalatest$BeforeAndAfter$$super$run(CommandLineAppTest.scala:32)
	at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:258)
	at io.archivesunleashed.CommandLineAppTest.run(CommandLineAppTest.scala:32)
	at org.scalatest.junit.JUnitRunner.run(JUnitRunner.scala:99)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Results :

Tests in error: 
  command line app tests(io.archivesunleashed.CommandLineAppTest): Illegal pattern component: XXX

Tests run: 106, Failures: 0, Errors: 1, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:06 min
[INFO] Finished at: 2018-12-03T13:49:11-05:00
[INFO] Final Memory: 95M/949M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project aut: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/nruest/git/aut/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ruebot]

@jrwiebe I think I hit the Guava problem in this one too.