Skip to content

Commit c7253da

Browse files
srowenpwendell
srowen
authored and
pwendell
committed
SPARK-1789. Multiple versions of Netty dependencies cause FlumeStreamSuite failure
TL;DR is there is a bit of JAR hell trouble with Netty, that can be mostly resolved and will resolve a test failure. I hit the error described at http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-startup-time-out-td1753.html while running FlumeStreamingSuite, and have for a short while (is it just me?) velvia notes: "I have found a workaround. If you add akka 2.2.4 to your dependencies, then everything works, probably because akka 2.2.4 brings in newer version of Jetty." There are at least 3 versions of Netty in play in the build: - the new Flume 1.4.0 dependency brings in io.netty:netty:3.4.0.Final, and that is the immediate problem - the custom version of akka 2.2.3 depends on io.netty:netty:3.6.6. - but, Spark Core directly uses io.netty:netty-all:4.0.17.Final The POMs try to exclude other versions of netty, but are excluding org.jboss.netty:netty, when in fact older versions of io.netty:netty (not netty-all) are also an issue. The org.jboss.netty:netty excludes are largely unnecessary. I replaced many of them with io.netty:netty exclusions until everything agreed on io.netty:netty-all:4.0.17.Final. But this didn't work, since Akka 2.2.3 doesn't work with Netty 4.x. Down-grading to 3.6.6.Final across the board made some Spark code not compile. If the build *keeps* io.netty:netty:3.6.6.Final as well, everything seems to work. Part of the reason seems to be that Netty 3.x used the old `org.jboss.netty` packages. This is less than ideal, but is no worse than the current situation. So this PR resolves the issue and improves the JAR hell, even if it leaves the existing theoretical Netty 3-vs-4 conflict: - Remove org.jboss.netty excludes where possible, for clarity; they're not needed except with Hadoop artifacts - Add io.netty:netty excludes where needed -- except, let akka keep its io.netty:netty - Change a bit of test code that actually depended on Netty 3.x, to use 4.x equivalent - Update SBT build accordingly A better change would be to update Akka far enough such that it agrees on Netty 4.x, but I don't know if that's feasible. Author: Sean Owen <[email protected]> Closes #723 from srowen/SPARK-1789 and squashes the following commits: 43661b7 [Sean Owen] Update and add Netty excludes to prevent some JAR conflicts that cause test issues (cherry picked from commit 2b7bd29) Signed-off-by: Patrick Wendell <[email protected]>
1 parent 4e9a0cb commit c7253da

File tree

8 files changed

+24
-70
lines changed

8 files changed

+24
-70
lines changed

‎core/src/test/scala/org/apache/spark/LocalSparkContext.scala‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,7 @@
1717

1818
package org.apache.spark
1919

20-
import org.jboss.netty.logging.InternalLoggerFactory
21-
import org.jboss.netty.logging.Slf4JLoggerFactory
20+
import _root_.io.netty.util.internal.logging.{Slf4JLoggerFactory, InternalLoggerFactory}
2221
import org.scalatest.BeforeAndAfterAll
2322
import org.scalatest.BeforeAndAfterEach
2423
import org.scalatest.Suite

‎examples/pom.xml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,10 @@
106106
<groupId>org.jboss.netty</groupId>
107107
<artifactId>netty</artifactId>
108108
</exclusion>
109+
<exclusion>
110+
<groupId>io.netty</groupId>
111+
<artifactId>netty</artifactId>
112+
</exclusion>
109113
<exclusion>
110114
<groupId>commons-logging</groupId>
111115
<artifactId>commons-logging</artifactId>

‎external/flume/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@
5050
<version>1.4.0</version>
5151
<exclusions>
5252
<exclusion>
53-
<groupId>org.jboss.netty</groupId>
53+
<groupId>io.netty</groupId>
5454
<artifactId>netty</artifactId>
5555
</exclusion>
5656
<exclusion>

‎external/mqtt/pom.xml‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -53,12 +53,6 @@
5353
<groupId>${akka.group}</groupId>
5454
<artifactId>akka-zeromq_${scala.binary.version}</artifactId>
5555
<version>${akka.version}</version>
56-
<exclusions>
57-
<exclusion>
58-
<groupId>org.jboss.netty</groupId>
59-
<artifactId>netty</artifactId>
60-
</exclusion>
61-
</exclusions>
6256
</dependency>
6357
<dependency>
6458
<groupId>org.scalatest</groupId>

‎external/twitter/pom.xml‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,12 +48,6 @@
4848
<groupId>org.twitter4j</groupId>
4949
<artifactId>twitter4j-stream</artifactId>
5050
<version>3.0.3</version>
51-
<exclusions>
52-
<exclusion>
53-
<groupId>org.jboss.netty</groupId>
54-
<artifactId>netty</artifactId>
55-
</exclusion>
56-
</exclusions>
5751
</dependency>
5852
<dependency>
5953
<groupId>org.scalatest</groupId>

‎external/zeromq/pom.xml‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,12 +48,6 @@
4848
<groupId>${akka.group}</groupId>
4949
<artifactId>akka-zeromq_${scala.binary.version}</artifactId>
5050
<version>${akka.version}</version>
51-
<exclusions>
52-
<exclusion>
53-
<groupId>org.jboss.netty</groupId>
54-
<artifactId>netty</artifactId>
55-
</exclusion>
56-
</exclusions>
5751
</dependency>
5852
<dependency>
5953
<groupId>org.scalatest</groupId>

‎pom.xml‎

Lines changed: 0 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -324,45 +324,21 @@
324324
<groupId>${akka.group}</groupId>
325325
<artifactId>akka-actor_${scala.binary.version}</artifactId>
326326
<version>${akka.version}</version>
327-
<exclusions>
328-
<exclusion>
329-
<groupId>org.jboss.netty</groupId>
330-
<artifactId>netty</artifactId>
331-
</exclusion>
332-
</exclusions>
333327
</dependency>
334328
<dependency>
335329
<groupId>${akka.group}</groupId>
336330
<artifactId>akka-remote_${scala.binary.version}</artifactId>
337331
<version>${akka.version}</version>
338-
<exclusions>
339-
<exclusion>
340-
<groupId>org.jboss.netty</groupId>
341-
<artifactId>netty</artifactId>
342-
</exclusion>
343-
</exclusions>
344332
</dependency>
345333
<dependency>
346334
<groupId>${akka.group}</groupId>
347335
<artifactId>akka-slf4j_${scala.binary.version}</artifactId>
348336
<version>${akka.version}</version>
349-
<exclusions>
350-
<exclusion>
351-
<groupId>org.jboss.netty</groupId>
352-
<artifactId>netty</artifactId>
353-
</exclusion>
354-
</exclusions>
355337
</dependency>
356338
<dependency>
357339
<groupId>${akka.group}</groupId>
358340
<artifactId>akka-testkit_${scala.binary.version}</artifactId>
359341
<version>${akka.version}</version>
360-
<exclusions>
361-
<exclusion>
362-
<groupId>org.jboss.netty</groupId>
363-
<artifactId>netty</artifactId>
364-
</exclusion>
365-
</exclusions>
366342
</dependency>
367343
<dependency>
368344
<groupId>colt</groupId>
@@ -513,10 +489,6 @@
513489
<artifactId>avro</artifactId>
514490
<version>${avro.version}</version>
515491
<exclusions>
516-
<exclusion>
517-
<groupId>org.jboss.netty</groupId>
518-
<artifactId>netty</artifactId>
519-
</exclusion>
520492
<exclusion>
521493
<groupId>io.netty</groupId>
522494
<artifactId>netty</artifactId>
@@ -551,10 +523,6 @@
551523
<artifactId>avro-mapred</artifactId>
552524
<version>${avro.version}</version>
553525
<exclusions>
554-
<exclusion>
555-
<groupId>org.jboss.netty</groupId>
556-
<artifactId>netty</artifactId>
557-
</exclusion>
558526
<exclusion>
559527
<groupId>io.netty</groupId>
560528
<artifactId>netty</artifactId>

‎project/SparkBuild.scala‎

Lines changed: 18 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -303,7 +303,8 @@ object SparkBuild extends Build {
303303
val parquetVersion = "1.4.3"
304304
val slf4jVersion = "1.7.5"
305305

306-
val excludeNetty = ExclusionRule(organization = "org.jboss.netty")
306+
val excludeJBossNetty = ExclusionRule(organization = "org.jboss.netty")
307+
val excludeIONetty = ExclusionRule(organization = "io.netty")
307308
val excludeEclipseJetty = ExclusionRule(organization = "org.eclipse.jetty")
308309
val excludeAsm = ExclusionRule(organization = "org.ow2.asm")
309310
val excludeOldAsm = ExclusionRule(organization = "asm")
@@ -337,17 +338,17 @@ object SparkBuild extends Build {
337338
"commons-daemon" % "commons-daemon" % "1.0.10", // workaround for bug HADOOP-9407
338339
"com.ning" % "compress-lzf" % "1.0.0",
339340
"org.xerial.snappy" % "snappy-java" % "1.0.5",
340-
"org.spark-project.akka" %% "akka-remote" % akkaVersion excludeAll(excludeNetty),
341-
"org.spark-project.akka" %% "akka-slf4j" % akkaVersion excludeAll(excludeNetty),
341+
"org.spark-project.akka" %% "akka-remote" % akkaVersion,
342+
"org.spark-project.akka" %% "akka-slf4j" % akkaVersion,
342343
"org.spark-project.akka" %% "akka-testkit" % akkaVersion % "test",
343344
"org.json4s" %% "json4s-jackson" % "3.2.6" excludeAll(excludeScalap),
344345
"colt" % "colt" % "1.2.0",
345346
"org.apache.mesos" % "mesos" % "0.13.0",
346347
"commons-net" % "commons-net" % "2.2",
347348
"net.java.dev.jets3t" % "jets3t" % jets3tVersion excludeAll(excludeCommonsLogging),
348349
"org.apache.derby" % "derby" % "10.4.2.0" % "test",
349-
"org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm),
350-
"org.apache.curator" % "curator-recipes" % "2.4.0" excludeAll(excludeNetty),
350+
"org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeJBossNetty, excludeAsm, excludeCommonsLogging, excludeSLF4J, excludeOldAsm),
351+
"org.apache.curator" % "curator-recipes" % "2.4.0" excludeAll(excludeJBossNetty),
351352
"com.codahale.metrics" % "metrics-core" % codahaleMetricsVersion,
352353
"com.codahale.metrics" % "metrics-jvm" % codahaleMetricsVersion,
353354
"com.codahale.metrics" % "metrics-json" % codahaleMetricsVersion,
@@ -421,15 +422,15 @@ object SparkBuild extends Build {
421422
v => "spark-examples-" + v + "-hadoop" + hadoopVersion + ".jar" },
422423
libraryDependencies ++= Seq(
423424
"com.twitter" %% "algebird-core" % "0.1.11",
424-
"org.apache.hbase" % "hbase" % HBASE_VERSION excludeAll(excludeNetty, excludeAsm, excludeOldAsm, excludeCommonsLogging, excludeJruby),
425+
"org.apache.hbase" % "hbase" % HBASE_VERSION excludeAll(excludeIONetty, excludeJBossNetty, excludeAsm, excludeOldAsm, excludeCommonsLogging, excludeJruby),
425426
"org.apache.cassandra" % "cassandra-all" % "1.2.6"
426427
exclude("com.google.guava", "guava")
427428
exclude("com.googlecode.concurrentlinkedhashmap", "concurrentlinkedhashmap-lru")
428429
exclude("com.ning","compress-lzf")
429430
exclude("io.netty", "netty")
430431
exclude("jline","jline")
431432
exclude("org.apache.cassandra.deps", "avro")
432-
excludeAll(excludeSLF4J),
433+
excludeAll(excludeSLF4J, excludeIONetty),
433434
"com.github.scopt" %% "scopt" % "3.2.0"
434435
)
435436
) ++ assemblySettings ++ extraAssemblySettings
@@ -561,11 +562,11 @@ object SparkBuild extends Build {
561562
def yarnEnabledSettings = Seq(
562563
libraryDependencies ++= Seq(
563564
// Exclude rule required for all ?
564-
"org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm),
565-
"org.apache.hadoop" % "hadoop-yarn-api" % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm),
566-
"org.apache.hadoop" % "hadoop-yarn-common" % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm),
567-
"org.apache.hadoop" % "hadoop-yarn-client" % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm),
568-
"org.apache.hadoop" % "hadoop-yarn-server-web-proxy" % hadoopVersion excludeAll(excludeNetty, excludeAsm, excludeOldAsm)
565+
"org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeJBossNetty, excludeAsm, excludeOldAsm),
566+
"org.apache.hadoop" % "hadoop-yarn-api" % hadoopVersion excludeAll(excludeJBossNetty, excludeAsm, excludeOldAsm),
567+
"org.apache.hadoop" % "hadoop-yarn-common" % hadoopVersion excludeAll(excludeJBossNetty, excludeAsm, excludeOldAsm),
568+
"org.apache.hadoop" % "hadoop-yarn-client" % hadoopVersion excludeAll(excludeJBossNetty, excludeAsm, excludeOldAsm),
569+
"org.apache.hadoop" % "hadoop-yarn-server-web-proxy" % hadoopVersion excludeAll(excludeJBossNetty, excludeAsm, excludeOldAsm)
569570
)
570571
)
571572

@@ -593,36 +594,36 @@ object SparkBuild extends Build {
593594
name := "spark-streaming-twitter",
594595
previousArtifact := sparkPreviousArtifact("spark-streaming-twitter"),
595596
libraryDependencies ++= Seq(
596-
"org.twitter4j" % "twitter4j-stream" % "3.0.3" excludeAll(excludeNetty)
597+
"org.twitter4j" % "twitter4j-stream" % "3.0.3"
597598
)
598599
)
599600

600601
def kafkaSettings() = sharedSettings ++ Seq(
601602
name := "spark-streaming-kafka",
602603
previousArtifact := sparkPreviousArtifact("spark-streaming-kafka"),
603604
libraryDependencies ++= Seq(
604-
"com.github.sgroschupf" % "zkclient" % "0.1" excludeAll(excludeNetty),
605+
"com.github.sgroschupf" % "zkclient" % "0.1",
605606
"org.apache.kafka" %% "kafka" % "0.8.0"
606607
exclude("com.sun.jdmk", "jmxtools")
607608
exclude("com.sun.jmx", "jmxri")
608609
exclude("net.sf.jopt-simple", "jopt-simple")
609-
excludeAll(excludeNetty, excludeSLF4J)
610+
excludeAll(excludeSLF4J)
610611
)
611612
)
612613

613614
def flumeSettings() = sharedSettings ++ Seq(
614615
name := "spark-streaming-flume",
615616
previousArtifact := sparkPreviousArtifact("spark-streaming-flume"),
616617
libraryDependencies ++= Seq(
617-
"org.apache.flume" % "flume-ng-sdk" % "1.4.0" % "compile" excludeAll(excludeNetty, excludeThrift)
618+
"org.apache.flume" % "flume-ng-sdk" % "1.4.0" % "compile" excludeAll(excludeIONetty, excludeThrift)
618619
)
619620
)
620621

621622
def zeromqSettings() = sharedSettings ++ Seq(
622623
name := "spark-streaming-zeromq",
623624
previousArtifact := sparkPreviousArtifact("spark-streaming-zeromq"),
624625
libraryDependencies ++= Seq(
625-
"org.spark-project.akka" %% "akka-zeromq" % akkaVersion excludeAll(excludeNetty)
626+
"org.spark-project.akka" %% "akka-zeromq" % akkaVersion
626627
)
627628
)
628629

0 commit comments

Comments
 (0)