Skip to content

Commit 2d45571

Browse files
committed
[SPARK-8959] [SQL] [HOTFIX] Removes parquet-thrift and libthrift dependencies
These two dependencies were introduced in #7231 to help testing Parquet compatibility with `parquet-thrift`. However, they somehow crash the Scala compiler in Maven builds. This PR fixes this issue by: 1. Removing these two dependencies, and 2. Instead of generating the testing Parquet file programmatically, checking in an actual testing Parquet file generated by `parquet-thrift` as a test resource. This is just a quick fix to bring back Maven builds. Need to figure out the root case as binary Parquet files are harder to maintain. Author: Cheng Lian <[email protected]> Closes #7330 from liancheng/spark-8959 and squashes the following commits: cf69512 [Cheng Lian] Brings back Maven builds
1 parent a0cc3e5 commit 2d45571

File tree

7 files changed

+8
-3494
lines changed

7 files changed

+8
-3494
lines changed

pom.xml

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,6 @@
161161
<fasterxml.jackson.version>2.4.4</fasterxml.jackson.version>
162162
<snappy.version>1.1.1.7</snappy.version>
163163
<netlib.java.version>1.1.2</netlib.java.version>
164-
<thrift.version>0.9.2</thrift.version>
165164
<!-- For maven shade plugin (see SPARK-8819) -->
166165
<create.dependency.reduced.pom>false</create.dependency.reduced.pom>
167166

@@ -181,7 +180,6 @@
181180
<hive.deps.scope>compile</hive.deps.scope>
182181
<parquet.deps.scope>compile</parquet.deps.scope>
183182
<parquet.test.deps.scope>test</parquet.test.deps.scope>
184-
<thrift.test.deps.scope>test</thrift.test.deps.scope>
185183

186184
<!--
187185
Overridable test home. So that you can call individual pom files directly without
@@ -1122,18 +1120,6 @@
11221120
<version>${parquet.version}</version>
11231121
<scope>${parquet.test.deps.scope}</scope>
11241122
</dependency>
1125-
<dependency>
1126-
<groupId>org.apache.parquet</groupId>
1127-
<artifactId>parquet-thrift</artifactId>
1128-
<version>${parquet.version}</version>
1129-
<scope>${parquet.test.deps.scope}</scope>
1130-
</dependency>
1131-
<dependency>
1132-
<groupId>org.apache.thrift</groupId>
1133-
<artifactId>libthrift</artifactId>
1134-
<version>${thrift.version}</version>
1135-
<scope>${thrift.test.deps.scope}</scope>
1136-
</dependency>
11371123
<dependency>
11381124
<groupId>org.apache.flume</groupId>
11391125
<artifactId>flume-ng-core</artifactId>

sql/core/pom.xml

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -106,16 +106,6 @@
106106
<artifactId>parquet-avro</artifactId>
107107
<scope>test</scope>
108108
</dependency>
109-
<dependency>
110-
<groupId>org.apache.parquet</groupId>
111-
<artifactId>parquet-thrift</artifactId>
112-
<scope>test</scope>
113-
</dependency>
114-
<dependency>
115-
<groupId>org.apache.thrift</groupId>
116-
<artifactId>libthrift</artifactId>
117-
<scope>test</scope>
118-
</dependency>
119109
</dependencies>
120110
<build>
121111
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>

0 commit comments

Comments
 (0)