Skip to content

Conversation

@turboFei
Copy link
Member

@turboFei turboFei commented Nov 15, 2025

Why are the changes needed?

Close #3943.

all not shaded class

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|wc -l                      
    1077

org.apache.thrift

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'org/apache/thrift'|wc -l
     179

org.apache.commons.io

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'org/apache/commons/io' | wc -l
     201

com.github.benmanes

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'com/github/benmanes' | wc -l
     692

org.threeten from orc-core https://github.com/apache/orc/tree/main/java/core/src/java/org/threeten

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep  'org/threeten' | wc -l      
       4

others:

jar -tf amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5/target/amoro-format-mixed-spark-runtime-3.5_2.12-0.9-SNAPSHOT.jar |grep '.class$' |grep -v amoro|grep -v shade|grep -v 'org/apache/spark'|grep -v 'org/apache/thrift' |grep -v 'org/apache/commons/io' |grep -v 'com/github/benmanes' |grep -v 'org/threeten'

META-INF/versions/9/module-info.class

Brief change log

  • Shade all third party classes for spark runtime

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

./mvnw clean package -Pspark-3.5 -Pscala-2.12 -pl amoro-format-mixed/amoro-mixed-spark/v3.5/amoro-mixed-spark-runtime-3.5 -am -DskipTests     

all classes shaded
image

./mvnw clean package -Pspark-3.3 -Pscala-2.12 -pl amoro-format-mixed/amoro-mixed-spark/v3.3/amoro-mixed-spark-runtime-3.3 -am -DskipTests     
image
  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added module:mixed-spark Spark module for Mixed Format type:build labels Nov 15, 2025
@turboFei turboFei changed the title relocate [AMORO-3943] Shade all third party classes for spark runtime Nov 15, 2025
@turboFei
Copy link
Member Author

cc @zhoujinsong @xxubai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-spark Spark module for Mixed Format type:build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: Shade all thirdparty classes for spark runtime

1 participant