[SPARK-782] Made Spark use existing shaded ASM and removed Spark's ASM dependency #232
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Merged build triggered. |
|
Merged build started. |
|
Looks good to me. Thanks TD! |
|
Merged build finished. |
|
All automated tests passed. |
|
Which dependency is providing this shaded version ? |
|
We're using the shaded version from Kryo. If they change this in future releases we can change accordingly, but other projects are also using Kryo's shaded version so I don't think it's something they are likely to change. |
|
Also, refer to the discussion here #100 This PR only backports the changes already accepted in master. |
asfgit
pushed a commit
that referenced
this pull request
Mar 26, 2014
…M dependency This ports the changes in #100 to branch 0.9. However, unlike that PR, it does not exclude ASM from all dependencies of Spark, to ensure compatibility in branch 0.9. Author: Tathagata Das <[email protected]> Closes #232 from tdas/asm and squashes the following commits: 999bb6f [Tathagata Das] Made Spark use existing shaded ASM and removed Spark's ASM depedency.
jhartlaub
referenced
this pull request
in jhartlaub/spark
May 27, 2014
jobWaiter.synchronized before jobWaiter.wait ...else ``IllegalMonitorStateException`` in ``SimpleFutureAction#ready``. (cherry picked from commit 0780498) Signed-off-by: Reynold Xin <[email protected]>
jhartlaub
referenced
this pull request
in jhartlaub/spark
May 27, 2014
jobWaiter.synchronized before jobWaiter.wait ...else ``IllegalMonitorStateException`` in ``SimpleFutureAction#ready``. (cherry picked from commit 0780498) Signed-off-by: Reynold Xin <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Nov 11, 2015
…mpiled classes This patch modifies Spark's closure cleaner (and a few other places) to use ASM 5, which is necessary in order to support cleaning of closures that were compiled by Java 8. In order to avoid ASM dependency conflicts, Spark excludes ASM from all of its dependencies and uses a shaded version of ASM 4 that comes from `reflectasm` (see [SPARK-782](https://issues.apache.org/jira/browse/SPARK-782) and #232). This patch updates Spark to use a shaded version of ASM 5.0.4 that was published by the Apache XBean project; the POM used to create the shaded artifact can be found at https://github.com/apache/geronimo-xbean/blob/xbean-4.4/xbean-asm5-shaded/pom.xml. http://movingfulcrum.tumblr.com/post/80826553604/asm-framework-50-the-missing-migration-guide was a useful resource while upgrading the code to use the new ASM5 opcodes. I also added a new regression tests in the `java8-tests` subproject; the existing tests were insufficient to catch this bug, which only affected Scala 2.11 user code which was compiled targeting Java 8. Author: Josh Rosen <[email protected]> Closes #9512 from JoshRosen/SPARK-6152.
asfgit
pushed a commit
that referenced
this pull request
Nov 11, 2015
…mpiled classes This patch modifies Spark's closure cleaner (and a few other places) to use ASM 5, which is necessary in order to support cleaning of closures that were compiled by Java 8. In order to avoid ASM dependency conflicts, Spark excludes ASM from all of its dependencies and uses a shaded version of ASM 4 that comes from `reflectasm` (see [SPARK-782](https://issues.apache.org/jira/browse/SPARK-782) and #232). This patch updates Spark to use a shaded version of ASM 5.0.4 that was published by the Apache XBean project; the POM used to create the shaded artifact can be found at https://github.com/apache/geronimo-xbean/blob/xbean-4.4/xbean-asm5-shaded/pom.xml. http://movingfulcrum.tumblr.com/post/80826553604/asm-framework-50-the-missing-migration-guide was a useful resource while upgrading the code to use the new ASM5 opcodes. I also added a new regression tests in the `java8-tests` subproject; the existing tests were insufficient to catch this bug, which only affected Scala 2.11 user code which was compiled targeting Java 8. Author: Josh Rosen <[email protected]> Closes #9512 from JoshRosen/SPARK-6152. (cherry picked from commit 529a1d3) Signed-off-by: Michael Armbrust <[email protected]>
dskrvk
pushed a commit
to dskrvk/spark
that referenced
this pull request
Nov 13, 2015
…mpiled classes This patch modifies Spark's closure cleaner (and a few other places) to use ASM 5, which is necessary in order to support cleaning of closures that were compiled by Java 8. In order to avoid ASM dependency conflicts, Spark excludes ASM from all of its dependencies and uses a shaded version of ASM 4 that comes from `reflectasm` (see [SPARK-782](https://issues.apache.org/jira/browse/SPARK-782) and apache#232). This patch updates Spark to use a shaded version of ASM 5.0.4 that was published by the Apache XBean project; the POM used to create the shaded artifact can be found at https://github.com/apache/geronimo-xbean/blob/xbean-4.4/xbean-asm5-shaded/pom.xml. http://movingfulcrum.tumblr.com/post/80826553604/asm-framework-50-the-missing-migration-guide was a useful resource while upgrading the code to use the new ASM5 opcodes. I also added a new regression tests in the `java8-tests` subproject; the existing tests were insufficient to catch this bug, which only affected Scala 2.11 user code which was compiled targeting Java 8. Author: Josh Rosen <[email protected]> Closes apache#9512 from JoshRosen/SPARK-6152.
liancheng
pushed a commit
to liancheng/spark
that referenced
this pull request
Mar 17, 2017
…rotocol ## What changes were proposed in this pull request? This avoids being latency bound on file reads. Initial testing shows this to be at least 5x faster than without the thread pool. ## How was this patch tested? Existing tests; perf test on dogfood. Author: Eric Liang <[email protected]> Closes apache#232 from ericl/sc-5937.
Igosuki
pushed a commit
to Adikteev/spark
that referenced
this pull request
Jul 31, 2018
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
Upload multiple junit files to avoid test timeout
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This ports the changes in #100 to branch 0.9. However, unlike that PR, it does not exclude ASM from all dependencies of Spark, to ensure compatibility in branch 0.9.