Skip to content

[SPARK-11194] [SQL] [BRANCH-1.5] [WIP] Use MutableURLClassLoader for the classLoader in IsolatedClientLoader. #9171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-11194] [SQL] [BRANCH-1.5] [WIP] Use MutableURLClassLoader for the classLoader in IsolatedClientLoader. #9171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,18 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) with Logging {
@transient
protected[sql] lazy val substitutor = new VariableSubstitution()

/**
* The classLoader used to store all user-added jars through "ADD JAR" command.
* This classLoader is a special URLClassLoader that exposes its addURL method.
* So, when we add jar, we can add this new jar directly through the addURL method
* instead of stacking a new URLClassLoader on top of it.
*/
@transient
protected[hive] val libraryClassLoader: NonClosableMutableURLClassLoader = {
val baseClassLoader = Utils.getContextOrSparkClassLoader
new NonClosableMutableURLClassLoader(baseClassLoader)
}

/**
* The copy of the hive client that is used for execution. Currently this must always be
* Hive 13 as this is the version of Hive that is packaged with Spark SQL. This copy of the
Expand Down
13 changes: 12 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import org.apache.commons.io.{FileUtils, IOUtils}

import org.apache.spark.Logging
import org.apache.spark.deploy.SparkSubmitUtils
import org.apache.spark.util.Utils
import org.apache.spark.util.{MutableURLClassLoader, Utils}

import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.HiveContext
Expand Down Expand Up @@ -197,3 +197,14 @@ private[hive] class IsolatedClientLoader(
Thread.currentThread.setContextClassLoader(baseClassLoader)
}
}

/**
* URL class loader that exposes the `addURL` and `getURLs` methods in URLClassLoader.
* This class loader cannot be closed (its `close` method is a no-op).
*/
private[sql] class NonClosableMutableURLClassLoader(
parent: ClassLoader)
extends MutableURLClassLoader(Array.empty, parent) {

override def close(): Unit = {}
}
7 changes: 3 additions & 4 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,20 +87,19 @@ case class AddJar(path: String) extends RunnableCommand {

override def run(sqlContext: SQLContext): Seq[Row] = {
val hiveContext = sqlContext.asInstanceOf[HiveContext]
val currentClassLoader = Utils.getContextOrSparkClassLoader

// Add jar to current context
val jarURL = new java.io.File(path).toURI.toURL
val newClassLoader = new java.net.URLClassLoader(Array(jarURL), currentClassLoader)
Thread.currentThread.setContextClassLoader(newClassLoader)
hiveContext.libraryClassLoader.addURL(jarURL)
Thread.currentThread.setContextClassLoader(hiveContext.libraryClassLoader)
// We need to explicitly set the class loader associated with the conf in executionHive's
// state because this class loader will be used as the context class loader of the current
// thread to execute any Hive command.
// We cannot use `org.apache.hadoop.hive.ql.metadata.Hive.get().getConf()` because Hive.get()
// returns the value of a thread local variable and its HiveConf may not be the HiveConf
// associated with `executionHive.state` (for example, HiveContext is created in one thread
// and then add jar is called from another thread).
hiveContext.executionHive.state.getConf.setClassLoader(newClassLoader)
hiveContext.executionHive.state.getConf.setClassLoader(hiveContext.libraryClassLoader)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this pr also suffers the same problem exposed in #9170. There was no test failure because we only set the mutable classloader when we actually do add jar (In #9170 we set the mutable classloader to executionHive.state.getConf whenever we need to use withHiveState).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have changed the classloader to a non-closable mutable url class loader.

// Add jar to isolated hive (metadataHive) class loader.
hiveContext.runSqlHive(s"ADD JAR $path")

Expand Down