[SPARK-4785] [SQL] Support udf instance ser/de after initialization #3640

chenghao-intel · 2014-12-09T03:55:32Z

UDF contract change in Hive 0.13.1. In Hive 0.12.0, it's always safe to construct and initialize a fresh UDF object on worker side, while in Hive 0.13.1, UDF objects should only be initialized on driver side and then serialized to the worker side. We provide the ability to serialize the initialized UDF instance and deserialize them cross process boundary.

SparkQA · 2014-12-09T03:57:53Z

Test build #24237 has started for PR 3640 at commit 19cbd46.

This patch merges cleanly.

SparkQA · 2014-12-09T04:10:23Z

Test build #24238 has started for PR 3640 at commit e9c3212.

This patch merges cleanly.

SparkQA · 2014-12-09T04:31:52Z

Test build #24237 has finished for PR 3640 at commit 19cbd46.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveFunctionCache(var functionClassName: String) extends java.io.Externalizable

AmplabJenkins · 2014-12-09T04:31:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24237/
Test FAILed.

SparkQA · 2014-12-09T04:46:11Z

Test build #24238 has finished for PR 3640 at commit e9c3212.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveFunctionCache(var functionClassName: String) extends java.io.Externalizable

AmplabJenkins · 2014-12-09T04:46:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24238/
Test FAILed.

SparkQA · 2014-12-09T06:20:27Z

Test build #24240 has started for PR 3640 at commit 396c0e1.

This patch merges cleanly.

SparkQA · 2014-12-09T07:27:24Z

Test build #24240 has finished for PR 3640 at commit 396c0e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveFunctionCache(var functionClassName: String) extends java.io.Externalizable

AmplabJenkins · 2014-12-09T07:27:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24240/
Test PASSed.

liancheng · 2014-12-09T08:03:26Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala

HiveFunctionWrapper might be a better name, since essentially this class is just used to make Hive function objects serializable. And traditionally Spark uses "wrapper" as suffix for classes with similar purposes.

Yes, that's a good suggestion, will update it.

Another comment, related to HiveShim. I was thinking to move this class rather than de/serializePlan methods into the shim layer. Hive 0.12.0 is not affected by SPARK-4785, thus the version in 0.12.0 shim can be very simple. We only need to handle 0.13.1 there. This also lowers the possibility of breaking 0.12.0 code paths.

Also after moving this class into the shim layer, as I've mentioned in another comment, instead of relying on de/serializePlan, we can just mimic Utilities.de/serializeObjectByKryo in read/writeExternal methods in this class.

liancheng · 2014-12-09T08:53:00Z

Appreciate a lot for fixing this case! The serialization wrapper class makes sense. However, would like to make some refactoring. A summary of my comments above:

Renaming HiveFunctionCache to HiveFunctionWrapper
Moving HiveFunctionWrapper to the shim layer, and keep Hive 0.12.0 code path exactly the same as before.

Considering Hive 0.12.0 is not affected by this issue, and 1.2.0 release is really close, I'd like to lower the possibility of breaking 0.12.0 code paths as much as possible.
Add a HiveSimpleUdfWrapper, which inherits from HiveFunctionWrapper.

As you've mentioned in the code, HiveSimpleUdf is a special case, it shouldn't be serialized, and should always create a fresh object on executor side. Currently this special case complicates HiveFunctionWrapper implementation and makes it somewhat confusing. Defining a special subclass for HvieSimpleUdf helps making HiveFunctionWrapper simpler (e.g. no need to serialize the boolean flag any more).
Avoid using Utilities.de/serializePlan by mimicking Utilities.de/serializeObjectByKryo in the read/writeExternal methods of the wrapper class, so that we can remove the expensive HiveConf instantiation.

In a word, we can add two classes, HiveFunctionWrapper and HiveSImpleUdfWrapper into the shim layer, and make sure that the 0.12.0 version behaves exactly the same as before.

SparkQA · 2014-12-09T14:00:22Z

Test build #24247 has started for PR 3640 at commit 74466a3.

This patch merges cleanly.

SparkQA · 2014-12-09T14:12:56Z

Test build #24248 has started for PR 3640 at commit 8e13756.

This patch merges cleanly.

SparkQA · 2014-12-09T15:05:08Z

Test build #24247 has finished for PR 3640 at commit 74466a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveFunctionWrapper(var functionClassName: String) extends java.io.Serializable
- class HiveFunctionWrapper(var functionClassName: String) extends java.io.Externalizable

AmplabJenkins · 2014-12-09T15:05:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24247/
Test PASSed.

SparkQA · 2014-12-09T15:16:49Z

Test build #24248 has finished for PR 3640 at commit 8e13756.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveFunctionWrapper(var functionClassName: String) extends java.io.Serializable
- class HiveFunctionWrapper(var functionClassName: String) extends java.io.Externalizable

AmplabJenkins · 2014-12-09T15:16:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24248/
Test PASSed.

liancheng · 2014-12-09T17:15:47Z

The current design of this PR is derived from some background knowledges. I'd like to provide a brief summary here for future reference.

As mentioned in the PR description, different from Hive 0.12.0, in Hive 0.13.1 UDF/UDAF/UDTF (aka Hive function) objects should only be initialized once on the driver side and then serialized to executors. However, not all function objects are serializable (e.g. GenericUDF doesn't implement Serializable). Hive 0.13.1 solves this issue with Kryo or XML serializer. Several utility ser/de methods are provided in class o.a.h.h.q.e.Utilities for this purpose. In this PR we chose Kryo for efficiency. The Kryo serializer used here is created in Hive. Spark Kryo serializer wasn't used because there's no available SparkConf instance.

…m with a wrapper Different from Hive 0.12.0, in Hive 0.13.1 UDF/UDAF/UDTF (aka Hive function) objects should only be initialized once on the driver side and then serialized to executors. However, not all function objects are serializable (e.g. GenericUDF doesn't implement Serializable). Hive 0.13.1 solves this issue with Kryo or XML serializer. Several utility ser/de methods are provided in class o.a.h.h.q.e.Utilities for this purpose. In this PR we chose Kryo for efficiency. The Kryo serializer used here is created in Hive. Spark Kryo serializer wasn't used because there's no available SparkConf instance. Author: Cheng Hao <[email protected]> Author: Cheng Lian <[email protected]> Closes #3640 from chenghao-intel/udf_serde and squashes the following commits: 8e13756 [Cheng Hao] Update the comment 74466a3 [Cheng Hao] refactor as feedbacks 396c0e1 [Cheng Hao] avoid Simple UDF to be serialized e9c3212 [Cheng Hao] update the comment 19cbd46 [Cheng Hao] support udf instance ser/de after initialization (cherry picked from commit 383c555) Signed-off-by: Michael Armbrust <[email protected]>

marmbrus · 2014-12-09T18:35:15Z

Thanks a lot guys for digging into this! Merged to master and branch 1.2

support udf instance ser/de after initialization

19cbd46

update the comment

e9c3212

avoid Simple UDF to be serialized

396c0e1

liancheng reviewed Dec 9, 2014
View reviewed changes

refactor as feedbacks

74466a3

Update the comment

8e13756

asfgit closed this in 383c555 Dec 9, 2014

chenghao-intel deleted the udf_serde branch July 2, 2015 08:41

JoshRosen mentioned this pull request Oct 19, 2015

[SPARK-10151] [SQL] Support invocation of hive macro #8354

Closed

[SPARK-4785] [SQL] Support udf instance ser/de after initialization #3640

[SPARK-4785] [SQL] Support udf instance ser/de after initialization #3640

Uh oh!

Conversation

chenghao-intel commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

AmplabJenkins commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

AmplabJenkins commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

AmplabJenkins commented Dec 9, 2014

Uh oh!

liancheng Dec 9, 2014

Choose a reason for hiding this comment

Uh oh!

chenghao-intel Dec 9, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng Dec 9, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

AmplabJenkins commented Dec 9, 2014

Uh oh!

SparkQA commented Dec 9, 2014

Uh oh!

AmplabJenkins commented Dec 9, 2014

Uh oh!

liancheng commented Dec 9, 2014

Uh oh!

marmbrus commented Dec 9, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants