[SPARK-24360][SQL] Support Hive 3.0 metastore #21404
Conversation
| val allSupportedHiveVersions = Set(v12, v13, v14, v1_0, v1_1, v1_2, v2_0, v2_1, v2_2, v2_3) | ||
| case object v3_0 extends HiveVersion("3.0.0", | ||
| exclusions = Seq("org.apache.curator:*", | ||
| "org.apache.hadoop:hadoop-aws", |
There was a problem hiding this comment.
What happened if we do not have this?
There was a problem hiding this comment.
Thanks. I'll remove this in this PR.
| private lazy val clazzLoadFileType = getClass.getClassLoader.loadClass( | ||
| "org.apache.hadoop.hive.ql.plan.LoadTableDesc$LoadFileType") | ||
|
|
||
| private lazy val loadPartitionMethod = |
There was a problem hiding this comment.
BTW, I tracked and checked all the signature changed.
There was a problem hiding this comment.
Thank you, @HyukjinKwon .
|
Test build #91001 has finished for PR 21404 at commit
|
|
The failures some from |
|
Retest this please. |
|
Test build #91003 has finished for PR 21404 at commit
|
|
Test build #91008 has finished for PR 21404 at commit
|
|
Can we remove the old hive support? such as 0.12, 0.13 and 0.14. |
|
Probably separate ticket for Spark 3.0.0. |
|
I'm investigating timing issue here. Spark loads Hive Metastore class lazily. Here, Spark is trying to access Hive metastore tables like |
|
@wangyum I do not think we should deprecate the support of the previous versions of Hive metastore. Many Spark users are still using them. |
|
@dongjoon-hyun Thanks for your investigation! |
|
@dongjoon-hyun @wangyum @gatorsmile @HyukjinKwon Anything left on this? can it be merged to master? |
|
Also, can hive 3.1 be supported easily or are there some breaking changes? |
| case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1 | ||
| case "2.2" | "2.2.0" => hive.v2_2 | ||
| case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" => hive.v2_3 | ||
| case "3.0" | "3.0.0" => hive.v3_0 |
There was a problem hiding this comment.
@dongjoon-hyun Please update sql-programming-guide.md and HiveUtils.scala:
spark/docs/sql-programming-guide.md
Line 1217 in 05974f9
|
Thank you for review, @tooptoop4 and @wangyum . |
|
bump |
|
@tooptoop4 . Since this is a new feature, it's now targeting for Apache Spark 2.5 because |
|
@dongjoon-hyun I was planning to do my own custom build by cherrypicking your PR if you had it available |
|
That would be helpful for us in the end. But, sorry for now. I'm currently not planning it open soon because the next release (Apache Spark 2.5 or 3.0) will be next year. |
|
Hi, All. |
6 participants
What changes were proposed in this pull request?
Hive 3.0 is released. This PR aims to support Hive 3.0 metastore.
How was this patch tested?
Pass the Jenkins with the updated test cases including 3.0.