Skip to content

[SPARK-10671] [SQL] Throws an analysis exception if we cannot find Hive UDFs #8800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-10671] [SQL] Throws an analysis exception if we cannot find Hive UDFs #8800

wants to merge 2 commits into from

Conversation

@yhuai
Copy link
Contributor

@yhuai yhuai commented Sep 17, 2015

yhuai
yhuai reviewed Sep 17, 2015
View reviewed changes
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala Outdated
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is for testing.

@SparkQA
Copy link

SparkQA commented Sep 17, 2015

Test build #42616 has finished for PR 8800 at commit 604dc33.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor

Currently saveAsTable tries its best to save the table as Hive compatible format, and falls back to Spark SQL specific format if it fails. I wonder whether it's too confusing for users to reason about behavior of saveAsTable. Would it be better to have a separate SQL option to force using Hive compatible format? With this option on, users can be quite confident that if a saveAsTable call succeeds, the saved table must be Hive compatible.

@yhuai
Copy link
Contributor Author

yhuai commented Sep 17, 2015

The impact of saving the metadata in a hive compatible way or not will appear when a user want to read the data. I am not sure they can remember how the metadata is stored. It is also very possible that the data producer and consumer are different persons/different teams.

@SparkQA
Copy link

SparkQA commented Sep 17, 2015

Test build #42617 has finished for PR 8800 at commit 5874854.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 18, 2015

Test build #1771 has finished for PR 8800 at commit 5874854.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 18, 2015

Test build #42631 has finished for PR 8800 at commit 693939d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai yhuai changed the title [SPARK-10671] [SPARK-10672] [SQL] [WIP] Throws an analysis exception if we cannot find Hive UDFs and do not fail when we cannot save the metadata of a data source table in a hive compatible way [SPARK-10671] [SPARK-10672] [SQL] Throws an analysis exception if we cannot find Hive UDFs and do not fail when we cannot save the metadata of a data source table in a hive compatible way Sep 18, 2015
yhuai
yhuai reviewed Sep 18, 2015
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala Outdated
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, it is not really safe to call dataType to trigger the evaluation of those lazy vals.

@yhuai
Copy link
Contributor Author

yhuai commented Sep 18, 2015

Will split this PR to two PRs.

@SparkQA
Copy link

SparkQA commented Sep 18, 2015

Test build #42672 has finished for PR 8800 at commit e133bee.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai yhuai changed the title [SPARK-10671] [SPARK-10672] [SQL] Throws an analysis exception if we cannot find Hive UDFs and do not fail when we cannot save the metadata of a data source table in a hive compatible way [SPARK-10671] [SQL] Throws an analysis exception if we cannot find Hive UDFs Sep 18, 2015
@SparkQA
Copy link

SparkQA commented Sep 18, 2015

Test build #42679 has finished for PR 8800 at commit e0e1a4d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 18, 2015

Test build #42692 has finished for PR 8800 at commit 4746b82.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@davies
Copy link
Contributor

davies commented Sep 24, 2015

LGTM

cloud-fan
cloud-fan reviewed Sep 28, 2015
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a comment here to say that we do type checking inside hive UDF class(such as HiveSimpleUDF) constructor?

@cloud-fan cloud-fan mentioned this pull request Sep 29, 2015
Closed
asfgit pushed a commit that referenced this pull request Oct 1, 2015
@cloud-fan @yhuai
[SPARK-10671] [SQL] Throws an analysis exception if we cannot find Hi…
02026a8 
…ve UDFs

Takes over #8800

Author: Wenchen Fan <[email protected]>

Closes #8941 from cloud-fan/hive-udf.
@yhuai yhuai closed this Oct 1, 2015
@yhuai yhuai deleted the udfAndDataSource branch October 1, 2015 20:26
kiszk pushed a commit to kiszk/spark-gpu that referenced this pull request Dec 26, 2015
@cloud-fan @kiszk
[SPARK-10671] [SQL] Throws an analysis exception if we cannot find Hi…
09ca8d5 
…ve UDFs

Takes over apache/spark#8800

Author: Wenchen Fan <[email protected]>

Closes #8941 from cloud-fan/hive-udf.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@yhuai @SparkQA @liancheng @davies @cloud-fan