[SPARK-23525] [SQL] Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table #20696
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jiangxb1987
commented
Feb 28, 2018
| // Ensure that change column will preserve other metadata fields. | ||
| sql("ALTER TABLE dbx.tab1 CHANGE COLUMN col1 col1 INT COMMENT 'this is col1'") | ||
| assert(getMetadata("col1").getString("key") == "value") | ||
| assert(getMetadata("col1").getString("comment") == "this is col1") |
Contributor
Author
There was a problem hiding this comment.
I didn't verify the comment is correctly set to the column metadata, and that caused the issue. :(
Contributor
Author
gatorsmile
reviewed
Feb 28, 2018
| // Find the origin column from schema by column name. | ||
| val originColumn = findColumnByName(table.schema, columnName, resolver) | ||
| // Find the origin column from dataSchema by column name. | ||
| val originColumn = findColumnByName(table.dataSchema, columnName, resolver) |
Member
There was a problem hiding this comment.
Only data schema? How about partitioning schema?
Contributor
Author
There was a problem hiding this comment.
As described in the comment, we only support altering non-partition columns:
A command to change the column for a table, only support changing the comment of a non-partition
column for now.
Member
There was a problem hiding this comment.
Do we have a negative test case to cover that?
|
Test build #87786 has finished for PR 20696 at commit
|
|
Test build #87842 has finished for PR 20696 at commit
|
gatorsmile
reviewed
Mar 1, 2018
| -- !query 30 output | ||
|
|
||
| org.apache.spark.sql.AnalysisException | ||
| Invalid column reference 'c', table data schema is 'StructType(StructField(a,IntegerType,true), StructField(b,StringType,true))'; |
Member
There was a problem hiding this comment.
We need a better error message here.
Contributor
There was a problem hiding this comment.
how about
Can't find column `c` given table data columns [`a`, `b`]
Contributor
|
LGTM |
|
Test build #88031 has finished for PR 20696 at commit
|
Contributor
|
thanks, merging to master/2.3! |
asfgit
pushed a commit
that referenced
this pull request
Mar 7, 2018
…rnal hive table ## What changes were proposed in this pull request? The following query doesn't work as expected: ``` CREATE EXTERNAL TABLE ext_table(a STRING, b INT, c STRING) PARTITIONED BY (d STRING) LOCATION 'sql/core/spark-warehouse/ext_table'; ALTER TABLE ext_table CHANGE a a STRING COMMENT "new comment"; DESC ext_table; ``` The comment of column `a` is not updated, that's because `HiveExternalCatalog.doAlterTable` ignores table schema changes. To fix the issue, we should call `doAlterTableDataSchema` instead of `doAlterTable`. ## How was this patch tested? Updated `DDLSuite.testChangeColumn`. Author: Xingbo Jiang <[email protected]> Closes #20696 from jiangxb1987/alterColumnComment. (cherry picked from commit ac76eff) Signed-off-by: Wenchen Fan <[email protected]>
jiangxb1987
added a commit
to jiangxb1987/spark
that referenced
this pull request
Mar 8, 2018
…rnal hive table ## What changes were proposed in this pull request? The following query doesn't work as expected: ``` CREATE EXTERNAL TABLE ext_table(a STRING, b INT, c STRING) PARTITIONED BY (d STRING) LOCATION 'sql/core/spark-warehouse/ext_table'; ALTER TABLE ext_table CHANGE a a STRING COMMENT "new comment"; DESC ext_table; ``` The comment of column `a` is not updated, that's because `HiveExternalCatalog.doAlterTable` ignores table schema changes. To fix the issue, we should call `doAlterTableDataSchema` instead of `doAlterTable`. ## How was this patch tested? Updated `DDLSuite.testChangeColumn`. Author: Xingbo Jiang <[email protected]> Closes apache#20696 from jiangxb1987/alterColumnComment.
tkakantousis
pushed a commit
to tkakantousis/spark
that referenced
this pull request
Mar 11, 2018
…rnal hive table ## What changes were proposed in this pull request? The following query doesn't work as expected: ``` CREATE EXTERNAL TABLE ext_table(a STRING, b INT, c STRING) PARTITIONED BY (d STRING) LOCATION 'sql/core/spark-warehouse/ext_table'; ALTER TABLE ext_table CHANGE a a STRING COMMENT "new comment"; DESC ext_table; ``` The comment of column `a` is not updated, that's because `HiveExternalCatalog.doAlterTable` ignores table schema changes. To fix the issue, we should call `doAlterTableDataSchema` instead of `doAlterTable`. ## How was this patch tested? Updated `DDLSuite.testChangeColumn`. Author: Xingbo Jiang <[email protected]> Closes apache#20696 from jiangxb1987/alterColumnComment.
5 tasks
peter-toth
pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
…rnal hive table ## What changes were proposed in this pull request? The following query doesn't work as expected: ``` CREATE EXTERNAL TABLE ext_table(a STRING, b INT, c STRING) PARTITIONED BY (d STRING) LOCATION 'sql/core/spark-warehouse/ext_table'; ALTER TABLE ext_table CHANGE a a STRING COMMENT "new comment"; DESC ext_table; ``` The comment of column `a` is not updated, that's because `HiveExternalCatalog.doAlterTable` ignores table schema changes. To fix the issue, we should call `doAlterTableDataSchema` instead of `doAlterTable`. ## How was this patch tested? Updated `DDLSuite.testChangeColumn`. Author: Xingbo Jiang <[email protected]> Closes apache#20696 from jiangxb1987/alterColumnComment. (cherry picked from commit ac76eff) Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The following query doesn't work as expected:
The comment of column
ais not updated, that's becauseHiveExternalCatalog.doAlterTableignores table schema changes. To fix the issue, we should calldoAlterTableDataSchemainstead ofdoAlterTable.How was this patch tested?
Updated
DDLSuite.testChangeColumn.