[SPARK-38350][SQL] Make the table name output of V1/V2 "desc extended table" consistent #35681
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
| rows += toCatalystRow("# Detailed Table Information", "", "") | ||
| rows += toCatalystRow("Name", table.name(), "") | ||
| rows += toCatalystRow("Catalog", resolvedTable.catalog.name(), "") | ||
| rows += toCatalystRow("Database", resolvedTable.identifier.namespace().mkString("."), "") |
There was a problem hiding this comment.
is namespace() returning a list? Should this be a single identifier?
Member
Author
There was a problem hiding this comment.
Yes, namespace() returns a list. It's a feature of DS V2. Usually the length should be 1.
There was a problem hiding this comment.
so if it returns a list will the Database name contain periods?
Member
Author
There was a problem hiding this comment.
If the length of the namespace is 1, there is no any period
There was a problem hiding this comment.
we spoke online with Wenchen and Gengliang, didn't realize the namespace is a multi part name.
| " COMMENT 'this is a test table'" + | ||
| " LOCATION 'file:/tmp/testcat/table_name'") | ||
| val descriptionDf = spark.sql("DESCRIBE TABLE EXTENDED testcat.table_name") | ||
| val descriptionDf = spark.sql("DESCRIBE TABLE EXTENDED testcat.default.table_name") |
There was a problem hiding this comment.
Does this also fix SHOW TABLE EXTENDED?
Member
Author
There was a problem hiding this comment.
There is no V2 version of SHOW TABLE EXTENDED yet.
gengliangwang
changed the title
cloud-fan
approved these changes
Mar 7, 2022
Member
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The V1 "DESC TABLE EXTENDED" command contains info of "Database" and "Table", which can be used by external tools like DBT: #17394
However, the V2 version contains only one field "name" representing "catalog.database.table". External tools can't recognize it.
Also, it is weird to have different command outputs from the same command.
This PR is to make the database/table name output of V1/V2 "desc extended table" consistent. For the output of V2 tables, this PR also shows the catalog name in the output.
Why are the changes needed?
Make the database/table name output of V1/V2 "desc extended table" consistent so that it can be parsed by external tools like DBT.
Does this PR introduce any user-facing change?
Yes, it changes the database/table name output of V2 "desc extended table" so that it becomes consistent with V1 tables.
How was this patch tested?
UT