-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-17177][SQL] Make grouping columns accessible from RelationalGroupedDataset
#14742
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-17177][SQL] Make grouping columns accessible from RelationalGroupedDataset
#14742
Conversation
|
Test build #64169 has finished for PR 14742 at commit
|
|
cc: @shivaram, @liancheng |
|
@liancheng, @rxin, Do you think adding |
| * | ||
| * @since 2.1.0 | ||
| */ | ||
| def columns: Array[String] = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might not be a column (a named expression). Thus, this might not be useful to end users.
|
Could we please close this PR now? |
|
Test build #78035 has started for PR 14742 at commit |
|
Test FAILed. |
|
Hi @gatorsmile, #14431 depends on this. Is there a way I can access the grouping columns from |
|
Let us close this one. We can discuss how to resolve the issue in #14431 |
|
yes, we can close this, but it would be great if you could help us to find a way to access the grouping columns from SparkR in #14431 |
What changes were proposed in this pull request?
Currently, once we create
RelationalGroupedDataset, we cannot access the grouping columns from its instance.Analog to
Datasetwe can have a public method which returns the list of grouping columns.https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L457
This can be useful for instance in SparkR when we want to have certain logic associated with the grouping columns, accessible from
RelationalGroupedDataset.Similar to
Dataset.columnsI've addedRelationalGroupedDataset.columnsmethod which makes grouping column names accessible.How was this patch tested?
Unit tests in
DataFrameAggregateSuite