Skip to content

feat: Upgrade to DataFusion 40.0.0-rc1 #644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Jul 9, 2024
Merged

feat: Upgrade to DataFusion 40.0.0-rc1 #644

merged 14 commits into from
Jul 9, 2024

Conversation

@andygrove
Copy link
Member

@andygrove andygrove commented Jul 8, 2024 •
edited
Loading

Which issue does this PR close?

N/A

Rationale for this change

DataFusion is preparing to release version 40.0.0 so I would like to make sure that we are testing with the latest prior to the release.

What changes are included in this PR?

Use a very recent revision of the official DataFusion repo.

How are these changes tested?

@andygrove andygrove changed the title feat: Upgrade to DataFusion 40.0.0 feat: Upgrade to latest DataFusion in preparation for upgrading to 40.0.0 Jul 8, 2024
@andygrove
Copy link
Member Author

There is a test failure:

- Windows support *** FAILED *** (224 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1241.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1241.0 (TID 4084) (192.168.64.28 executor driver): org.apache.comet.CometNativeException: General execution error with reason: count not supported for window function.

@viirya
Copy link
Member

viirya commented Jul 8, 2024

There is a test failure:

- Windows support *** FAILED *** (224 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1241.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1241.0 (TID 4084) (192.168.64.28 executor driver): org.apache.comet.CometNativeException: General execution error with reason: count not supported for window function.

I recall that count, max, min are supported as window function recently. Maybe there are some changes in DataFusion.

cc @huaxingao

@huaxingao
Copy link
Contributor

I recall that count, max, min are supported as window function recently. Maybe there are some changes in DataFusion.

maybe this PR? I need to take a closer look.

@andygrove andygrove changed the title feat: Upgrade to latest DataFusion in preparation for upgrading to 40.0.0 feat: Upgrade to DataFusion 40.0.0-rc1 Jul 9, 2024
@andygrove
Copy link
Member Author

I recall that count, max, min are supported as window function recently. Maybe there are some changes in DataFusion.

maybe this PR? I need to take a closer look.

We rely on find_df_window_func to find window functions, and count was removed from the built-in functions in apache/datafusion#10893

@andygrove andygrove marked this pull request as ready for review July 9, 2024 14:10
@andygrove
Copy link
Member Author

andygrove commented Jul 9, 2024 •
edited
Loading

@viirya @huaxingao I made a change so that we fall back to Spark for count in window aggregate for now, and I will file a follow on issue. This PR is ready for review now.

edit: I filed #645

@andygrove andygrove mentioned this pull request Jul 9, 2024
Closed
viirya
viirya reviewed Jul 9, 2024
View reviewed changes
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
expr match {
case agg: AggregateExpression =>
agg.aggregateFunction match {
case _: Min | _: Max | _: Count =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This disables count for both aggregation and window?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, nvm, I checked the source code. This is only for window functions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a comment here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

@andygrove andygrove merged commit 15e7baa into apache:main Jul 9, 2024
@andygrove andygrove deleted the update-df-40 branch July 9, 2024 21:07
himadripal pushed a commit to himadripal/datafusion-comet that referenced this pull request Sep 7, 2024
@andygrove @viirya
feat: Upgrade to DataFusion 40.0.0-rc1 (apache#644)
250c324 
* Partial upgrade to DataFusion 40.0.0

* fix

* implement more udaf

* update bitwise agg

* add func names

* remove unused imports

* remove arrow-string dep

* fix copy and paste error

* use 40.0.0-rc1 and temporarily ignore failing test

* clippy

* fall back to Spark for count windows aggregate

* address feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@viirya viirya viirya approved these changes

@kazuyukitanimura kazuyukitanimura kazuyukitanimura approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove @viirya @huaxingao @kazuyukitanimura