Release DataFusion `44.0.0`

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/43.0.0 November 8th, 2024 so next major release would be around December 8, 2024

• Voting Thread: https://lists.apache.org/thread/78zxfpn7fh1csz6fdolpgh3lb4ltxngp

Prior release tickets:

• 43.0.0 Release DataFusion 43.0.0 #12470

Items to possibly fix before release

• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510
• [Epic] A Collection of Additional UTF8View support tickets #13504

Improved upgrade experience:

• Make migration to Signature::nullary in 44.0.0 easier / less confusing #13763
• Make DocBuilder migration in 44.0.0 easier #13764
• Making the recursive dependency an optional feature #13766
• Deprecate ScalarUDFImpl::return_type #13717

Downstream project testing:

• datafusion python: chore: Upgrade to DataFusion 44 datafusion-python#972
• comet: Test DataFusion 44.0.0 with Comet #13835
• delta.rs: Test DataFusion 44.0.0 with delta.rs #13834
• Test DataFusion 44.0.0 with Sail #13855

[alamb]

I would love to get this one in:

• Perf: Allow User defined functions to potentially reuse their argument arrays (to avoid new allocations) #13516

Some things to highlight (that are already merged)

• Support vectorized append and compare for multi group by #12996 from @Rachelint as well

[alamb]

Also this hopefully:

• Add SessionConfig reference to ScalarFunctionArgs #13519

[alamb]

@Omega359 and @andygrove suggested #13525 (comment) that for this release we

we file tickets as part of every release to test DF with latest versions of common external dependencies (datafusion-python, iceberg, lance, delta, etc) as part of the gating criteria for a release. Perhaps this would be as simple as changing the DF version in those projects and running their test suite. Any blockers that are found in DF have tickets filed and are fixed, other blockers in the respective projects have tickets filed in those projects

Perhaps for 44.0.0 we can try with own subprojects (Ballista, Comet, DF Python, DF Ray)

[alamb]

I started gathering a list of items we think we should fix prior to the release in the description

[alamb]

Does anyone have any opinion about holding the DataFusion 44 release for the next major arrow release?

• Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342

That would fix a bunch of the StringView issues, but would likely delay DataFusion 44 for a few more weeks

[Omega359]

It's soon to be the holiday season so I'm all for cooking the release a bit more.

[alamb]

I would personally love to see DataFusion 44.0.0 be lauded as "super stable" and have few upgrade issues (we would largely achieve this by testing upgrades with other projects prior to release)

The downside is that the changeset in DataFusion 44.0.0 might be larger

[andygrove]

I'd like to see an upgrade guide for the 44 release (and am willing to take the lead on this). I am trying to upgrade Comet now and am running into some issues.

I am +1 for waiting for the next Arrow release.

[andygrove]

I filed #13702

[andygrove]

How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?

Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.