Test DataFusion 44.0.0 with Sail

[shehabgamin]

Is your feature request related to a problem or challenge?

• Part of of [DISCUSS] More extensive pre-release testing #13661

We are trying to make sure the upgrade experience for DataFusion is better and to do so we are hoping to improve the testing of major downstream projects

For version 44.0.0 we would like to test with Sail prior to release.

Describe the solution you'd like

I would like to create a PR to test DataFusion Sail with a pre-released version of DataFusion (pinned to the main branch)

Describe alternatives you've considered

No response

Additional context

lakehq/sail#335

[shehabgamin]

Take

[shehabgamin]

Smooth Sailing testing commit 5d563d9 from DataFusion main branch.

The following test reports run tests on the PR branch and compares them against the tests from the Sail main branch:

• DataFusion 44.0.0 lakehq/sail#335 (comment)
• DataFusion 44.0.0 lakehq/sail#335 (comment)

Will re-test once there is a release candidate.

[Omega359]

Smooth Sailing testing commit 5d563d9 from DataFusion main branch.

The following test reports run tests on the PR branch and compares them against the tests from the Sail main branch:

* [DataFusion 44.0.0 Pre-Release Testing lakehq/sail#335 (comment)](https://github.com/lakehq/sail/pull/335#issuecomment-2556353768)

* [DataFusion 44.0.0 Pre-Release Testing lakehq/sail#335 (comment)](https://github.com/lakehq/sail/pull/335#issuecomment-2556416669)

After a brief review of the errors I suspect there may be something up with lists and structs but I would have to see the actual logic being tested to know for certain. Many of the errors are expected - functions not existing, etc.

[alamb]

After a brief review of the errors I suspect there may be something up with lists and structs but I would have to see the actual logic being tested to know for certain. Many of the errors are expected - functions not existing, etc.

I feel like we have had several bugs / issues with structs / lists the last few releases so it would be amazing if we could figure out what is going on before the release. @shehabgamin is this something you can help with?

[shehabgamin]

After a brief review of the errors I suspect there may be something up with lists and structs but I would have to see the actual logic being tested to know for certain. Many of the errors are expected - functions not existing, etc.

I feel like we have had several bugs / issues with structs / lists the last few releases so it would be amazing if we could figure out what is going on before the release. @shehabgamin is this something you can help with?

@Omega359 @alamb For sure, I will gladly take a look! If there are issues with lists and/or structs, it likely wouldn't be due to changes in the 44.0.0 pre-release that I tested, though. The Sail main branch and the PR branch that is testing the pre-release have the exact same errors (with the exception of slight rewording of some error logs).

This leads me to believe that any issues with lists and/or structs would likely be preexisting. I will take a close look at all of the errors, but, with that in mind, should we scope the issues into a separate GitHub issue if we do find errors? Also @Omega359, let me know if any errors in particular caught your eye.

[alamb]

This leads me to believe that any issues with lists and/or structs would likely be preexisting. I will take a close look at all of the errors, but, with that in mind, should we scope the issues into a separate GitHub issue if we do find errors? Also @Omega359, let me know if any errors in particular caught your eye.

That would make sense. Glad to hear the errors aren't new

[shehabgamin]

Smooth Sailing testing commit 073a3b1 from DataFusion Release Candidate branch (https://github.com/apache/datafusion/tree/073a3b110852f97ccb7085ce4bfd19473b8a3f4f).

The following test reports run tests on the PR branch and compares them against the tests from the Sail main branch:

• DataFusion 44.0.0 lakehq/sail#335 (comment)
• DataFusion 44.0.0 lakehq/sail#335 (comment)

Note: Additional errors in Test 2 are expected. These occur because code using the datafusion-functions-json crate (from datafusion-contrib) had to be commented out. This was necessary since the crate depends on DataFusion v43, while the current release candidate uses v44. This wasn't an issue in Test 1 because the commit from main hadn't yet been updated to version v44.

[shehabgamin]

Closing this issue now, but please feel free to re-open if necessary @alamb.

Also, @Omega359 please let me know regarding our earlier discussion (linked below). I haven't taken a look yet because of the holidays, but will create a new issue if I discover anything.
#13855 (comment)

Happy Holidays!

[alamb]

Thank you very much @shehabgamin for the help and assistance. I love it when incentives are aligned like this

[Omega359]

Also, @Omega359 please let me know regarding our earlier discussion (linked below). I haven't taken a look yet because of the holidays, but will create a new issue if I discover anything. #13855 (comment)

Thanks for reminding me about this - I've added it to my TODO list and I'll do my best to try and get to this in the next week.