[WIP] Add spark 35 #127
Closed
[WIP] Add spark 35 #127
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…mports from either place it can be in 3.4/3.5 respectively.
…ns of Spark stuff, add some notes to development about formatting.
Contributor
|
Some class and function has changed in spark 3.5, maybe we need some more shim classes to support both spark 3.5 and the versions before. I has a similar commit to do this: |
Contributor
Author
|
Yup this adds some shims (there are existing shims too for 3.3 etc but they use reflection heavily). |
Member
|
FWIW I created #140 to improve the current Spark shim handling. |
Contributor
Author
|
I love it :)
…On Thu, Feb 29, 2024 at 10:45 AM Chao Sun ***@***.***> wrote:
FWIW I created #140
<#140> to improve
the current Spark shim handling.
—
Reply to this email directly, view it on GitHub
<#127 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAOT5NRC355APXEYYRFIADYV53MJAVCNFSM6AAAAABD5DERESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRG42DCNRUGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Contributor
|
FYI #318 is just merged |
Member
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Adds Spark 3.5
Rationale for this change
Spark 3.5 is the latest Spark release.
What changes are included in this PR?
Adds Spark 3.5 support, this introduces a new per-spark version of shims as part of adding interfacing to the 3.5 APIs.
This has the advantage of static compile checks over the existing shim approach which does reflection for everything, but I can switch to the runtime reflection model for consistency if that's preferred.
How are these changes tested?
WIP