[SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage #15227
Conversation
|
Can one of the admins verify this patch? |
There was a problem hiding this comment.
Hi, the goal of this PR looks interesting but the approach needs more explanation.
Could you please elaborate on the semantics of this new isWholeStageResultVar argument, give an example of how that affects the code in GenerateUnsafeProjection, and why is GenerateUnsafeProjection.createCode() the only place you're applying this new feature?
| initCode: String, isWholeStageResultVar: Boolean = false): Unit = { | ||
| if (isWholeStageResultVar) { | ||
| mutableStates = mutableStates.filterNot(state => (state._1 == javaType) && state._4) | ||
| mutableStates += ((javaType, variableName, initCode, isWholeStageResultVar)) |
There was a problem hiding this comment.
So you're effectively letting later-added states to override earlier-added ones that are equivalent and replaceable. Could that be done in a add-if-absent way instead?
Also, what's the criteria for a state to be "replaceable" in your design? Could you please make that explicit as comment in the code?
|
Could you please add the simple generated code in the description before and after applying this PR? |
|
gentle ping @yaooqinn on ^. |
5 participants
What changes were proposed in this pull request?
A WholeStageCodeGened stage with multiple CodegenSupport Operators generates unused result rows and their associated buffer holders and row writers, which can be removed.
How was this patch tested?
existing ut.
cc @davies @rxin @srowen