[SPARK-15327] [SQL] fix split expression in whole stage codegen #13235
Conversation
|
Test build #59041 has finished for PR 13235 at commit
|
| |"a": [{"e": "testing", "count": 3}], "b": [{"e": "test", "count": 1}]}}, | ||
| |"f": {"b": {"c": [{"e": "adfgd"}], "a": [{"e": "testing", "count": 3}], | ||
| |"b": [{"e": "test", "count": 1}]}}, "b": {"b": {"c": [{"e": "adfgd"}], | ||
| |"a": [{"count": 3}], "b": [{"e": "test", "count": 1}]}}}' |
|
Thanks! Not sure if I understand it correctly, but what happens when the whole-stage codegen generates code longer then 64k? Because I thought about fixing this issue by passing some of the variables to the generated functions (but was not sure how to do that exactly). |
|
hm this pull request outlines a problem with whole stage codegen, i.e. it is too convoluted at the moment and needs cleanup. Looking at the diff it is difficult to understand why this fixes a problem. We should still fix it, but really refactor and clean things up soon. |
|
It looks like #12351 is the same issue about whole stage codegen with |
|
LGTM |
|
Test build #59679 has finished for PR 13235 at commit
|
|
Merging this into master and 2.0, thanks! |
## What changes were proposed in this pull request? Right now, we will split the code for expressions into multiple functions when it exceed 64k, which requires that the the expressions are using Row object, but this is not true for whole-state codegen, it will fail to compile after splitted. This PR will not split the code in whole-stage codegen. ## How was this patch tested? Added regression tests. Author: Davies Liu <[email protected]> Closes #13235 from davies/fix_nested_codegen. (cherry picked from commit 2df6ca8) Signed-off-by: Davies Liu <[email protected]>
7 participants
What changes were proposed in this pull request?
Right now, we will split the code for expressions into multiple functions when it exceed 64k, which requires that the the expressions are using Row object, but this is not true for whole-state codegen, it will fail to compile after splitted.
This PR will not split the code in whole-stage codegen.
How was this patch tested?
Added regression tests.