[SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation: Constant Pool Limit - Class Splitting

[bdrillard]

What changes were proposed in this pull request?

This is a backport patch for Spark 2.2.x of the class splitting feature over excess generated code as was merged in #18075.

How was this patch tested?

The same test provided in #18075 is included in this patch.

[cloud-fan]

ok to test

[SparkQA]

Test build #78393 has finished for PR 18377 at commit 8998ee5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to 2.2!

[sameeragarwal]

@bdrillard @cloud-fan given that this is not a regression and 2.2 is in code-freeze, do you think this is safe to merge it in the 2.2 branch? More specifically, could the new code path affect (or regress) classes that would otherwise not grow beyond 1600k limit? Thanks!

[bdrillard]

@sameeragarwal @cloud-fan If I understand well what you mean by affect/regress, this patch would induce no change at all on the code generated for any class below the 1600k limit. For such classes smaller than the limit, all functions are inlined to the OuterClass, which the declareAddedFuctions method now references. Structurally, there would be no change to classes with code generated below the threshold. I don't know if this explanation suffices to address your concerns, but I'd say the patch is pretty passive in that code generation is only affected for cases over the 1600k threshold.

[rxin]

Why did we backport this? This seems too risky.

[cloud-fan]

as we are very close to 2.2, and this patch is not trivial(200+ lines), I'm going to revert it and merge it back after 2.2 release, so this patch can be in 2.2.1 and we can have plenty of time to test it. cc @bdrillard what do you think?

[cloud-fan]

reverted, @bdrillard ping me if you have any other concerns, thanks!

[bdrillard]

@cloud-fan Sure! That makes perfect sense to me.

[sameeragarwal]

Thanks guys!

[rxin]

Hm I'm not even sure if we should backport this in branch-2.2. Let's wait and see ...

[bdrillard]

@cloud-fan @rxin @sameeragarwal What can we say is the status of these backports? Including this PR, #18354, and #18579, it was decided to revert the changes until the 2.2.0 release was finished. Will all the backports remain reverted? Or can we find a way to get these merged? It seems some people would find backports useful (see SPARK-18016 jira discussion), but naturally I'd want to make sure the patches fit your testing requirements/release schedule.

[gatorsmile]

@bdrillard Sorry, we do not plan to backport this PR to the maintenance branch 2.2. Could you please close this PR? Thanks!