[SPARK-43883][SQL] Make CTAS Have a UnaryRunnableCommand Trait Supporting Children

[ted-jenks]

What changes were proposed in this pull request?

Make CTAS have a UnaryRunnableCommand trait rather than LeafRunnableCommand node to facilitate them having children (the query) that can be traversed.

Why are the changes needed?

The changes introduced to resolve SPARK-41713 in #39220 modified the CTAS commands from having a DataWritingCommand trait to a LeafRunnableCommand trait. The DataWritingCommand trait extends UnaryCommand, and has children set to the value of query in the CTAS command. This means that when transform is called to traverse the tree with the CTAS command at the root, the entire query is traversed. LeafRunnableCommand has a LeafLike trait which explicitly sets the value of children to Nil. This means that when transform is called on the command, no children are found and the query is unaffected by the rule.

In practice, this means that optimizer rules that rely on transform (such as BooleanSimplification) to traverse the tree do not work with a CTAS.

This can be demonstrated with a simple query in spark-shell. Without the CTAS we can run a command with an easily simplified boolean expression (id == 9 && id == 9) and see it gets optimized out:

With a CTAS, the optimisation does not get applied (as we can see from the AND still present in the optimized and physical plans):

This works in 3.2.0 which had the old CTAS implementation:

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests cover the existing behavior, then adding a test assert that CTAS command nodes are unary.

[ulysses-you]

We run a nested query execution inside ctas, so the real command for writing is V1WriteCommand. Make ctas inherit leaf node is to avoid applying rules more than once, because we would apply rules for the nested query execution.

The explain of ctas does not show the nested query exectuion, you can check that by UI.