SPARK-3329: [SQL] Don't depend on Hive SET pair ordering.

[willb]

This fixes some possible spurious test failures in HiveQuerySuite by comparing sets of key-value pairs as sets, rather than as lists.

[concretevitamin]

ok to test

[SparkQA]

QA tests have started for PR 2220 at commit 36ff52a.

• This patch merges cleanly.

[concretevitamin]

Ah, so this problem was fixed in PR #1514, but it seems like this later PR reverted the change accidentally. I think it'd be good to re-adapt 1514's solution.

/cc @aarondav @liancheng

[willb]

Thanks, @concretevitamin!

[SparkQA]

QA tests have finished for PR 2220 at commit 36ff52a.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[willb]

@concretevitamin I cherry-picked @aarondav's fix (and added a very simple fix to handle cases that it didn't).

[SparkQA]

QA tests have started for PR 2220 at commit 6525d8e.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2220 at commit 6525d8e.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Minor style nit: this could just be: case Row(KV(key,value)) => ... I believe.

[marmbrus]

Hey @willb, thanks for looking into / fixing this! Minor pattern matching suggestion only.

[SparkQA]

QA tests have started for PR 2220 at commit 3b3e205.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2220 at commit 3b3e205.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[willb]

This failure (in SparkSubmitSuite) appears unrelated to my patch.

[SparkQA]

QA tests have started for PR 2220 at commit 3b3e205.

• This patch merges cleanly.

[SparkQA]

Tests timed out after a configured wait of 120m.

[marmbrus]

Thanks for cleaning this up! Since this passed tests before I'm going to merge to master.

[JoshRosen]

@marmbrus @liancheng Can we backport this into branch-1.1 (1.1.2)? I've been observing a lot of flakiness in the "HiveQuerySuite.SET commands semantics for a HieContext" suite in branch-1.1 and it sounds like this patch might fix it. There are a couple of merge conflicts, but I think they should be simple to fix.