-
Couldn't load subscription status.
- Fork 260
Description
Is your feature request related to a problem? Please describe.
apache/spark#30297 introduce different behaviors for element_at, getArrayItem and getMapValue when invalid index or key is given to lhs with ansi mode enabled and disabled.
For select element_at(Array(1,2,3), 4):
Spark 3.0.2 + spark.sql.ansi.enabled=true:
+-----------------------------+
|element_at(array(1, 2, 3), 4)|
+-----------------------------+
| null|
+-----------------------------+
Spark 3.1.1 + spark.sql.ansi.enabled=true:
java.lang.ArrayIndexOutOfBoundsException: Invalid index: 4, numElements: 3
For select element_at(map(1, 'a', 2, 'b'), 3):
Spark 3.0.2 + spark.sql.ansi.enabled=true:
+------------------------------+
|element_at(map(1, a, 2, b), 3)|
+------------------------------+
| null|
+------------------------------+
Spark 3.1.1 + spark.sql.ansi.enabled=true:
java.util.NoSuchElementException: Key 3 does not exist.
Initial support for elementAt, getArrayItem will always return null for invalid index or key.
Describe the solution you'd like
Put corresponding code into 3.1.1 shim layers.
- GetArrayItem
- GetMapValue
- ElementAt
Additional context
Initial PR: #2260