Skip to content

[FEA] elementAt, getArrayItem and getMapValue have different behaviors since Spark-3.1.1 when honoring AnsiMode #2272

@wjxiz1992

Description

@wjxiz1992

Is your feature request related to a problem? Please describe.
apache/spark#30297 introduce different behaviors for element_at, getArrayItem and getMapValue when invalid index or key is given to lhs with ansi mode enabled and disabled.

For select element_at(Array(1,2,3), 4):

Spark 3.0.2 + spark.sql.ansi.enabled=true:

+-----------------------------+
|element_at(array(1, 2, 3), 4)|
+-----------------------------+
|                         null|
+-----------------------------+

Spark 3.1.1 + spark.sql.ansi.enabled=true:

java.lang.ArrayIndexOutOfBoundsException: Invalid index: 4, numElements: 3

For select element_at(map(1, 'a', 2, 'b'), 3):

Spark 3.0.2 + spark.sql.ansi.enabled=true:

+------------------------------+
|element_at(map(1, a, 2, b), 3)|
+------------------------------+
|                          null|
+------------------------------+

Spark 3.1.1 + spark.sql.ansi.enabled=true:

java.util.NoSuchElementException: Key 3 does not exist.

Initial support for elementAt, getArrayItem will always return null for invalid index or key.

Describe the solution you'd like
Put corresponding code into 3.1.1 shim layers.

  • GetArrayItem
  • GetMapValue
  • ElementAt

Additional context
Initial PR: #2260

Metadata

Metadata

Assignees

Labels

taskWork required that improves the product but is not user facing

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions