Skip to content

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Sep 22, 2022

What changes were proposed in this pull request?

This PR is a followup of #37850 that removes non-ANSI compliant example in element_at.

Why are the changes needed?

ANSI build fails to run the example.

https://github.com/apache/spark/actions/runs/3094607589/jobs/5008176959

    Caused by: org.apache.spark.SparkArrayIndexOutOfBoundsException: [INVALID_ARRAY_INDEX_IN_ELEMENT_AT] The index -4 is out of bounds. The array has 3 elements. Use `try_element_at` to tolerate accessing element at invalid index and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
    	at org.apache.spark.sql.errors.QueryExecutionErrors$.invalidElementAtIndexError(QueryExecutionErrors.scala:264)
    	...

/usr/local/pypy/pypy3.7/lib-python/3/runpy.py:125: RuntimeWarning: 'pyspark.sql.functions' found in sys.modules after import of package 'pyspark.sql', but prior to execution of 'pyspark.sql.functions'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
/__w/spark/spark/python/pyspark/context.py:310: FutureWarning: Python 3.7 support is deprecated in Spark 3.4.
  warnings.warn("Python 3.7 support is deprecated in Spark 3.4.", FutureWarning)
**********************************************************************
   1 of   6 in pyspark.sql.functions.element_at

Does this PR introduce any user-facing change?

No. The example added is not exposed to end users yet.

How was this patch tested?

Manually tested with enabling the ANSI configuration (spark.sql.ansi.enabled)

@HyukjinKwon
Copy link
Member Author

@gengliangwang mind taking a quick look please? 🙏

Copy link
Contributor

@itholic itholic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HyukjinKwon
Copy link
Member Author

Merged to master.

@HyukjinKwon HyukjinKwon deleted the SPARK-40142-followup branch January 15, 2024 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants