-
Notifications
You must be signed in to change notification settings - Fork 3.9k
ARROW-2799: [Python] Add safe option to Table.from_pandas to avoid unsafe casts #2504
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
There is a spurious but pertinacious conda error: |
|
This happens quite often. Is there a way to configure conda to retry some number of times, or wait and retry? |
|
The 500 errors were a general outage of conda which was resolved. |
|
@kszucs The dependencies were merged. Can you rebase? |
6dfd903 to
ff1c4e5
Compare
|
@xhochy rebased, CI fails because mesonbuild |
xhochy
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM
|
The parquet-cpp merge has caused our merge script to do something pretty wacky. I'm going to try to figure it out |
|
Rebased |
…ted by PyArrow ## What changes were proposed in this pull request? Since 0.11.0, PyArrow supports to raise an error for unsafe cast ([PR](apache/arrow#2504)). We should use it to raise a proper error for pandas udf users when such cast is detected. Added a SQL config `spark.sql.execution.pandas.arrowSafeTypeConversion` to disable Arrow safe type check. ## How was this patch tested? Added test and manually test. Closes apache#22807 from viirya/SPARK-25811. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
Depends on both #2497 and #2503