Skip to content

Commit c2daee4

Browse files
HyukjinKwonrshkv
authored andcommitted
[SPARK-27276][PYTHON][DOCS][FOLLOW-UP] Update documentation about Arrow version in PySpark as well
## What changes were proposed in this pull request? Looks updating documentation from 0.8.0 to 0.12.1 was missed. ## How was this patch tested? N/A Closes apache#24504 from HyukjinKwon/SPARK-27276-followup. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Bryan Cutler <[email protected]>
1 parent c33f870 commit c2daee4

File tree

1 file changed

+3
-4
lines changed

1 file changed

+3
-4
lines changed

docs/sql-pyspark-pandas-with-arrow.md

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ displayTitle: PySpark Usage Guide for Pandas with Apache Arrow
77
* Table of contents
88
{:toc}
99

10-
## Apache Arrow in Spark
10+
## Apache Arrow in PySpark
1111

1212
Apache Arrow is an in-memory columnar data format that is used in Spark to efficiently transfer
1313
data between JVM and Python processes. This currently is most beneficial to Python users that
@@ -20,7 +20,7 @@ working with Arrow-enabled data.
2020

2121
If you install PySpark using pip, then PyArrow can be brought in as an extra dependency of the
2222
SQL module with the command `pip install pyspark[sql]`. Otherwise, you must ensure that PyArrow
23-
is installed and available on all cluster nodes. The current supported version is 0.8.0.
23+
is installed and available on all cluster nodes. The current supported version is 0.12.1.
2424
You can install using pip or conda from the conda-forge channel. See PyArrow
2525
[installation](https://arrow.apache.org/docs/python/install.html) for details.
2626

@@ -128,8 +128,7 @@ For detailed usage, please see [`pyspark.sql.functions.pandas_udf`](api/python/p
128128
### Supported SQL Types
129129

130130
Currently, all Spark SQL data types are supported by Arrow-based conversion except `MapType`,
131-
`ArrayType` of `TimestampType`, and nested `StructType`. `BinaryType` is supported only when
132-
installed PyArrow is equal to or higher than 0.10.0.
131+
`ArrayType` of `TimestampType`, and nested `StructType`.
133132

134133
### Setting Arrow Batch Size
135134

0 commit comments

Comments
 (0)