From 0f16db95c2beb461ed0606bc9427dd5d0b094a5f Mon Sep 17 00:00:00 2001 From: Yin Huai Date: Tue, 8 Dec 2015 16:44:42 -0800 Subject: [PATCH] Document basePath in the programming guide. --- docs/sql-programming-guide.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index 7b1d97baa3823..3c24e6ef54a99 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -1161,6 +1161,13 @@ infer the data types of the partitioning columns. For these use cases, the autom can be configured by `spark.sql.sources.partitionColumnTypeInference.enabled`, which is default to `true`. When type inference is disabled, string type will be used for the partitioning columns. +Starting from Spark 1.6.0, partition discovery only finds partitions under the given paths +by default. For the above example, if users pass `path/to/table/gender=male` to either +`SQLContext.read.parquet` or `SQLContext.read.load`, `gender` will not be considered as a +partitioning column. If users need to specify the base path that partition discovery +should start with, they can set `basePath` in the data source options. For example, +when `path/to/table/gender=male` is the path of the data and +users set `basePath` to `path/to/table/`, `gender` will be a partitioning column. ### Schema Merging