Skip to content

Commit 13ff475

Browse files
author
assafmendelson
committed
Additional File source options for spark 2.2 appeared under File sink
1 parent f804458 commit 13ff475

File tree

1 file changed

+9
-8
lines changed

1 file changed

+9
-8
lines changed

docs/structured-streaming-programming-guide.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -515,6 +515,14 @@ Here are the details of all the sources in Spark.
515515
<br/>
516516
<code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files (default: false)
517517
<br/>
518+
<code>fileNameOnly</code>: whether to check new files based on only the filename instead of on the full path (default: false). With this set to `true`, the following files would be considered as the same file, because their filenames, "dataset.txt", are the same:
519+
<br/>
520+
· "file:///dataset.txt"<br/>
521+
· "s3://a/dataset.txt"<br/>
522+
· "s3n://a/b/dataset.txt"<br/>
523+
· "s3a://a/b/c/dataset.txt"<br/>
524+
<br/>
525+
518526
<br/>
519527
For file-format-specific options, see the related methods in <code>DataStreamReader</code>
520528
(<a href="api/scala/index.html#org.apache.spark.sql.streaming.DataStreamReader">Scala</a>/<a href="api/java/org/apache/spark/sql/streaming/DataStreamReader.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.streaming.DataStreamReader">Python</a>/<a
@@ -1239,14 +1247,7 @@ Here are the details of all the sinks in Spark.
12391247
<td>Append</td>
12401248
<td>
12411249
<code>path</code>: path to the output directory, must be specified.
1242-
<br/>
1243-
<code>fileNameOnly</code>: whether to check new files based on only the filename instead of on the full path (default: false). With this set to `true`, the following files would be considered as the same file, because their filenames, "dataset.txt", are the same:
1244-
<br/>
1245-
· "file:///dataset.txt"<br/>
1246-
· "s3://a/dataset.txt"<br/>
1247-
· "s3n://a/b/dataset.txt"<br/>
1248-
· "s3a://a/b/c/dataset.txt"<br/>
1249-
<br/>
1250+
<br/><br/>
12501251
For file-format-specific options, see the related methods in DataFrameWriter
12511252
(<a href="api/scala/index.html#org.apache.spark.sql.DataFrameWriter">Scala</a>/<a href="api/java/org/apache/spark/sql/DataFrameWriter.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter">Python</a>/<a
12521253
href="api/R/write.stream.html">R</a>).

0 commit comments

Comments
 (0)