You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/structured-streaming-programming-guide.md
+9-8Lines changed: 9 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -515,6 +515,14 @@ Here are the details of all the sources in Spark.
515
515
<br/>
516
516
<code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files (default: false)
517
517
<br/>
518
+
<code>fileNameOnly</code>: whether to check new files based on only the filename instead of on the full path (default: false). With this set to `true`, the following files would be considered as the same file, because their filenames, "dataset.txt", are the same:
519
+
<br/>
520
+
· "file:///dataset.txt"<br/>
521
+
· "s3://a/dataset.txt"<br/>
522
+
· "s3n://a/b/dataset.txt"<br/>
523
+
· "s3a://a/b/c/dataset.txt"<br/>
524
+
<br/>
525
+
518
526
<br/>
519
527
For file-format-specific options, see the related methods in <code>DataStreamReader</code>
@@ -1239,14 +1247,7 @@ Here are the details of all the sinks in Spark.
1239
1247
<td>Append</td>
1240
1248
<td>
1241
1249
<code>path</code>: path to the output directory, must be specified.
1242
-
<br/>
1243
-
<code>fileNameOnly</code>: whether to check new files based on only the filename instead of on the full path (default: false). With this set to `true`, the following files would be considered as the same file, because their filenames, "dataset.txt", are the same:
1244
-
<br/>
1245
-
· "file:///dataset.txt"<br/>
1246
-
· "s3://a/dataset.txt"<br/>
1247
-
· "s3n://a/b/dataset.txt"<br/>
1248
-
· "s3a://a/b/c/dataset.txt"<br/>
1249
-
<br/>
1250
+
<br/><br/>
1250
1251
For file-format-specific options, see the related methods in DataFrameWriter
0 commit comments