Skip to content

Commit f70d069

Browse files
committed
Adding docs for spark.serializer.objectStreamReset configuration
1 parent 7ccc74b commit f70d069

File tree

1 file changed

+11
-0
lines changed

1 file changed

+11
-0
lines changed

docs/configuration.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,17 @@ Apart from these, the following properties are also available, and may be useful
237237
exceeded" exception inside Kryo. Note that there will be one buffer <i>per core</i> on each worker.
238238
</td>
239239
</tr>
240+
<tr>
241+
<td>spark.serializer.objectStreamReset</td>
242+
<td>10000</td>
243+
<td>
244+
When serializing using org.apache.spark.serializer.JavaSerializer, the serializer caches
245+
objects to prevent writing redundant data, however that stops garbage collection of those
246+
objects. By calling 'reset' you flush that info from the serializer, and allow old
247+
objects to be collected. To turn off this periodic reset set it to a value of <= 0.
248+
By default it will reset the serializer every 10,000 objects.
249+
</td>
250+
</tr>
240251
<tr>
241252
<td>spark.broadcast.factory</td>
242253
<td>org.apache.spark.broadcast.<br />HttpBroadcastFactory</td>

0 commit comments

Comments
 (0)