DuckDBConnection.preparedStatements is never pruned

[mediapartners]

The array of weakrefs grows with each query, even if the statement gets collected the array will keep growing.

[jraymakers]

Did you observe this happening at runtime, or are you inferring from reading the code?

If the latter, you may have missed the call to prune whenever a new prepared statement is added the collection. This ensures that, before each new weakref is added, any stale ones are removed from the array.

[mediapartners]

@jraymakers both. I observed the array growing when I did console.log(connection) you have to be using parameter binding or creating a prepared statement manually.

[jraymakers]

Finally ran some tests. I observed the list being pruned, but it only happens after the garbage collector runs and collects the prepared statements objects behind the weakrefs. When this happens is unpredictable. Even if the process is idle, it may not happen for several seconds. If the process is busy, it may take longer, or not happen at all.

In a test generating prepared statements in a tight loop, I've observed the list growing to thousands or tens of thousands before getting pruned. It seem inserting an occasional 1ms sleep can make the difference between giving the collector a chance to run.

Given how much time can pass (and how many prepared statements can be created) before the prune succeeds, attempting to prune every time a new prepared statement is added is likely inefficient. I can throttle that a bit to avoid the cost of the check.

Have you observed any negative side effects of this list getting long? If so, inserting an occasional short (1ms) sleep might help. That may be a good idea anyway to ensure the collector gets a chance.

[jraymakers]

Here's the change to throttle the pruning: #268

(This will go out with the next release.)

[jraymakers]

Throttling of pruning released in 1.3.2-alpha.26.