QueryBatcher under utilizes threadpool

[aebadirad]

Version: 4.0.3
ML Version: 9.03-1 and 9.04
Java Version: Oracle 9.0.1
OS: Windows 10.1

Input: Passing a list of 1,000 or more URIs to QueryBatcher (can also just DHF quickstart - online store example against a cluster with 16 threads at batches of 50 to see the behavior)

Actual output: Java logs indicate WAITING on multiple threads, initial calls go out as expected, using the full threadcount available to the threadpoolexecutor, but subsequent calls fail to pick up the additional threads resulting in many (usually more than half) being left in a wait state. ML access logs (and app server thread counts) mirror this assessment.

Expected output: When 12 threads are specified, for 12 threads to generally be used until there is nothing left for the threads to grab, and the thread usage dwindles down at the very end with no more batches to run.

Alternatives: I've modified the QueryThreadPoolExecutor constructor on line 796 of QueryBatcherImpl with varying capacities (as well as watching the startIterating and run() methods in conjunction) and it presents the hypothesis that the available queue size (currently set to threadCount * 5) is far too small for the queryBatcher's needs and is spending more time waiting for the queue (rejectedExecutions) to be available than processing the batches.

[sammefford]

@aebadirad could you share an example of the relevant log lines showing the behavior? Also, do you have code and data you can share to reproduce?

[aebadirad]

Extracting this to it's own directory and running gradle hubInit, then supply a valid mlhost/user/pass in gradle.properties and then running gradle --Pprofile doAll will execute the performance-test suite that is used for performancing testing DHF (which relies directly on this as a library):
performance-sample.zip

The threadCount property in the build.gradle file will specify 16 threads per harmonize flow. When run against the current version of DHF (which uses latest released java api client), ML will start with 16 threads on its app server and dwindle down (jstack can be used to grab the java execution thread to show the stack trace).

Hope this helps in creating a reliable way to watch the threadcount. If more 'time' is needed during the tests, an xdmp.sleep(500) can be added to the harmonize flows to stretch out the time.

[sammefford]

The short-term fix was to increase the queue size (will use more memory) in QueryBatcher so the thread stepping through the Iterator and feeding the queue won't be distracted as often running batches because of the use of ThreadPoolExecutor.CallerRunsPolicy. This way that thread can spend more of its time feeding the queue so more threads will stay fully utilized as the queue is less likely to get starved while the Iterator thread is busy.