JAVA/REST API Stress client system reports sockets on CLOSE_WAIT on java stress program after about 3 hours in netstat

[sammefford]

The following issue was shared with me by Ajit George:

Ran a JAVA / REST API Stress test with failover on 7.0 nightly (ML Server 7.0-20150210) for more than 3hrs. Some time after this, it was noticed that the rest app server was not handling any new request, though the server itself is very responsive.

There were no new documents being written into the database.

The stress client Java program log in one of the test run reported :
com.sun.jersey.api.client.ClientHandlerException: org.apache.http.NoHttpResponseException: The target server failed to respond
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond

Looking at the netstat -p JavaClient output on the server where Stress program is being produced a list of 100 CLOSE_WAIT socket connections on port 8011. 8011 is the port for restappserver on the E Node.

Here is a snippet of netstat -p command. Was able to reproduce this on multiple runs.

builder@rh6-intel64-6$ netstat -p 10467
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 localhost:21416 localhost:sunrpc TIME_WAIT -
tcp 0 0 localhost:785 localhost:cadview-3d TIME_WAIT -
tcp 0 0 localhost:787 localhost:cadview-3d TIME_WAIT -
tcp 0 0 srv-403k.colo.marklogic:ssh ageorge-z620.marklogi:43084 ESTABLISHED -
tcp 0 0 srv-403k.colo.marklogic:ssh ageorge-z620.marklogi:44150 ESTABLISHED -
tcp 0 0 srv-403k.colo.marklogic:799 homer2.marklogic.com:nfs ESTABLISHED -
tcp 0 0 localhost:11076 localhost:sunrpc TIME_WAIT -
tcp 1 0 srv-403k.colo.marklog:39966 srv-410h.colo.marklogi:8011 CLOSE_WAIT 10467/java
tcp 1 0 srv-403k.colo.marklog:39901 srv-410h.colo.marklogi:8011 CLOSE_WAIT 10467/java
tcp 1 0 srv-403k.colo.marklog:39961 srv-410h.colo.marklogi:8011 CLOSE_WAIT 10467/java
tcp 1 0 srv-403k.colo.marklog:39884 srv-410h.colo.marklogi:8011 CLOSE_WAIT 10467/java

[sammefford]

Ajit, can you tell me if you performed a failover during this test? If so, can you tell me how the timing of it compared to the NoHttpResponseException? We saw one NoHttpResponseException during our 8.0 stress tests: #252

That issue we decided not to stop shipment for because it occurred during a failover/recovery. We can perhaps make the same decision here.

[sammefford]

From Ajit (he emailed me):

Here is the output from the Stress script and it looks like the issue is happening around the time the Downed D Node tries to come up.

18:05:57 about to load the content
18:05:57 About to load 2501 documents.
Entering int  endFinishedTests()
Exiting int  endFinishedTests()0
Currently running 24
MinUsers 20
com.sun.jersey.api.client.ClientHandlerException: org.apache.http.NoHttpResponseException: The target server failed to respond
    at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:184)
    at com.marklogic.client.impl.DigestChallengeFilter.handle(DigestChallengeFilter.java:34)
    at com.sun.jersey.api.client.filter.HTTPDigestAuthFilter.handle(HTTPDigestAuthFilter.java:493)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:537)
    at com.marklogic.client.impl.JerseyServices.putPostDocumentImpl(JerseyServices.java:1238)
    at com.marklogic.client.impl.JerseyServices.putDocument(JerseyServices.java:954)
    at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:453)
    at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:388)
    at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:358)
    at test.stress.JavaAPISession.putDocument(JavaAPISession.java:146)
    at test.stress.RestLoadTester.loadContentFromDir(RestLoadTester.java:161)
    at test.stress.RestCRUDTester.runTest(RestCRUDTester.java:533)
    at test.stress.StressTest.run(StressTest.java:71)

[grechaw]

A missing leaky handle somewhere? Seems potentially quite serious