This is regarding the usage of NettyNioAsyncHttpClient connection pool with (aws 2.17.99 sqs) sdk which is being used in our sqs poller application

[sharmanalin59]

Describe the issue

This is regarding the usage of NettyNioAsyncHttpClient connection pool with aws 2.17.99 sqs sdk which is being used in our sqs poller application,
on setting maxConcurrency=200(or even 1000) and maxPendingConnectionAcquires = 20000,

Java sqs async sdk client works to fetch data, however while higher aws sqs fetch tps, we encounter the log saying time as "10000 milliseconds." and not 20000 as set by our code as below. Hence we believe that sqs client is using the default connection pool of maxPendingConnectionAcquires = 10000 & maxConcurrency = 50, dishonouring our configured connection pool of maxPendingConnectionAcquires = 20000 & maxConcurrency = 200 (NettyNioAsyncHttpClient).

code usage -:
sqsClient = super.initBuilderConfig(sqsClientBuilder).httpClientBuilder(ApacheHttpClient.builder()
.maxConnections(200).connectionAcquisitionTimeout(Duration.ofMillis(20000)))
.build();
sqsAsyncClient = super.initBuilderConfig(sqsAsyncClientBuilder).httpClientBuilder(NettyNioAsyncHttpClient.builder()
.maxConcurrency(200).maxPendingConnectionAcquires(20000))
.build();

Reference-:
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration-netty.html

we are facing the below stack trace-:

=============================================================================================

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62) ~[utils-2.17.99.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) ~[sdk-core-2.17.99.jar!/:na]
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.99.jar!/:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103) ~[sdk-core-2.17.99.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:181) ~[sdk-core-2.17.99.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[sdk-core-2.17.99.jar!/:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.99.jar!/:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:104) ~[sdk-core-2.17.99.jar!/:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$WrappedErrorForwardingResponseHandler.onError(MakeAsyncHttpRequestStage.java:158) ~[sdk-core-2.17.99.jar!/:na]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:302) ~[netty-nio-client-2.17.99.jar!/:na]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:174) ~[netty-nio-client-2.17.99.jar!/:na]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) ~[netty-transport-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312-debug]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.17.99.jar!/:na]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.17.99.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[sdk-core-2.17.99.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[sdk-core-2.17.99.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:179) ~[sdk-core-2.17.99.jar!/:na]
... 28 common frames omitted
Caused by: java.lang.Throwable: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:308) ~[netty-nio-client-2.17.99.jar!/:na]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:301) ~[netty-nio-client-2.17.99.jar!/:na]
... 11 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer than 10000 milliseconds.
at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.timeoutAcquire(HealthCheckedChannelPool.java:77) ~[netty-nio-client-2.17.99.jar!/:na]
at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda$acquire$0(HealthCheckedChannelPool.java:67) ~[netty-nio-client-2.17.99.jar!/:na]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) ~[netty-common-4.1.68.Final.jar!/:4.1.68.Final]
... 6 common frames omitted

=============================================================================================

Java sqs async sdk client works to fetch data, however while higher aws sqs fetch tps, we encounter the log saying time as "10000 milliseconds." and not 20000 as set by our code as below

code usage -:
sqsClient = super.initBuilderConfig(sqsClientBuilder).httpClientBuilder(ApacheHttpClient.builder()
.maxConnections(200).connectionAcquisitionTimeout(Duration.ofMillis(20000)))
.build();
sqsAsyncClient = super.initBuilderConfig(sqsAsyncClientBuilder).httpClientBuilder(NettyNioAsyncHttpClient.builder()
.maxConcurrency(200).maxPendingConnectionAcquires(20000))
.build();

Reference-:
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration-netty.html

Expected behavior

connection pool of maxPendingConnectionAcquires = 20000 & maxConcurrency = 200 (NettyNioAsyncHttpClient) should be applied

Current behavior

connection pool of maxPendingConnectionAcquires = 20000 & maxConcurrency = 200 (NettyNioAsyncHttpClient) is not being applied and currently using the default of maxPendingConnectionAcquires= 10000, maxConcurrency = 50

Steps to Reproduce

code usage -:
sqsClient = super.initBuilderConfig(sqsClientBuilder).httpClientBuilder(ApacheHttpClient.builder()
.maxConnections(200).connectionAcquisitionTimeout(Duration.ofMillis(20000)))
.build();
sqsAsyncClient = super.initBuilderConfig(sqsAsyncClientBuilder).httpClientBuilder(NettyNioAsyncHttpClient.builder()
.maxConcurrency(200).maxPendingConnectionAcquires(20000))
.build();

protected <E extends AwsClientBuilder> E initBuilderConfig(E builder) {
StsClient stsClient = StsClient.builder().region(Region.of(getConfig().getRegion())).credentialsProvider(StaticCredentialsProvider.create(config.getCredential())).build();
AssumeRoleRequest assumeRoleRequest = AssumeRoleRequest.builder()
.roleArn(config.getRoleArn())
.roleSessionName(config.getRoleSessionName())
.build();
AssumeRoleResponse assumeRoleResponse = stsClient.assumeRole(assumeRoleRequest);
Credentials credentials = assumeRoleResponse.credentials();
AwsSessionCredentials awsSessionCredentials = AwsSessionCredentials.create(credentials.accessKeyId(), credentials.secretAccessKey(), credentials.sessionToken());
builder.region(Region.of(getConfig().getRegion())).credentialsProvider(StaticCredentialsProvider.create(awsSessionCredentials)).build();
return builder;
}

Possible Solution

No response

Context

No response

AWS Java SDK version used

2.17.99

JDK version used

8

Operating System and version

centos rhel fedora (Amazon Linux 2)

[debora-ito]

Hi @sharmanalin59

The setting you're looking for is connectionAcquisitionTimeout and not maxPendingConnectionAcquires -

NettyNioAsyncHttpClient.builder()
    .maxConcurrency(200)
    .connectionAcquisitionTimeout(Duration.ofMillis(20000))

You can read a description of each setting in the NettyNioAsyncHttpClient.Builder javadoc.

[sharmanalin59]

Setting ".maxConcurrency(200)" shows 20000 millisec in logs, so that is fine.

However, I wish to achieve 100 tps with 90 sec sla for downstream for each message sent, How shall I configure aws sdke for sqs if I am polling sqs with 100 tps?

I did two configurations but did not achieve it-:

With below configuration-:

---

sqsAsyncClient = super.initBuilderConfig(sqsAsyncClientBuilder)
.httpClientBuilder(NettyNioAsyncHttpClient
.builder()
.connectionAcquisitionTimeout(Duration.ofSeconds(20))
.maxConcurrency(200)
)
.build();

---

I'm getting -:

---

Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer than 20000 milliseconds.
2022-02-06 20:14:28.173 ERROR 24395 --- [tyEventLoop-5-1] c.f.platform.aws.sqs.handler.SqsPoller : Sqs: platformservice_sqs_qa, message: exception caught in poller from processSQSMessagesAsync, exception: {}
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62) ~[utils-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) ~[sdk-core-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:318) ~[netty-nio-client-2.17.101.jar!/:na]
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer than 20000 milliseconds.
2022-02-06 20:14:28.736 ERROR 24395 --- [tyEventLoop-5-1] c.f.platform.aws.sqs.handler.SqsPoller : Sqs: platformservice_sqs_qa, message: exception caught in poller from processSQSMessagesAsync, exception: {}
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62) ~[utils-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) ~[sdk-core-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:318) ~[netty-nio-client-2.17.101.jar!/:na]
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer than 20000 milliseconds.

---

and with below confirguration-:

---

sqsAsyncClient = super.initBuilderConfig(sqsAsyncClientBuilder)
.httpClientBuilder(NettyNioAsyncHttpClient
.builder()
.tcpKeepAlive(true)
.connectionAcquisitionTimeout(Duration.ofSeconds(20))
.maxConcurrency(200)
)
.build();

---

I'm getting-:

---

Caused by: java.io.IOException: Server failed to send complete response. The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because there was a handshake error, the request took too long, or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
2022-02-06 19:41:42.564 ERROR 25479 --- [c-response-23-5] c.f.platform.aws.sqs.handler.SqsPoller : Sqs: platformservice_sqs_qa, message: exception caught in poller from processSQSMessagesAsync, exception: {}
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Server failed to send complete response. The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because there was a handshake error, the request took too long, or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62) ~[utils-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) ~[sdk-core-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Server failed to send complete response. The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because there was a handshake error, the request took too long, or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[sdk-core-2.17.101.jar!/:na]
Caused by: java.io.IOException: Server failed to send complete response. The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because there was a handshake error, the request took too long, or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
2022-02-06 19:41:42.567 ERROR 25479 --- [c-response-23-0] c.f.platform.aws.sqs.handler.SqsPoller : Sqs: platformservice_sqs_qa, message: exception caught in poller from processSQSMessagesAsync, exception: {}
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Server failed to send complete response. The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because there was a handshake error, the request took too long, or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62) ~[utils-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51) ~[sdk-core-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.101.jar!/:na]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[na:1.8.0_312-debug]
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Server failed to send complete response. The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because there was a handshake error, the request took too long, or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[sdk-core-2.17.101.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[sdk-core-2.17.101.jar!/:na]
Caused by: java.io.IOException: Server failed to send complete response. The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because there was a handshake error, the request took too long, or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).

---

[debora-ito]

Setting ".maxConcurrency(200)" shows 20000 millisec in logs, so that is fine.

maxConcurrency sets the maximum number of allowed concurrent requests, it's unrelated to the connectionAcquisition timeout config.

How shall I configure aws sdke for sqs if I am polling sqs with 100 tps?

We can't really give you recommendations on timeout values, these settings depend on your use-case, your application environment, your network workload, etc.

What I can recommend is to enable the client-side metrics to gather insights on your own application performance. You can read more about client-side metrcis in the blog -
https://aws.amazon.com/blogs/developer/using-the-new-client-side-metrics-feature-in-the-aws-sdk-for-java-v2/

The metrics related to concurrency like MaxConcurrency, LeasedConcurrency and PendingConcurrencyAcquires would be helpful in this case.

How often do you see the error "Server failed to send complete response. The channel was closed."?

[sharmanalin59]

For 300 fetch aws tps + 300 delete aws tps (followed 1 sec sla for each message downstream), logs are flooded with this exception "Channel was closed before it could be written to"

code config used-:

sqsAsyncClient = super.initBuilderConfig(sqsAsyncClientBuilder).httpClientBuilder(NettyNioAsyncHttpClient.builder()
.maxConcurrency(300)
.connectionAcquisitionTimeout(Duration.ofSeconds(60))
.maxPendingConnectionAcquires(20000)
).build();

Q1 : I see eventLoopGroupBuilder, is it to power aws connections under netty nio with eventLoopGroupBuilder.numberOfThreads number of netty threads for nio event looping? what is the default value for this?
Q 2: What does aws version 1 use, netty like aws 2 or blocking http pool and what are the default values?
Q 3: Do you open a new connection until maxConcurrency for every (http/tcp) request?
Q 4: Can you also please shed some more light on tcpKeepAlive, does it open maxConcurrency connections and do not close them unless 2 hours of un acknowledged packet by either parties?https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/nio/netty/NettyNioAsyncHttpClient.Builder.html