Failed to resolve hostname behind proxy in 5.6.0.RELEASE/5.7.0.RELESE

[postngsb]

cloudfoundry-client-reactor:5.6.0.RELEASE/5.7.0.RELESE have the error behind Proxy.

The same proxyconfiguration works fine with cloudfoundry-client-reactor:4.16.0.RELEASE

Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'api.sys.xxx' and search domain query for configured domains failed as well: [xxx]
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1047) ~[netty-resolver-dns-4.1.73.Final.jar:4.1.73.Final]

[dmikusa]

My first thought would be that you have issues with a dependency in your application. The DNS resolution is not something that cf-java-client is doing, as you can see from the error it netty is handling that.

The primary difference between cf-java-client 4.x and 5.x is that they expect different versions of dependencies. The 4.x line is tested to work with Spring Boot 2.3 provided dependencies. The 5.x line is tested with Spring Boot 2.4 - 2.6 provided dependencies. The indicated versions of Spring Boot provide those dependencies and if you stick to what they provide, you'll generally be in good shape. Nothing really forces you to use those dependencies though, you can override them or use cf-java-client 5.x with older Netty dependencies. Once you start customizing versions, you can create problems though.

Having said that, its really hard to say what might be happening here. It could be this or any number of other things. If you could please provide the following info, it'll help to discuss further.

https://github.com/cloudfoundry/cf-java-client#troubleshooting

Thanks

[postngsb]

Thanks for your response, @dmikusa-pivotal

It looks like my proxyconfiguration is not honored, or I did not set the proxy correctly?

For testing, I copy the six beans defined in the https://github.com/cloudfoundry/cf-java-client#troubleshooting

The only change I made is:

 @Bean
    DefaultConnectionContext connectionContext(@Value("${cf.apiHost}") String apiHost) {
  
        ProxyConfiguration proxyConfiguration = ProxyConfiguration.builder()
        .host(xxx)
        .port(xxx)
        .build();

    return DefaultConnectionContext.builder()
        .apiHost(apiHost)
        .skipSslValidation(false).proxyConfiguration(proxyConfiguration)
        .build();
}

The gradle file is generated from https://start.spring.io/

plugins {
  id 'org.springframework.boot' version '2.6.4'
  id 'io.spring.dependency-management' version '1.0.11.RELEASE'
  id 'java'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '11'

repositories {
	mavenCentral()
}

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-webflux'
	
	implementation 'org.cloudfoundry:cloudfoundry-client-reactor:5.7.0.RELEASE'
	implementation 'org.cloudfoundry:cloudfoundry-operations:5.7.0.RELEASE'

	compileOnly 'org.projectlombok:lombok'
	annotationProcessor 'org.projectlombok:lombok'

	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'io.projectreactor:reactor-test'
}

tasks.named('test') {
	useJUnitPlatform()
}

Error Message during start the application:

2022-03-21 11:15:00.475 ERROR 80305 --- [ctor-http-nio-1] cloudfoundry-client.compatibility        : An error occurred while checking version compatibility:

io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'api.sys.xxx' and search domain query for configured domains failed as well: [xxx]
        at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1047) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoSwitchIfEmpty] :
        reactor.core.publisher.Mono.checkpoint(Mono.java:2177)
        org.cloudfoundry.reactor._InfoPayloadRootProvider.getInfo(_InfoPayloadRootProvider.java:64)
Error has been observed at the following site(s):
        *__checkpoint() ⇢ at org.cloudfoundry.reactor._InfoPayloadRootProvider.getInfo(_InfoPayloadRootProvider.java:64)
        *__checkpoint() ⇢ at org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.token(AbstractUaaTokenProvider.java:261)
        *__checkpoint() ⇢ at org.cloudfoundry.reactor.client.v2.info.ReactorInfo.get(ReactorInfo.java:49)
Original Stack Trace:
                at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1047) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1000) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:418) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsResolveContext.onResponse(DnsResolveContext.java:629) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsResolveContext.access$400(DnsResolveContext.java:66) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:462) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsQueryContext.trySuccess(DnsQueryContext.java:216) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:208) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1314) ~[netty-resolver-dns-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[netty-transport-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.74.Final.jar:4.1.74.Final]
                at java.base/java.lang.Thread.run(Thread.java:829) ~[na:an]

[dmikusa]

OK, thanks. I can try those steps.

Two more things I'd like to confirm:

1. ${cf.apiHost} is 'api.sys.xxx' from the error message, correct?
2. I suspect replicating this is going to depend on having a network similar to yours. What does your network setup look like? Can you ascii diagram roughly what the traffic flow looks like? In particular, what is the relation to the machine where you're working & getting this error and the DNS server? How is traffic expected to flow between the two? Is there direct network access to the DNS server? Does the DNS request need to go through the proxy? Do you normally configure an HTTP(S) Proxy or do you normally configure a SOCKS5 proxy to enable lookups?

[postngsb]

1. The cf.apiHost value is the same as the one in the error.
2. We use HTTP proxy server, not SOCKS5 proxy based on my understanding.

I like to share the following observation, again I doubt the proxy setting may not work at cf-java-client 5.7.0.

PROXY WITH VPN
Spring Boot 2.3.12.RELEASE/cf-java-client 4.16.0.RELEASE works as expected.
Spring Boot 2.6.4/cf-java-client 5.7.0.RELEASE: dnsResolve issue (Unexpected)

PROXY WITHOUT VPN
Spring Boot 2.3.12.RELEASE/cf-java-client 4.16.0.RELEASE: java.nio.channels.UnresolvedAddressException: null (Expected)
Spring Boot 2.6.4/cf-java-client 5.7.0.RELEASE: works but Unexpected because I set proxy configuration.

Tried to run On Prem PCF to access public PCF also, and no luck either with cf-java-client 5.7.0

[dmikusa]

That's interesting. Adding the VPN into the mix complicates things even more, at the network level.

About all I can say is that the code on the cf-java-client side regarding proxy configuration really hasn't changed. It's configuring Netty in the same way, by instructing Netty to use the HTTP proxy configuration you define.

What stands out as odd to me is that a DNS resolution issue is probably not related to HTTP proxy settings. The client isn't making a DNS request (which would typically be UDP) through an HTTP proxy.

What usually happens is that the client sends the HTTP request with the FQDN in the HTTP request, so GET https://example.com/foo instead of just GET /foo. This causes the proxy to resolve example.com on the proxy, so the proxy's DNS would determine how example.com resolves.

That's not the only thing that can happen though. The client might have a NO_PROXY list where the client skips sending to the proxy altogether. In this case, the client would resolve DNS locally. I believe that if you're using a SOCKS proxy, DNS resolution happens locally as well, although I'm not an expert on proxies. The point here is that the expected behavior can vary and your network setup can impact that as well. All together the presented condition of "it works with 4.16 and fails with 5.7" isn't really enough detail to establish a root cause.

You probably want to enable wire logging, step #5 https://github.com/cloudfoundry/cf-java-client#troubleshooting, and look at what traffic is being sent and to where it's being sent, and possibly also do some Wireshark captures to again see what traffic is being generated and where that traffic is going.

It seems like in the failure case for 5.7 that it's likely sending out an actual DNS request, based on the error message. Thus you'd want to know where the DNS request is supposed to go (i.e. where it goes when the request is successful, if there is a DNS request sent in this case) and compare that to where the DNS request goes when it fails. That should help to illuminate what's going wrong and what would need to be done to correct that.

[postngsb]

@dmikusa-pivotal Without VPN, we tested it on Pivotal Cloud Foundry in our data center, and got the same error when accessing PCF on public cloud through proxy.

Do you have any working sample (Spring Boot 2.6/CF-JAVA-CLIENT 5.7) with Proxy Configuration? I like to try it our environment.

Thanks.

[dmikusa]

What you specified here looks reasonable to me. It is configuring HTTP(S) proxy for the CF Java Client.

I can't say why it's working for you in some situations and not in others. There are a lot of variables here and I don't have an environment to replicate your Proxy/VPN setup. As previously mentioned, I would suggest some network troubleshooting. Look at the wire traces or wireshark captures and compare the working/not-working traces. I think if we had something concrete like that, we could discuss the situation further.

[thall6]

@postngsb Any progress or workaround on your issue? I too have this issue, with client app failing through corporate HTTP proxy server when trying to reach foundations that reside outside our network. Spring Boot 2.4.x and 2.5.x + cf-java-client 5.7 fails through proxy but works for Foundations inside our network. Spring Boot 2.3.x + cf-java-client 4.x works fine through proxy, but we really need to get off this version of Spring Boot.

[swagataroy]

I think it is this change that was introduced in Spring Boot 2.4.x because of the change in Netty versions.
spring-projects/spring-boot#24743

Spring 2.3.x references 4.1.59 version of Netty https://docs.spring.io/spring-boot/docs/2.3.9.RELEASE/reference/html/appendix-dependency-versions.html#dependency-versions
Spring 2.4.x references 4.1.76 version of Netty https://docs.spring.io/spring-boot/docs/current/reference/html/dependency-versions.html

It has been discussed on this thread
(I ran into an issue with the Netty DNS resolver (that became the default in reactor/reactor-netty#1252)....)

There is a workaround on that thread. Could you please try that?
reactor/reactor-netty#1431 (comment)
reactor/reactor-netty#1431

[dmikusa]

Thanks for all of the information here. I think we have hit upon the issue. Stay with me here, this one is a little wild, but I believe this to be the cause.

1. The getHttpClient(..) method is being called to fetch the HttpClient.

2. The method creates an HttpClient

3. If you have set "additional HTTP client configuration" on the DefaultContextBuilder then we call that method passing in the created HttpClient. If you don't have this set then the created HttpClient is returned.

   Example:

   DefaultConnectionContext.builder()
   			.apiHost(apiHost)
   			.additionalHttpClientConfiguration(httpClient -> httpClient)
   ...
   

The problem is that to make the check we are doing the following:

    getAdditionalHttpClientConfiguration().map(configuration -> configuration.apply(configureHttpClient(client))).orElse(client)

It's subtle, but before we call the custom configuration callback, we also call configureHttpClient(client) on the HttpClient. This is the problem because this method, which is what sets up proxy configuration and wiretrace logging, only gets called if you pass in an additional configuration callback.

It seems like an odd root cause, but without the proxy configuration being set it makes sense why connections would fail. It's as if the proxy was never configured and that will not work.

I'm working on a fix, hope to have that out shortly.