Envoy proxy takes up a lot of RAM

[FlorianSW]

Summary

When deploying an app with the route integrity feature enabled, the deployed envoy proxy takes up a lot more RAM as the documentation states, which can (and did in our deployment) result in a "crash" of the app container as CloudFoundry sees a higher memory usage as it was specified as the max value during the deployment.

Expected Result

The envoy proxy should not take up more memory than the described amount (as documented).

Actual Result

The app container gets restarted by CloudFoundry.

Context

I first asked in the diego support channel in slack and got routed here :)

We've an application, which is not doing so much stuff, which constatntly crashes (Out of Memory) when serving some hundreds of requests. We're sure (as we also checked it locally) that the app does not take up more than 100 MB memory.

The app container itself has 250 MB memory and the app still crashes. Our investigation showed, that not our app is taking up the memory, but the envoy reverse proxy.

All the information that are a result of our investigation are available in the following gist:
https://gist.github.com/FlorianSW/090046027fef348d410041e1137b7f4c

In order to reproduce the problem locally, I started a cfdev environment, enabled the TLS connection from the gorouter to the app instance container and run the same stress test (300 req/s with one app instance) again. However, this time, the app did not crash at all. While looking at the memory consumption of the envoy proxy, it does not go over 34 MB, which looks like the value one would expect when reading the diego-release documentation regarding the proxy.
The cfdev deployment version is 9.0.0
The deployed CloudFoundry version uses the cf-deployment 10.1.0 and did not change anything related to the reverse proxy feature, stated by a colleague of mine.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/167925159

The labels on this github issue will be updated when the story is started.

[n4wei]

@FlorianSW We noticed in your deployments that the gorouter has connection keepalive turned off for backends. We suspect at a high rate of requests/sec, additional RAM is taken up because connections are in the TIME_WAIT state and waiting to be cleaned up. Could you please try the following gorouter configuration and let us know if you are still experiencing this OOM killing your app?

Set this value to something larger than 0. We tested with 100.

[FlorianSW]

@n4wei What deployment are you talking about, the local cfdev deployment or the actually deployed one? If the latter, may I ask where you get this info from? I don't see this config being set in neither of the files I added to the gist (just wanting to understand where this is coming from) :) I'll need to check with the guys who are maintaining the CloudFoundry deployment if we actually are able to change this value :)

[sunjayBhatia]

@FlorianSW if you do not set https://github.com/cloudfoundry/routing-release/blob/23725ac12e40537f84a2099436881d9d5ccd3493/jobs/gorouter/spec#L169-L171 to a value in your manifest, it will be set to the default of 0 which disables keepalives from the gorouter to backends

Given that the property does not show up in the manifest you shared, keepalives have been disabled. If the gorouter settings match your production deployment, then that also has keepalives disabled.

We have been chatting with the networking team about changing the defaults on the gorouter to enable idle connections/keepalives by default

[FlorianSW]

To my defense, I wrote the comment in github before checking the comments in the pivotaltracker task :) Anyways: Thanks for the explanation, I thought about that already.

I talked with the guys from the team which manages our CloudFoundry runtime and we'll working on getting the keepalive setting tested. Because of the workload of all of us, it may take some time. However, I'll come back to you as soon as we've the results :)

[heyjcollins]

@FlorianSW - I'm going to go ahead and close this out given the networking team will be changing defaults on the gorouter to enable idle connections/keepalives, and with the assumption that enabling keepalives will address this issue.

ere's the networking team's story for updating the defaults in gorouter that Sunjay mentioned a few days ago: https://www.pivotaltracker.com/n/projects/2245594/stories/168602911

Please feel free to comment and reopen if this issue persists after your keepalive setting is updated/tested or if you'd like to share the results of your analysis. Thanks!