[BUG REPORT]: Route unmapped but still routes to a zombie container.

[pusherofbrooms]

Summary

On occasion (it has happened thrice so far in the past year), a customer will unmap a route from an application and delete the application, and yet one of the containers will still be running, and route emitter will still be advertising the route to the gorouters. I didn't see any related errors in the route-emitter logs, not in the rep logs, though rep logs may have rotated out before I came on the scene.

The short term fix up to now is to "monit stop all", ensure all container processes are dead, and then start the processes. Alternatively, shutting down the cell and letting "scan and fix" sweep it up also works.

Expected Result

Route is unmapped
App is stopped
Containers stop
route no longer routes to any containers.

Actual Result

Route is unmapped
App is stopped
One container is not stopped
the unmapped route still routes to the undead container.

Diego repo

I don't have a lot of evidence, but since the container didn't stop I primarily suspect garden-runc https://github.com/cloudfoundry/garden-runc-release

Secondarily, route-emitter https://github.com/cloudfoundry/route-emitter since unmapping of the route didn't complete.

Context

Diego-release 2.42.0
as part of cf-deployment v12.28.0 running on AWS.

Steps to Reproduce

I don't have a consistent way to reproduce the problem as it has only happened 3 times in the past year and isn't predictable.

Possible Causes or Fixes (optional)

Additional Text Output or Screenshots (optional)

[aminjam]

@pusherofbrooms Do you have route integrity enabled in this foundation? There was some work done sometime ago that ensured unregistration messages are sent periodically more context (https://www.pivotaltracker.com/n/projects/1003146/stories/166555852).

[pusherofbrooms]

We do run route integrity in this foundation (and all of our other foundations).

It seems like the route-emitter is sending updates about the bogus route. If I stop the route-emitter on the afflicted cell, eventually the route will drop out of the gorouters and I'll receive the expected 404 for that route.

[mdelillo]

@pusherofbrooms we tried to reproduce this but weren't able to see the error. It's hard to debug without it being reproducible or being able to see the logs. We're going to close this for now, but please feel free to reopen if you have more info and can send us some logs.

[pusherofbrooms]

@mdelillo do you have a sense of which logs will be useful for tracking this down the next time it occurs? I begin to suspect that this problem is related to the one I described at https://cloudfoundry.slack.com/archives/C02FM2BPE/p1585320574008900 .

I don't understand all of the mechanisms for why a container might not stop when given a signal, or even if that signal is making it to the container in this case.

[sunjayBhatia]

@pusherofbrooms If you have a sense of which cells the "extra" app instances are on and can get the app guid, you can grep through the rep logs on that cell to find the process_guid and instance_guid for the instances of that app on the cell. From there, you can use those guids to look through the garden logs, bbs logs, etc. or just provide the guids to us and the complete set of logs.

To narrow down what we're looking at, I would also use cfdot to get an overall picture of the environment:

• to get a view of desired LRPs: cfdot desired-lrps | jq -c '[.process_guid, .instances]' | sort
• to get a view of the actual running LRPS: cfdot actual-lrps | jq -c '[.process_guid, .instance_guid, .cell_id, .state, .presence]' | sort
  I would pipe the output of these to files, pick some guids that look interesting, grep for counts of LRPs that look like they are extras, and look for those instance guids in the logs, etc.