BBS Crashing and Failing over

[andrew-edgar]

Thank you for submitting an issue to the diego-release repository. We appreciate the feedback. To help us address your issue, please fill out the sections in the following template to the best of your ability:

Summary

We have seen in our very busy production env we see the BBS will crash and failover. Once the failover occurs the new BBS sometimes takes a while to recover and be in a performant state.

Expected Result

When the BBS is running there should be no unexpected crashes and failovers.

Actual Result

The BBS crashes :)

Context

This is IBM Public Cloud us-south deployment. We have about 780 Cells. This is diego 2.25.0 version with CF-Deployment 6.8.0.

This is on SoftLayer. This is a very large 16 core VM, we are using postgres as the backend and have 400 db connections (min and max). We cannot change this due to limitations in the postgres deployment.
The file description limit is 100000 as set in bpm.yml.

Steps to Reproduce

Unable to reproduce in smaller envs as this must be due to load or particular errors in some queries.

Possible Causes or Fixes (optional)

Additional Text Output or Screenshots (optional)

I will be attaching logs (or sending them via slack as they are very large in the next day or two.

This was already discussed here ...

https://cloudfoundry.slack.com/archives/C02FM2BPE/p1556639064033100

There are some inital goroutine dumps there (small parts) but will try and get the rest shortly.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/165795047

The labels on this github issue will be updated when the story is started.

[andrew-edgar]

This is happening several times a day in our env. can you please prioritize this so we can get a fix for why the bbs is crashing? Is there anything else I can provide?

[sunjayBhatia]

@andrew-edgar just pulled this up in priority, should be the next thing in the backlog

[mariash]

@andrew-edgar are all failures due to the same error: concurrent map read and map write? Could you turn on debug logs in bbs and share them?

[andrew-edgar]

@mariash we cannot turn on debug logs. this is our largest production env and that would cause a HUGE amount of logging which would effect performance

[sunjayBhatia]

@andrew-edgar We have a fix for the issues we saw in your logs around panics with the error concurrent map read and map write. Do you have any evidence of other failure scenarios causing BBS crashes you see in your environment?