brain/0 failing to run on bosh-lite/garden-runc

[drnic]

Running diego 1.6.2 and earlier on a bosh-lite deployed using garden-runc both cc_uploader and stager monit jobs are failing.

[replaced some theories that I don't think are correct - routing-release has the same implementation of running_in_containers afaict]

Neither /var/vcap/sys/log/stager/ nor /var/vcap/sys/log/cc_uploader have any files in them; nor does /var/vcap/sys/log/monit/ have anything for these two failing jobs.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/139444001

The labels on this github issue will be updated when the story is started.

[emalm]

Hi, @drnic, what version of cf-release do you have deployed? The stager and cc-uploader jobs now come from that release, via capi-release. In any case, it also sounds like your deployment deviates from the manifests generated via the diego-release scripts and templates, as those BOSH job templates are still on the cc_bridge_z{1,2,3} jobs.

Anyway, this issue sounds the same as the one discussed on cloudfoundry/routing-release#56, where the previous running_in_container function is not broad enough to detect the cgroups that garden-runc uses for its containers. A fix for the version of that function used in capi-release is also in v1.16.0, but the latest CF final release, v251, contains only v1.15.0 of capi-release.

Best,
Eric

[drnic]

Ok, thanks Eric. I'll revisit all the version of things running and where they come from.

[drnic]

The running_in_container functions look similar between the routing-release + diego-release:

routing-release's:

running_in_container() {
  grep --quiet --invert-match ':/$' /proc/self/cgroup
}

vs diego-release's

running_in_container() {
  # look for a non-root cgroup
  grep --quiet --invert-match ':/$' /proc/self/cgroup
}

vs capi-releases'

running_in_container() {
  # look for a non-root cgroup
  grep --quiet --invert-match ':/$' /proc/self/cgroup
}

update: am now rolling in 1.16.0 capi for all its job templates (looks like it was fixed then; and 1.15.0 was included in v250 of cf-release)

[emalm]

But capi-release at v1.15.0 has the old version: https://github.com/cloudfoundry/capi-release/blob/1.15.0/src/capi_utils/container_utils.sh#L4.

[drnic]

Cool, replacing v250's jobs with cap-release 1.16.0 fixed this issue.