Trying to decrease VM size and running into some issues

[StanleyShen]

Hello, all

I am trying to create a new deployment file for minimizing VM size, and I started the work based on https://github.com/cloudfoundry-community/cf-boshworkspace

It looks it's not maintained right now, my question got no response for several days.
Anyway, I create an deployment file based on it, and when deploying I always got error about consul component.

The deployment file starts from https://github.com/cloudfoundry/cf-release/tree/master/example_manifests and adds diego components initially.

  Started updating job backbone_z1 > backbone_z1/0 (canary)
  Started updating job data > data/0 (canary)
  Started updating job services_z1 > services_z1/0 (canary)
  Started updating job health_z1 > health_z1/0 (canary)
  Started updating job cell_z1 > cell_z1/0 (canary)
  Started updating job ha_proxy_z1 > ha_proxy_z1/0 (canary)
  Started updating job api_z1 > api_z1/0 (canary)
  Started updating job brain_z1 > brain_z1/0 (canary)
     Done updating job backbone_z1 > backbone_z1/0 (canary) (00:02:24)
     Done updating job brain_z1 > brain_z1/0 (canary) (00:03:25)
   Failed updating job ha_proxy_z1 > ha_proxy_z1/0 (canary): `ha_proxy_z1/0' is not running after update (00:10:28)
   Failed updating job data > data/0 (canary): `data/0' is not running after update (00:10:39)
   Failed updating job cell_z1 > cell_z1/0 (canary): `cell_z1/0' is not running after update (00:10:55)
   Failed updating job health_z1 > health_z1/0 (canary): `health_z1/0' is not running after update (00:11:25)
   Failed updating job services_z1 > services_z1/0 (canary): `services_z1/0' is not running after update (00:11:27)
   Failed updating job api_z1 > api_z1/0 (canary): `api_z1/0' is not running after update (00:14:08)

Error 400007: `ha_proxy_z1/0' is not running after update

And after checking the log of ha_proxy_z1 job, I can see the main problem is the consul.
Finally Itt complains

==> Error starting agent: Failed to start Consul client: Failed to start lan serf: Failed to start TCP listener. Err: listen tcp 10.0.0.25:8301: bind: address already in use

I checked on on working cf instance, and I can see the same consul processes are running, and I didn't see any difference between them.

minimal-vms.txt
ha_proxy_z1.0.2016-03-17-19-05-08.zip

Attached are the deployment file and logs of ha_proxy_z1.
Could someone show some guide on it, thanks in advance.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/115850423.

[StanleyShen]

In my case, I see error "Err: listen tcp 10.0.0.25:8301: bind: address already in use", which means the consul_agent is not started correctly, and I also see it's not correct via "sudo monit summary"

But I do see they are running

root@81a1c783-da5d-4d5d-a419-c2c44345e285:/var/vcap/sys/log/consul_agent# ps axu | grep consul
root      4171  0.0  0.0  18124  1696 ?        S<   10:50   0:00 /bin/bash -exu /var/vcap/jobs/consul_agent/bin/agent_ctl start
root      4172  0.0  0.0  18124  1696 ?        S<   10:50   0:00 /bin/bash -exu /var/vcap/jobs/consul_agent/bin/agent_ctl start
root      4173  0.0  0.0   4352  1404 ?        S<   10:50   0:00 tee -a /var/vcap/sys/log/consul_agent/consul_agent.stdout.log
root      4174  0.0  0.0   4344  1488 ?        S<   10:50   0:00 logger -p user.info -t vcap.consul-agent
root      4175  0.0  0.0   4352  1400 ?        S<   10:50   0:00 tee -a /var/vcap/sys/log/consul_agent/consul_agent.stderr.log
root      4176  0.0  0.0   4344  1524 ?        S<   10:50   0:00 logger -p user.error -t vcap.consul-agent
vcap      4184  0.2  0.4 156040 18340 ?        S<l  10:50   0:47 /var/vcap/packages/consul/bin/consul agent -config-dir=/var/vcap/jobs/consul_agent/config -recursor=10.0.0.2
root     24312  0.0  0.0  10472  2176 pts/0    S+   17:15   0:00 grep --color=auto consul

[emalm]

Hi, @StanleyShen,

I'm closing this issue since it's related to cf-release, not diego-release. Please open an issue on the cf-release repository if you're seeing issues colocating or deploying certain jobs from cf-release. Also please include what version of CF you're using and how you're generating the manifest when you file the report.

One thing you could check on the failing VMs is what processes have already bound to port 8301. Use bosh ssh to log into the VM, run sudo -i to get root, and then run netstat -anp and look for port 8301 in the output. From the ps output, it may be that monit has lost track of the consul job.

Best,
Eric