KVMHAMonitor getting initialized without host ha enabled

[sbrueseke]

ISSUE TYPE

• Bug Report

COMPONENT NAME

HA

CLOUDSTACK VERSION

4.19.0

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

Even when HA has been disabled on cluster and/or host level, KVMHAMonitor getting initialized on KVM hosts.
It looks like code does not check if HA is enabled or not:
https://github.com/apache/cloudstack/blob/8f6721ed4c4e1b31081a951c62ffbe5331cf16d4/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

STEPS TO REPRODUCE

1) disable HA on cluster level via UI
2) login to kvm host
3) take a look at agent.log

EXPECTED RESULTS

When HA is disabled I would expect that the management server adds the following parameter to agent.properties file of each host:
reboot.host.and.alert.management.on.heartbeat.timeout=false

or

KVMHAMonitor will not getting initialized at all.

ACTUAL RESULTS

Nothing happens when disabling HA on cluster and/or host level. KVMHAMonitor is getting initialized and will perform checks of the host. In some situations this will lead to an automatic reboot of the host because of KVMHAMonitor is not able to write heartbeat for pool to primary storage:

Feb 19 12:01:08 kvm-2 java[9465]: WARN  [kvm.resource.KVMHAMonitor] (Thread-1:) (logid:) Write heartbeat for pool [71c272d3-b180-4b18-a0fc-cfc1dc5b86c9] failed: Down; try: 5 of 5.
Feb 19 12:01:08 kvm-2 java[9465]: WARN  [kvm.resource.KVMHAMonitor] (Thread-1:) (logid:) Write heartbeat for pool [71c272d3-b180-4b18-a0fc-cfc1dc5b86c9] failed: Down; stopping cloudstack-agent.
Feb 19 12:02:08 kvm-2 heartbeat: kvmspheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage.

[DaanHoogland]

@sbrueseke are you addressing (designing/implementing a solution to) this bug?

[sbrueseke]

@sbrueseke are you addressing (designing/implementing a solution to) this bug?

I'm not deep enough into the topic to suggest a solution. We noticed the bug because we were carrying out some tests with Linstor in our lab and suddenly all the KVM hosts restarted. By evaluating the logs, we saw that it was due to the KVMHAMonitor. However, we did not have HA enabled, not on the cluster and not on the hosts.
The question is why the KVMHAMonitor is running on the hosts even if HA is not configured. Maybe there is a reason, but I don't know it. But what, in my opinion, needs to be prevented is that KVMHAMonitor restarts hosts despite HA being deactivated.
One solutions for this would be to set reboot.host.and.alert.management.on.heartbeat.timeout=false in the agent.properties by the management server if HA is not configured or deactivated.
But as I sad before, I'm not deep into that topic and I do not know if this is a valid solution or if there is a much better one.

[adidiborg]

@sbrueseke, We are also facing similar issue, but as per our logs ( on all KVM ) we are only seeing below log lines -

Feb 19 12:01:08 kvm-2 java[9465]: WARN  [kvm.resource.KVMHAMonitor] (Thread-1:) (logid:) Write heartbeat for pool [71c272d3-b180-4b18-a0fc-cfc1dc5b86c9] failed: Down; try: 5 of 5.
Feb 19 12:01:08 kvm-2 java[9465]: WARN  [kvm.resource.KVMHAMonitor] (Thread-1:) (logid:) Write heartbeat for pool [71c272d3-b180-4b18-a0fc-cfc1dc5b86c9] failed: Down; stopping cloudstack-agent.

and not seeing this one which you have mentioned

Feb 19 12:02:08 kvm-2 heartbeat: kvmspheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage.

[dseralathan]

heartbeat[13962]: kvmheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage.

I received the same error when my Primary NFS Storage was in accessible .

[DaanHoogland]

Hey @sbrueseke , I finally got a chance to look at this again.If you are using NFS this is how it is supposed to work , I think. If you are using LinStor, see the merged PR for 4.19 (#8670). Is this still an issue?

[sbrueseke]

@DaanHoogland now I am confused! Why is KVMHAMonitor rebooting hosts when the host is unable to write a heartbeat file to any storage when any HA setting is disabled in the UI? Do you know the reason for that?

[weizhouapache]

@DaanHoogland now I am confused! Why is KVMHAMonitor rebooting hosts when the host is unable to write a heartbeat file to any storage when any HA setting is disabled in the UI? Do you know the reason for that?

@sbrueseke
You can add a setting in /etc/cloudstack/agent.properties and restart cloudstack-agent

reboot.host.and.alert.management.on.heartbeat.timeout=false