kvm: Handle storage issue on NFS/KVM in multiple ways #4708

ustcweizhou · 2021-02-18T20:26:14Z

Description

This PR provides multiple options to handle storage issue on kvm.

FS: to be added

Two main parts
(1) kvm: Handle storage issue on NFS/KVM in multiple ways : hardreset (default), noaction, destroyvms, stopagent
(2) kvm: kvm: Schedule investigate tasks for disconnected hosts

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

weizhouapache · 2021-02-18T21:00:23Z

related to #2722 #2890 #2984 #4586

rohityadavcloud · 2021-05-25T13:58:33Z

@blueorangutan package

blueorangutan · 2021-05-25T13:59:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-05-25T14:43:26Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 91

GabrielBrascher · 2021-05-25T19:02:45Z

@ustcweizhou thanks for the PR. From what I understood (really quick glance of eyes) this avoid also situations of HA monitor a removed NFS storage. Am I right?

I have seen some weird issues where a Cluster that had NFS and removes it (standard storage removal process Up > Maintenance > Removed) but still HA expects NFS pool to be "Healthy" and simply Fence all hosts on the cluster.

nvazquez · 2021-06-23T15:01:53Z

Hi @ustcweizhou can you please resolve the conflicts?

DaanHoogland

I wonder about the use of Thread directly, @weizhouapache. Any particular reason?

DaanHoogland · 2021-06-30T13:25:35Z

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java

-                Set<String> removedPools = new HashSet<String>();
-                for (String uuid : _storagePool.keySet()) {
-                    NfsStoragePool primaryStoragePool = _storagePool.get(uuid);
+    protected class CheckPoolThread extends Thread {


why not make this a ManagemedContextRunnable as well?

ManagedContextRunnable and Thread are both implemetation of Runnable interface.
Thread works in testing.

ManagedContextRunnable should also be ok. The following might work as well

protected class CheckPoolThread implements Runnable {

rohityadavcloud · 2021-09-14T06:20:36Z

Ping @weizhouapache can you fix the conflicts, is this essential for 4.16?

sureshanaparti · 2021-09-16T08:05:45Z

engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java

        _monitorExecutor = new ScheduledThreadPoolExecutor(1, new NamedThreadFactory("AgentMonitor"));

+        _scanHostsExecutor =  new ScheduledThreadPoolExecutor(1, new NamedThreadFactory("HostsScanner"));
+        _investigatorExecutor = new ScheduledThreadPoolExecutor(InvestigateDisconnectedHostsPoolSize.value(), new NamedThreadFactory("DisconnectHostsInvestigator"));


Suggested change

_investigatorExecutor = new ScheduledThreadPoolExecutor(InvestigateDisconnectedHostsPoolSize.value(), new NamedThreadFactory("DisconnectHostsInvestigator"));

_investigatorExecutor = new ScheduledThreadPoolExecutor(InvestigateDisconnectedHostsPoolSize.value(), new NamedThreadFactory("DisconnectedHostsInvestigator"));

sureshanaparti · 2021-09-16T08:07:04Z

engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java

        _connectExecutor.shutdownNow();
        _monitorExecutor.shutdownNow();
+        _investigatorExecutor.shutdownNow();
+


_scanHostsExecutor.shutdownNow() missing

sureshanaparti · 2021-09-16T08:09:01Z

engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java

+    protected final ConfigKey<Integer> InvestigateDisconnectedHostsInterval = new ConfigKey<>("Advanced", Integer.class, "investigate.disconnected.hosts.interval",
+        "300", "The time (in seconds) between VM investigation on disconnected hosts.", false);
+    protected final ConfigKey<Integer> InvestigateDisconnectedHostsPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "investigate.disconnected.hosts.pool.size", "10",
+            "Default pool size to investigate disconnected hosts", false);


Suggested change

"Default pool size to investigate disconnected hosts", false);

"The thread pool size to investigate disconnected hosts", false);

sureshanaparti · 2021-09-16T08:18:24Z

@weizhouapache can you address outstanding remarks, and fix the conflicts.

weizhouapache · 2021-09-16T09:02:31Z

@rhtyd @sureshanaparti
this PR has many conflicts with #5239 which has been merged into master.
I think it is better to re-target to 4.17

what do you think ? @ravening @soreana

soreana · 2021-09-16T09:26:44Z

@weizhouapache I'm agree 👍

soreana · 2021-12-01T10:46:56Z

@weizhouapache I hope you are doing well :)
I'm going to work on this pr. How can I reproduce it?

weizhouapache · 2021-12-01T10:58:34Z

@weizhouapache I hope you are doing well :) I'm going to work on this pr. How can I reproduce it?

@soreana
what do you want to reproduce ? this is an enhancement, not a bug.

soreana · 2021-12-01T11:27:26Z

@weizhouapache I hope you are doing well :) I'm going to work on this pr. How can I reproduce it?

@soreana what do you want to reproduce ? this is an enhancement, not a bug.

Some test cases to see if it works or not would be enough.

weizhouapache · 2021-12-01T15:48:29Z

@weizhouapache I hope you are doing well :) I'm going to work on this pr. How can I reproduce it?

@soreana what do you want to reproduce ? this is an enhancement, not a bug.

Some test cases to see if it works or not would be enough.

@soreana
I think you can get more info on your wiki/jira, etc.

ravening · 2021-12-16T09:20:27Z

@weizhouapache I created a new pr for this in #5783

sureshanaparti · 2021-12-16T11:06:39Z

@weizhouapache I created a new pr for this in #5783

@weizhouapache @ravening is this PR no longer relevant (as the changes are ported to #5783)?

ravening · 2021-12-16T11:08:08Z

@weizhouapache I created a new pr for this in #5783

@weizhouapache @ravening is this PR no longer relevant (as the changes are ported to #5783)?

@sureshanaparti yes. this has conflicts.. i have resolved the conflicts and created new pr

ustcweizhou added 3 commits February 18, 2021 20:23

kvm: Handle storage issue on NFS/KVM in multiple ways

1cd7495

kvm: Schedule investigate tasks for disconnected hosts

ce62820

kvm: Add option 'noaction' for storage issue handling

f4e2f20

weizhouapache closed this Feb 22, 2021

weizhouapache reopened this Feb 22, 2021

weizhouapache marked this pull request as ready for review April 23, 2021 09:19

rohityadavcloud added this to the 4.16.0.0 milestone May 25, 2021

weizhouapache mentioned this pull request Jun 17, 2021

When using multiple NICs, VM HA does not work if you have configured zone wide NFS. #2708

Closed

weizhouapache mentioned this pull request Jun 29, 2021

kvm: fix VM HA on zone-wide storage pools #5164

Merged

12 tasks

Merge remote-tracking branch 'apache/main' into 4.16-kvm-storage-issues

9fd17bd

DaanHoogland reviewed Jun 30, 2021

View reviewed changes

sureshanaparti reviewed Sep 16, 2021

View reviewed changes

sureshanaparti modified the milestones: 4.16.0.0, 4.17.0.0 Sep 16, 2021

ravening mentioned this pull request Dec 16, 2021

4.16 kvm storage issues #5783

Closed

12 tasks

weizhouapache closed this Dec 16, 2021

rohityadavcloud removed this from the 4.17.0.0 milestone Mar 24, 2022

weizhouapache mentioned this pull request Apr 8, 2024

KVMHAMonitor getting initialized without host ha enabled #8682

Closed

	_investigatorExecutor = new ScheduledThreadPoolExecutor(InvestigateDisconnectedHostsPoolSize.value(), new NamedThreadFactory("DisconnectHostsInvestigator"));
	_investigatorExecutor = new ScheduledThreadPoolExecutor(InvestigateDisconnectedHostsPoolSize.value(), new NamedThreadFactory("DisconnectedHostsInvestigator"));

	"Default pool size to investigate disconnected hosts", false);
	"The thread pool size to investigate disconnected hosts", false);

kvm: Handle storage issue on NFS/KVM in multiple ways #4708

kvm: Handle storage issue on NFS/KVM in multiple ways #4708

Uh oh!

Conversation

ustcweizhou commented Feb 18, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

weizhouapache commented Feb 18, 2021

Uh oh!

rohityadavcloud commented May 25, 2021

Uh oh!

blueorangutan commented May 25, 2021

Uh oh!

blueorangutan commented May 25, 2021

Uh oh!

GabrielBrascher commented May 25, 2021

Uh oh!

nvazquez commented Jun 23, 2021

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland Jun 30, 2021

Choose a reason for hiding this comment

Uh oh!

weizhouapache Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

rohityadavcloud commented Sep 14, 2021

Uh oh!

sureshanaparti Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

sureshanaparti Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

sureshanaparti Sep 16, 2021

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Sep 16, 2021

Uh oh!

weizhouapache commented Sep 16, 2021

Uh oh!

soreana commented Sep 16, 2021

Uh oh!

soreana commented Dec 1, 2021

Uh oh!

weizhouapache commented Dec 1, 2021

Uh oh!

soreana commented Dec 1, 2021

Uh oh!

weizhouapache commented Dec 1, 2021

Uh oh!

ravening commented Dec 16, 2021

Uh oh!

sureshanaparti commented Dec 16, 2021

Uh oh!

ravening commented Dec 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants