KVM live storage migration intra cluster from NFS source and destination #2983

nvazquez · 2018-10-29T19:20:41Z

Description

Feature Specification: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95653548

Live storage migration on KVM under these conditions:

From source and destination hosts within the same cluster
From NFS primary storage to NFS cluster-wide primary storage
Source NFS and destination NFS storage mounted on hosts

In order to enable this functionality, database should be updated in order to enable live storage capacibilty for KVM, if previous conditions are met. This is due to existing conflicts between qemu and libvirt versions. This has been tested on CentOS 6 hosts.

Additional notes:

To use this feature set the storage_motion_supported=1 in the hypervisor_capability table for KVM. This is done by default as the feature may not work in some environments, read below.
This feature of online storage+VM migration for KVM will only work with CentOS6 and possible Ubuntu as KVM hosts but not with CentOS7 due to:

https://bugs.centos.org/view.php?id=14026
https://bugzilla.redhat.com/show_bug.cgi?id=1219541
On CentOS7 the error we see is: " error: unable to execute QEMU command 'migrate': this feature or command is not currently supported" (reference https://ask.openstack.org/en/question/94186/live-migration-unable-to-execute-qemu-command-migrate/). Reading through various lists looks like the migrate feature with qemu may be available with paid versions of RHEL-EV but not centos7 however this works with CentOS6.

Fix for CentOS 7:

Create repo file on /etc/yum.repos.d/:

[qemu-kvm-rhev]
name=oVirt rebuilds of qemu-kvm-rhev
baseurl=http://resources.ovirt.org/pub/ovirt-3.5/rpm/el7Server/
mirrorlist=http://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-3.5-el7Server
enabled=1
skip_if_unavailable=1
gpgcheck=0

yum install qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 qemu-kvm-ev-2.3.0-29.1.el7.x86_64 qemu-img-ev-2.3.0-29.1.el7.x86_64
Reboot host

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

Tested on environment having 2 KVM CentOS 6 hosts.

Note: CentOS 7 hosts report an issue around qemu version:
org.libvirt.LibvirtException: internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported

Fix for CentOS 7:

Create repo file on /etc/yum.repos.d/:

[qemu-kvm-rhev]
name=oVirt rebuilds of qemu-kvm-rhev
baseurl=http://resources.ovirt.org/pub/ovirt-3.5/rpm/el7Server/
mirrorlist=http://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-3.5-el7Server
enabled=1
skip_if_unavailable=1
gpgcheck=0

yum install qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 qemu-kvm-ev-2.3.0-29.1.el7.x86_64 qemu-img-ev-2.3.0-29.1.el7.x86_64
Reboot host

...tion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

...rc/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapperTest.java

DaanHoogland

looks good overall, some comments. A biggy is; there is no integration test. That would seem needed for a functional change like this. @nvazquez /cc @rhtyd @borisstoyanov ?

wido · 2018-10-30T09:18:11Z

core/src/main/java/com/cloud/agent/api/storage/CheckStorageAvailabilityCommand.java

+    public boolean executeInSequence() {
+        return false;
+    }
+}


Add a newline here :)

Done, thanks

wido · 2018-10-30T09:18:37Z

engine/schema/src/main/resources/META-INF/db/schema-41120to41200.sql


 -- PR#1448 update description of 'execute.in.sequence.network.element.commands' parameter to reflect an unused command that has been removed. The removed class command is 'UserDataCommand'.
-update `cloud`.`configuration` set description = 'If set to true, DhcpEntryCommand, SavePasswordCommand, VmDataCommand will be synchronized on the agent side. If set to false, these commands become asynchronous. Default value is false.' where name = 'execute.in.sequence.network.element.commands'; 
+update `cloud`.`configuration` set description = 'If set to true, DhcpEntryCommand, SavePasswordCommand, VmDataCommand will be synchronized on the agent side. If set to false, these commands become asynchronous. Default value is false.' where name = 'execute.in.sequence.network.element.commands';


Update by accident of this file?

Had previously included an update sql on the hypervisor_cappabilities to enable live storage motion on KVM. But due to conflicting versions on qemu just decided to remove it and must have removed an end of line or extra space, will sort it out

wido · 2018-10-30T09:21:15Z

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImg.java


+    private String getQemuImgPathScript = String.format("which %s >& /dev/null; " +
+                    "if [ $? -gt 0 ]; then echo \"%s\"; else echo \"%s\"; fi",
+            cloudQemuImgPath, _qemuImgPath, cloudQemuImgPath);


We have a Qemu utility in the KVM code, can't we offload this there?

As this feature requires qemu-img command supporting snapshots I have introduced this check. This cloud-qemu-img can be included on KVM hosts to support snapshots as per: http://www.nux.ro/oldblog/archive/2014/01/Taking_KVM_volume_snapshots_with_Cloudstack_4_2_on_CentOS_6_5.html
If cloud-qemu-img is present then we can use it, and if not just qemu-img. It is also done this way on scripts: https://github.com/apache/cloudstack/search?q=cloud-qemu-img&unscoped_q=cloud-qemu-img.

wido · 2018-10-30T09:21:45Z

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuImg.java


    /* The qemu-img binary. We expect this to be in $PATH */
    public String _qemuImgPath = "qemu-img";
+    private String cloudQemuImgPath = "cloud-qemu-img";


This binary does not exist on Ubuntu/Debian. Why the name change?

Just to make sure it works with snapshot support. If the binary is present on the KVM host it will be used, and if not just the qemu-img.

nvazquez · 2018-10-30T13:37:25Z

Thanks @wido @DaanHoogland for reviewing, I'll address your comments soon. Along with @borisstoyanov we'll be adding integration tests on this PR.

nvazquez · 2018-10-30T21:06:23Z

Feature specification link added to the PR description

mike-tutkowski · 2018-10-31T01:11:10Z

@nvazquez Hi Nicolas - Just an FYI that I plan to look at this tomorrow.

nvazquez · 2018-10-31T11:38:32Z

Thanks @mike-tutkowski that's great

mike-tutkowski

This looks good. I just had the one comment on the code.

...tion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java

borisstoyanov

@nvazquez, I think this one is missing the marvin tests we created in the private repo

rohityadavcloud · 2018-12-28T11:17:29Z

@nvazquez I've rebased and fixed conflicts against latest master. Can you check the final rebased commit against the original commit here: 5439cf3

rohityadavcloud · 2018-12-29T11:26:26Z

@blueorangutan package

rohityadavcloud · 2018-12-31T09:53:56Z

@blueorangutan package

blueorangutan · 2018-12-31T09:54:26Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-07T00:09:35Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2856

nvazquez · 2019-06-07T00:09:53Z

@blueorangutan test

blueorangutan · 2019-06-07T00:11:00Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-06-07T08:53:49Z

Trillian test result (tid-3662)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29564 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2983-t3662-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 69 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_upload_direct_download_certificates	`Failure`	0.04	test_direct_download.py
test_02_redundant_VPC_default_routes	`Error`	0.75	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Failure`	317.93	test_vpc_redundant.py

rohityadavcloud · 2019-06-07T09:14:15Z

@nvazquez can you check the test_02_upload_direct_download_certificates failure?
@blueorangutan package

blueorangutan · 2019-06-07T09:15:02Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-07T09:49:03Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2864

nvazquez · 2019-06-07T20:32:17Z

@blueorangutan package

blueorangutan · 2019-06-07T20:33:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-07T20:53:10Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2865

nvazquez · 2019-06-07T20:59:42Z

@blueorangutan test

blueorangutan · 2019-06-07T21:00:04Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-06-08T04:33:56Z

Trillian test result (tid-3669)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 25753 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2983-t3669-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 71 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2019-06-08T04:51:54Z

@rhtyd marvin test failure fixed, now all tests are passing

Considering that we have addressed a considerable effort on allowing KVM to perform storage data motion, this PR proposes updating the 'hypervisor_capabilities' table setting the 'storage_motion_supported' to '1' for KVM. PRs that implemented KVM storage motion features: Non-managed storages apache#2997 KVM VM live migration with ROOT volume on file storage type apache#2983 KVM live storage migration intra cluster from NFS source and destination Managed storages apache#2298 CLOUDSTACK-9620: Enhancements for managed storage Signed-off-by: Rohit Yadav <[email protected]>

- Removes CentOS6/el6 packaging (voting thread reference https://markmail.org/message/u3ka4hwn2lzwiero) - Add upgrade path from 4.13 to 4.14 - Enable live storage migration support for KVM by default as el6 is deprecated - PRs using live storage migration #2997 KVM VM live migration with ROOT volume on file storage type #2983 KVM live storage migration intra cluster from NFS source and destination #2298 CLOUDSTACK-9620: Enhancements for managed storage Signed-off-by: Rohit Yadav <[email protected]>

nvazquez requested review from DaanHoogland, GabrielBrascher, mike-tutkowski, rafaelweingartner and rohityadavcloud October 29, 2018 19:20

DaanHoogland reviewed Oct 30, 2018

View reviewed changes

...tion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java Outdated Show resolved Hide resolved

DaanHoogland reviewed Oct 30, 2018

View reviewed changes

...tion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java Show resolved Hide resolved

DaanHoogland reviewed Oct 30, 2018

View reviewed changes

...rc/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapperTest.java Show resolved Hide resolved

DaanHoogland previously requested changes Oct 30, 2018

View reviewed changes

wido previously requested changes Oct 30, 2018

View reviewed changes

mike-tutkowski reviewed Nov 1, 2018

View reviewed changes

...tion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java Show resolved Hide resolved

mike-tutkowski approved these changes Nov 1, 2018

View reviewed changes

...tion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java Show resolved Hide resolved

GabrielBrascher mentioned this pull request Nov 3, 2018

Allow KVM VM live migration with ROOT volume on file storage type #2997

Merged

nvazquez mentioned this pull request Nov 5, 2018

[Placeholder] Add new data motion strategy for KVM non managed storage live migrations #3004

Closed

5 tasks

nvazquez added this to the 4.12.0.0 milestone Nov 5, 2018

nvazquez added component:kvm component:primary-storage labels Nov 5, 2018

borisstoyanov reviewed Dec 28, 2018

View reviewed changes

rohityadavcloud self-assigned this Dec 28, 2018

rohityadavcloud force-pushed the kvmlivestoragemigration-master branch from 50de309 to 5439cf3 Compare December 28, 2018 09:54

rohityadavcloud force-pushed the kvmlivestoragemigration-master branch from 5439cf3 to 224b74d Compare December 28, 2018 11:37

rohityadavcloud approved these changes Dec 28, 2018

View reviewed changes

Fix failing marvin test

ae397df

nvazquez closed this Jun 8, 2019

nvazquez reopened this Jun 8, 2019

rohityadavcloud approved these changes Jun 10, 2019

View reviewed changes

rohityadavcloud merged commit 0fbf500 into apache:master Jun 10, 2019

GabrielBrascher mentioned this pull request Jul 22, 2019

Enable KVM storage data motion on KVM hypervisor_capabilities #3508

Closed

5 tasks

ustcweizhou mentioned this pull request Mar 4, 2020

server: fix vm migration failed if iso is attached #3929

Closed

5 tasks

nvazquez deleted the kvmlivestoragemigration-master branch April 6, 2020 14:45

slavkap mentioned this pull request Apr 2, 2021

issues with live migration on centos (was: Trillian test failures) #4757

Closed

snyk-bot mentioned this pull request Aug 4, 2021

[Snyk] Fix for 2 vulnerabilities hixio-mh/cloudstack#78

Open

nvazquez mentioned this pull request Aug 5, 2021

Fail to do a Live Migration with storage via migrateVirtualMachineWithVolume API. #4183

Closed

snyk-bot mentioned this pull request Aug 5, 2021

[Snyk] Fix for 1 vulnerabilities hixio-mh/cloudstack#80

Open

snyk-bot mentioned this pull request Sep 2, 2021

[Snyk] Fix for 3 vulnerabilities hixio-mh/cloudstack#96

Open

weizhouapache mentioned this pull request Jun 29, 2023

Move KVM VM between Hosts using Local Disk Storage and Clusters with different Primary Storage #7480

Closed

KVM live storage migration intra cluster from NFS source and destination #2983

KVM live storage migration intra cluster from NFS source and destination #2983

Uh oh!

Conversation

nvazquez commented Oct 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Oct 30, 2018

Uh oh!

nvazquez commented Oct 30, 2018

Uh oh!

mike-tutkowski commented Oct 31, 2018

Uh oh!

nvazquez commented Oct 31, 2018

Uh oh!

mike-tutkowski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

rohityadavcloud commented Dec 28, 2018

Uh oh!

rohityadavcloud commented Dec 29, 2018

Uh oh!

rohityadavcloud commented Dec 31, 2018

Uh oh!

blueorangutan commented Dec 31, 2018

Uh oh!

blueorangutan commented Jun 7, 2019

Uh oh!

nvazquez commented Jun 7, 2019

Uh oh!

blueorangutan commented Jun 7, 2019

Uh oh!

blueorangutan commented Jun 7, 2019

Uh oh!

rohityadavcloud commented Jun 7, 2019

Uh oh!

blueorangutan commented Jun 7, 2019

Uh oh!

blueorangutan commented Jun 7, 2019

Uh oh!

nvazquez commented Jun 7, 2019

Uh oh!

blueorangutan commented Jun 7, 2019

Uh oh!

blueorangutan commented Jun 7, 2019

Uh oh!

nvazquez commented Jun 7, 2019

Uh oh!

blueorangutan commented Jun 7, 2019

nvazquez commented Oct 29, 2018 •

edited

Loading