Update VM priority (cpu_shares) when live scaling it #6031

BryanMLima · 2022-02-22T17:22:28Z

Description

This PR addresses an issue when live scaling a VM with the KVM hypervisor. When the VM has the CPU cores or CPU speed increased, the cpu_shares (priority) of the VM is not updated accordingly. To overcome that, it was needed to manually restart the VM or using the command: virsh schedinfo --domain <vm_internal_name> --live cpu_shares=<cpu_shares_value>. This PR changes this to automatically update the cpu_shares when the CPU cores or speed is increased. It is worth mention that the current behavior caused steal time when VMs would compete for the CPU after a dynamic scale process.

Steps to reproduce:

Host
- 6 vCPUs X 2.39Ghz
- 16GB RAM
Custom service offering
- 1 to 6 vCPUs
- 2.35Ghz
VM (deploy config)
- 1 vCPU
- 1024MB RAM

The cpu_shares of a VM is calculated as follow:
cpu_shares = number of vCPUs X vCPU frequency in Mhz

Using the command virsh schedinfo --domain <internal_vm_name> the cpu_shares displayed was 2350, which is expected. However, when live scaling the VM to 4 vCPUs, which should return 9400 shares, it was the initial 2350 shares. If the VM is restarted, however, the 9400 shares is used, as it should without the need to restart the VM.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

How Has This Been Tested?

Using the same configuration and steps I used to describe the bug. Then, when live scaling a VM, it returns the correct cpu_shares without the need to restart it. Therefore, when the VM with the 2350 shares is live scaled to 4 vCPUs, the cpu_shares when using the command mentioned above, returned the correct value of 9400.

Moreover, I created unit tests for the affected methods.

GutoVeronezi

CLGTM, tested manually.

I've pointed just some writing fixes.

...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtScaleVmCommandWrapper.java

DaanHoogland

cltgm but some remarks and questions.

...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtScaleVmCommandWrapper.java

DaanHoogland · 2022-02-23T07:41:50Z

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtScaleVmCommandWrapper.java

+    protected void updateCpuShares(Domain dm, int newCpuShares) throws LibvirtException {
+        int oldCpuShares = LibvirtComputingResource.getCpuShares(dm);
+
+        if (oldCpuShares < newCpuShares) {


only up, never down? is this like nice for users or can the root user force something?

In ACS, the live scale process does not allow to reduce the resource of a running VM. Therefore, it can only occur upwards and the cpu_shares is calculated as follow:
cpu_shares = number of vCPUs X vCPU frequency in Mhz.
This if (the conditional) is to check if the vCPUs or CPU speed is higher than the current compute offering, as it makes no sense to decrease the priority of a VM that is having its resources increased. In other words, this if is to check if it is a memory scale only, which should not have its priority changed in the host.

it looks the new speed >= current speed, new cpu core >= current cpu core
so new cpu shares is always equal or bigger than old cpu shares.
the check may be unnecessary but looks ok.

Co-authored-by: dahn <[email protected]> Co-authored-by: Daniel Augusto Veronezi Salvador <[email protected]>

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtScaleVmCommandWrapper.java

sureshanaparti

code LGTM

DaanHoogland · 2022-02-25T10:33:54Z

@blueorangutan package

blueorangutan · 2022-02-25T10:35:02Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-25T11:41:06Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✔️ suse15. SL-JID 2714

DaanHoogland · 2022-02-25T12:50:48Z

@blueorangutan test

blueorangutan · 2022-02-25T12:52:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

weizhouapache

code lgtm

blueorangutan · 2022-02-25T22:54:40Z

Trillian test result (tid-3440)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34779 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6031-t3440-kvm-centos7.zip
Smoke tests completed. 91 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_disable_oobm_ha_state_ineligible	`Error`	1512.05	test_hostha_kvm.py

BryanMLima · 2022-02-28T13:57:06Z

@blueorangutan package

blueorangutan · 2022-02-28T13:58:03Z

@BryanMLima a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-28T14:40:04Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2730

BryanMLima · 2022-02-28T15:15:20Z

The error was caused while trying to download a Template, so it had no relation to the code itself.

weizhouapache · 2022-03-02T07:40:26Z

@blueorangutan package

blueorangutan · 2022-03-02T07:41:03Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-03-02T08:29:45Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2743

weizhouapache · 2022-03-02T09:13:20Z

@blueorangutan test

weizhouapache · 2022-03-02T09:14:38Z

@blueorangutan test ubuntu20 kvm-ubuntu20

blueorangutan · 2022-03-02T09:15:02Z

@weizhouapache a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

weizhouapache · 2022-03-02T09:23:38Z

@blueorangutan test

blueorangutan · 2022-03-02T09:25:05Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-03-02T18:44:37Z

Trillian test result (tid-3456)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32137 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6031-t3456-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2022-03-02T19:56:08Z

Trillian test result (tid-3452)
Environment: kvm-ubuntu20 (x2), Advanced Networking with Mgmt server u20
Total time taken: 36667 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6031-t3452-kvm-ubuntu20.zip
Smoke tests completed. 88 look OK, 4 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_primary_storage_disabled_host	`Error`	0.71	test_primary_storage.py
test_01_primary_storage_nfs	`Error`	0.18	test_primary_storage.py
ContextSuite context=TestStorageTags>:setup	`Error`	0.33	test_primary_storage.py
test_01_migrate_VM_and_root_volume	`Error`	75.34	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	50.01	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	161.85	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	283.74	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	149.28	test_vm_life_cycle.py
test_08_migrate_vm	`Error`	44.17	test_vm_life_cycle.py
test_02_list_snapshots_with_removed_data_store	`Error`	9.61	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	9.61	test_snapshots.py
test_hostha_kvm_host_degraded	`Error`	696.87	test_hostha_kvm.py
test_hostha_kvm_host_fencing	`Error`	689.52	test_hostha_kvm.py

weizhouapache · 2022-03-03T08:40:01Z

@blueorangutan test ubuntu20 kvm-ubuntu20

blueorangutan · 2022-03-03T08:41:04Z

@weizhouapache a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

blueorangutan · 2022-03-03T20:45:02Z

Trillian test result (tid-3466)
Environment: kvm-ubuntu20 (x2), Advanced Networking with Mgmt server u20
Total time taken: 41641 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6031-t3466-kvm-ubuntu20.zip
Smoke tests completed. 89 look OK, 3 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	`Error`	0.10	test_internal_lb.py
test_01_migrate_VM_and_root_volume	`Error`	79.33	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	47.82	test_vm_life_cycle.py
test_10_attachAndDetach_iso	`Error`	12.54	test_vm_life_cycle.py
test_hostha_enable_ha_when_host_disabled	`Error`	1.85	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	303.01	test_hostha_kvm.py

BryanMLima · 2022-03-07T19:02:03Z

@weizhouapache Could you run the tests again, since I am not in the whitelist? I verified the errors and, apparently, they have no relation to the code I wrote.

weizhouapache · 2022-03-07T19:13:29Z

@blueorangutan test ubuntu20 kvm-ubuntu20

blueorangutan · 2022-03-07T19:14:03Z

@weizhouapache a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

blueorangutan · 2022-03-07T19:18:08Z

Trillian Build Failed (tid-3519)

weizhouapache · 2022-03-07T19:22:18Z

@blueorangutan test ubuntu20 kvm-ubuntu20

blueorangutan · 2022-03-07T19:23:03Z

@weizhouapache a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

blueorangutan · 2022-03-08T05:54:10Z

Trillian test result (tid-3520)
Environment: kvm-ubuntu20 (x2), Advanced Networking with Mgmt server u20
Total time taken: 36070 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6031-t3520-kvm-ubuntu20.zip
Smoke tests completed. 91 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	70.02	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	49.82	test_vm_life_cycle.py

weizhouapache · 2022-03-10T10:23:08Z

@blueorangutan package

blueorangutan · 2022-03-10T10:24:03Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-03-10T11:11:00Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2828

weizhouapache · 2022-03-10T11:47:40Z

@blueorangutan test ubuntu20 kvm-ubuntu20

blueorangutan · 2022-03-10T11:49:03Z

@weizhouapache a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

blueorangutan · 2022-03-10T21:56:17Z

Trillian test result (tid-3559)
Environment: kvm-ubuntu20 (x2), Advanced Networking with Mgmt server u20
Total time taken: 34697 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6031-t3559-kvm-ubuntu20.zip
Smoke tests completed. 91 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	88.75	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	50.75	test_vm_life_cycle.py
test_10_attachAndDetach_iso	`Error`	12.54	test_vm_life_cycle.py

BryanMLima · 2022-03-11T20:02:18Z

The recurrent errors test_01_migrate_VM_and_root_volume and test_02_migrate_VM_with_two_data_disks seems to be related to the Ubuntu OS. I tested both situations on my lab, and the command ran smoothly. Moreover, my PR is not related to these commands. @DaanHoogland could you look into this? The test on CentOS didn't shown any errors either.

Pearl1594 · 2022-03-14T14:07:10Z

Ran the above failed tests on a 4.16.1 env - and the same behavior was observed. Test failures as mentioned aren't related to this PR - the specific issue is:

 Failed to migrated vm VM instance {id: "14", name: "i-9-14-VM", uuid: "a1ba2a16-2e59-43c9-bd43-d55309f51a0b", type="User"} along with its volumes. Can't find strategy to move data. Source Host: ref-trl-2732-k-Mu20-pearl-dsilva-kvm1, Destination Host: ref-trl-2732-k-Mu20-pearl-dsilva-kvm2, Volume UUIDs: 0b96a192-0ecc-4de3-84f6-6090230cd4a0

weizhouapache · 2022-03-14T14:09:10Z

thanks @Pearl1594 for investigation.

let's merge this PR @nvazquez

weizhouapache · 2022-03-14T14:11:45Z

I have created an issue for tracking. #6114
cc @nvazquez @Pearl1594

nvazquez · 2022-03-14T14:34:02Z

Merging based on approvals and tests results - failing tests are only for ubuntu 20 which are being tracked on #6114

Update VM priority (cpu_chares) when live scaling it

f23ecbc

DaanHoogland changed the title ~~Update VM priority (cpu_chares) when live scaling it~~ Update VM priority (cpu_shares) when live scaling it Feb 22, 2022

GutoVeronezi approved these changes Feb 22, 2022

View reviewed changes

...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java Outdated Show resolved Hide resolved

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtScaleVmCommandWrapper.java Outdated Show resolved Hide resolved

DaanHoogland reviewed Feb 23, 2022

View reviewed changes

weizhouapache added this to the 4.17.0.0 milestone Feb 23, 2022

BryanMLima and others added 2 commits February 23, 2022 11:50

Apply suggestions from code review

a5f7a5f

Co-authored-by: dahn <[email protected]> Co-authored-by: Daniel Augusto Veronezi Salvador <[email protected]>

Addressing javadoc review

1121349

DaanHoogland reviewed Feb 23, 2022

View reviewed changes

...vm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtScaleVmCommandWrapper.java Show resolved Hide resolved

Addressing review typo

bc0b698

DaanHoogland approved these changes Feb 25, 2022

View reviewed changes

sureshanaparti approved these changes Feb 25, 2022

View reviewed changes

weizhouapache approved these changes Feb 25, 2022

View reviewed changes

apache deleted a comment from blueorangutan Mar 8, 2022

apache deleted a comment from blueorangutan Mar 9, 2022

weizhouapache mentioned this pull request Mar 14, 2022

Smoke test failed on ubuntu 20.04 #6114

Closed

nvazquez merged commit afdc73f into apache:main Mar 14, 2022

Update VM priority (cpu_shares) when live scaling it #6031

Update VM priority (cpu_shares) when live scaling it #6031

Uh oh!

Conversation

BryanMLima commented Feb 22, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

How Has This Been Tested?

Uh oh!

GutoVeronezi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DaanHoogland Feb 23, 2022

Choose a reason for hiding this comment

Uh oh!

BryanMLima Feb 23, 2022

Choose a reason for hiding this comment

Uh oh!

weizhouapache Feb 25, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Feb 25, 2022

Uh oh!

blueorangutan commented Feb 25, 2022

Uh oh!

blueorangutan commented Feb 25, 2022

Uh oh!

DaanHoogland commented Feb 25, 2022

Uh oh!

blueorangutan commented Feb 25, 2022

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Feb 25, 2022

Uh oh!

BryanMLima commented Feb 28, 2022

Uh oh!

blueorangutan commented Feb 28, 2022

Uh oh!

blueorangutan commented Feb 28, 2022

Uh oh!

BryanMLima commented Feb 28, 2022

Uh oh!

weizhouapache commented Mar 2, 2022

Uh oh!

blueorangutan commented Mar 2, 2022

Uh oh!

blueorangutan commented Mar 2, 2022

Uh oh!

weizhouapache commented Mar 2, 2022

Uh oh!

weizhouapache commented Mar 2, 2022

Uh oh!

blueorangutan commented Mar 2, 2022

Uh oh!

weizhouapache commented Mar 2, 2022

Uh oh!

blueorangutan commented Mar 2, 2022

Uh oh!

blueorangutan commented Mar 2, 2022

Uh oh!

blueorangutan commented Mar 2, 2022

Uh oh!

weizhouapache commented Mar 3, 2022

Uh oh!