Add the option to filter by host when retrieving of unregistered VMs

[DaanHoogland]

Description

This PR allows to get unmanaged VMS for migrate to KVM per hosts to prevent timeouts rendering the UI useless for converting VMs.

Fixes: #9782

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

in an environment with KVM hosts and a vmware DC (defined in CloudStack or not)

• make sure there are VMs on every ESXi and
• go to the tooling page to migrate VMs from vmware to KVM
• go through the DC selection
• observe a host choice dropdown box and
• see how only VMs from the selected host are available.

How did you try to break this feature and the system with this change?

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[sureshanaparti]

@DaanHoogland is it possible to keep the results in memory for different list vmware vm calls, & provide some kind on pagination (maybe, using pagescount, page params) from UI?

[DaanHoogland]

@DaanHoogland is it possible to keep the results in memory for different list vmware vm calls, & provide some kind on pagination (maybe, using pagescount, page params) from UI?

yes, this would be in a map with the caller session as key and would have to return after initial results return and start an async job to get the rest in batches. not a very trivial change.

[DaanHoogland]

@sureshanaparti , I removed the needs-testing label as I don't think it is ready for testing yet.

[sureshanaparti]

@DaanHoogland is it possible to keep the results in memory for different list vmware vm calls, & provide some kind on pagination (maybe, using pagescount, page params) from UI?

yes, this would be in a map with the caller session as key and would have to return after initial results return and start an async job to get the rest in batches. not a very trivial change.

also, is it possible to support keyword based retrieval? if one knows the vm name in vmware, they can pass as keyword and can see in the result (this way results are also less, and can show up vms in the end of the original retrieval list)

[codecov]

Codecov Report

Attention: Patch coverage is 2.27273% with 473 lines in your changes missing coverage. Please review.

Project coverage is 15.16%. Comparing base (21416cd) to head (d39e23e).
Report is 63 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...m/cloud/hypervisor/vmware/mo/VirtualMachineMO.java	0.89%	108 Missing and 3 partials ⚠️
...d/hypervisor/vmware/manager/VmwareManagerImpl.java	6.89%	107 Missing and 1 partial ⚠️
...k/api/command/admin/zone/ListVmwareDcHostsCmd.java	0.00%	52 Missing ⚠️
...a/com/cloud/hypervisor/vmware/mo/DatacenterMO.java	0.00%	52 Missing ⚠️
...in/java/com/cloud/hypervisor/vmware/mo/HostMO.java	0.00%	42 Missing ⚠️
...in/java/com/cloud/hypervisor/vmware/mo/BaseMO.java	0.00%	31 Missing ⚠️
...ack/api/command/admin/zone/ListVmwareDcVmsCmd.java	0.00%	17 Missing ⚠️
...stack/api/command/admin/zone/ListVmwareDcsCmd.java	0.00%	13 Missing ⚠️
...mand/admin/zone/ListVsphereStoragePoliciesCmd.java	0.00%	8 Missing ⚠️
...nd/admin/zone/ImportVsphereStoragePoliciesCmd.java	0.00%	7 Missing ⚠️
... and 9 more

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #9925      +/-   ##
============================================
+ Coverage     15.13%   15.16%   +0.03%     
- Complexity    11272    11305      +33     
============================================
  Files          5408     5411       +3     
  Lines        473955   473802     -153     
  Branches      57810    57805       -5     
============================================
+ Hits          71725    71871     +146     
+ Misses       394212   393897     -315     
- Partials       8018     8034      +16     

Flag	Coverage Δ
uitests	4.29% <ø> (-0.01%)	⬇️
unittests	15.89% <2.27%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[DaanHoogland]

@DaanHoogland is it possible to keep the results in memory for different list vmware vm calls, & provide some kind on pagination (maybe, using pagescount, page params) from UI?

yes, this would be in a map with the caller session as key and would have to return after initial results return and start an async job to get the rest in batches. not a very trivial change.

also, is it possible to support keyword based retrieval? if one knows the vm name in vmware, they can pass as keyword and can see in the result (this way results are also less, and can show up vms in the end of the original retrieval list)

I am going to leave this out of scope. It requires some extra work that I do not know of yet.

[shwstppr]

@DaanHoogland this PR is missing any useful description so I'm not sure if I understood the change completely but based on the changes I think it might be missing some changes and will need testing.

• A lot many changes are unrelated to the issue and only refactoring, maybe worth specifying what is intended.
• With UI, I mentioned my concern - this may result in unnecessary retrieval of results
• New parameters are added to the API but their use is not clear.
• listVmwareDcVms API class should not be based on the BaseListCmd as with this change it may follow pageSize param but it still doesn't follow page param which can be confusing during custom implementations

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12378

[DaanHoogland]

@blueorangutan test matrix

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-12343)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 45678 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9925-t12343-kvm-ol8.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[blueorangutan]

[SF] Trillian test result (tid-12344)
Environment: kvm-ubuntu22 (x2), Advanced Networking with Mgmt server u22
Total time taken: 51406 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9925-t12344-kvm-ubuntu22.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[blueorangutan]

[SF] Trillian test result (tid-12346)
Environment: xcpng82 (x2), Advanced Networking with Mgmt server ol9
Total time taken: 69584 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9925-t12346-xcpng82.zip
Smoke tests completed. 131 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_condensed_drs_algorithm	Failure	170.04	test_cluster_drs.py
test_02_balanced_drs_algorithm	Failure	175.24	test_cluster_drs.py
test_01_non_strict_host_anti_affinity	Error	233.81	test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity	Error	139.78	test_nonstrict_affinity_group.py

[DaanHoogland]

@vladimirpetrov , can you approve this yet?
@shwstppr , are your concerns answered?

[vladimirpetrov]

LGTM based on manual testing.