Skip to content

Conversation

@davidjumani
Copy link
Contributor

@davidjumani davidjumani commented Dec 8, 2020

Description

This PR enhances #3945 by ensuring that all to-be System templates are cross zone

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

@rohityadavcloud rohityadavcloud added this to the 4.15.0.0 milestone Dec 8, 2020
@rohityadavcloud
Copy link
Member

cc @ravening @weizhouapache pl review
@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@ravening
Copy link
Member

ravening commented Dec 8, 2020

cc @ravening @weizhouapache pl review
@blueorangutan package

@rhtyd will review this when its ready

@blueorangutan
Copy link

@ravening a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@davidjumani
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2468

@davidjumani
Copy link
Contributor Author

@blueorangutan test

@blueorangutan
Copy link

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

s_logger.info("Copying template to all zones since it will be a system template");
List<DataCenterVO> zones = _dcDao.listEnabledZones();
if (zones != null) {
List<Long> zoneIds = zones.stream().map(zone -> zone.getId()).collect(Collectors.toList());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we skip the zone on which the template is already in?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to be handled by

DataStore dstSecStore = getImageStore(destZoneId, templateId);
if (dstSecStore != null) {
s_logger.debug("There is template " + templateId + " in secondary storage " + dstSecStore.getName() +
" in zone " + destZoneId + " , don't need to copy");
continue;
}

@blueorangutan
Copy link

Trillian test result (tid-3321)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32648 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3321-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 86 look OK, 0 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File

@davidjumani davidjumani changed the title template: Copying templates to all zones if type changed to system template: Ensuring template is cross zone if type changed to system Dec 9, 2020
@davidjumani
Copy link
Contributor Author

Updated the PR to just add a cross zone check when trying to change a template to a System / Builtin template

@davidjumani davidjumani force-pushed the fix-template-type branch 3 times, most recently from 022a476 to ab2f68c Compare December 9, 2020 06:31
@davidjumani
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2470

@davidjumani
Copy link
Contributor Author

@blueorangutan test

@blueorangutan
Copy link

Trillian test result (tid-3323)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 70571 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3323-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_storage_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 23 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_03_deploy_and_upgrade_kubernetes_cluster Failure 249.40 test_kubernetes_clusters.py
ContextSuite context=TestResetVmOnReboot>:setup Error 0.00 test_reset_vm_on_reboot.py
ContextSuite context=TestRouterDHCPHosts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup Error 0.00 test_resource_accounting.py
ContextSuite context=TestRouterDns>:setup Error 0.00 test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup Error 0.00 test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestIsolatedNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
test_01_sys_vm_start Failure 0.14 test_secondary_storage.py
ContextSuite context=TestCpuCapServiceOfferings>:setup Error 0.00 test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup Error 0.19 test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup Error 0.00 test_snapshots.py
test_01_list_sec_storage_vm Failure 0.06 test_ssvm.py
test_02_list_cpvm_vm Failure 0.05 test_ssvm.py
test_03_ssvm_internals Failure 0.05 test_ssvm.py
test_04_cpvm_internals Failure 0.05 test_ssvm.py
test_05_stop_ssvm Failure 0.05 test_ssvm.py
test_06_stop_cpvm Failure 0.05 test_ssvm.py
test_07_reboot_ssvm Failure 0.05 test_ssvm.py
test_08_reboot_cpvm Failure 0.05 test_ssvm.py
test_09_destroy_ssvm Failure 0.05 test_ssvm.py
test_10_destroy_cpvm Failure 0.05 test_ssvm.py
ContextSuite context=TestVMWareStoragePolicies>:setup Error 0.00 test_storage_policy.py
test_02_create_template_with_checksum_sha1 Error 65.55 test_templates.py
test_03_create_template_with_checksum_sha256 Error 65.55 test_templates.py
test_04_create_template_with_checksum_md5 Error 65.55 test_templates.py
test_05_create_template_with_no_checksum Error 65.54 test_templates.py
test_02_deploy_vm_from_direct_download_template Error 1.30 test_templates.py
test_03_deploy_vm_wrong_checksum Error 1.34 test_templates.py
ContextSuite context=TestTemplates>:setup Error 17.41 test_templates.py
ContextSuite context=TestISOUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestLBRuleUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestNatRuleUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestPublicIPUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestSnapshotUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestVmUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestVolumeUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestVpnUsage>:setup Error 0.00 test_usage.py
ContextSuite context=Test01DeployVM>:setup Error 0.00 test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setup Error 0.00 test_vm_life_cycle.py
test_14_secure_to_secure_vm_migration Error 11.41 test_vm_life_cycle.py
test_15_secured_to_nonsecured_vm_migration Error 74.06 test_vm_life_cycle.py
test_16_nonsecured_to_secured_vm_migration Error 1.29 test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setup Error 1.87 test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setup Error 0.00 test_volumes.py
ContextSuite context=TestVolumes>:setup Error 0.00 test_volumes.py
ContextSuite context=TestVPCRedundancy>:setup Error 0.00 test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup Error 0.00 test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible Error 1514.91 test_hostha_kvm.py

@rohityadavcloud rohityadavcloud modified the milestones: 4.15.0.0, 4.15.1.0 Dec 11, 2020
@davidjumani davidjumani marked this pull request as ready for review December 11, 2020 10:26
@rohityadavcloud
Copy link
Member

@davidjumani can you kick pkg/test again?

@davidjumani
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2479

@davidjumani
Copy link
Contributor Author

@blueorangutan test

@blueorangutan
Copy link

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan
Copy link

Trillian test result (tid-3336)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34583 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3336-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_04_rvpc_privategw_static_routes Failure 863.31 test_privategw_acl.py

@rohityadavcloud
Copy link
Member

@davidjumani can you revert the two lines I've comments on, move the conditional up if that makes sense? Let's keep the changes minimal if possible, easier to review and test. Thanks.

@davidjumani
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2490

@rohityadavcloud
Copy link
Member

@weizhouapache @DaanHoogland should we consider this for RC2, with current changes it addresses concerns from #3945
@blueorangutan test

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@weizhouapache
Copy link
Member

@weizhouapache @DaanHoogland should we consider this for RC2, with current changes it addresses concerns from #3945
@blueorangutan test

@rhtyd I am ok to merge this into 4.15.0.0

@DaanHoogland DaanHoogland modified the milestones: 4.15.1.0, 4.15.0.0 Dec 14, 2020
@DaanHoogland DaanHoogland merged commit 2f78b88 into apache:master Dec 14, 2020
@DaanHoogland DaanHoogland deleted the fix-template-type branch December 14, 2020 08:59
@blueorangutan
Copy link

Trillian test result (tid-3347)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 72524 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3347-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_migration.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_storage_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 23 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_04_rvpc_privategw_static_routes Failure 239.75 test_privategw_acl.py
ContextSuite context=TestResetVmOnReboot>:setup Error 0.00 test_reset_vm_on_reboot.py
ContextSuite context=TestRouterDHCPHosts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup Error 0.00 test_resource_accounting.py
ContextSuite context=TestRouterDns>:setup Error 0.00 test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup Error 0.00 test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestIsolatedNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
test_01_sys_vm_start Failure 0.12 test_secondary_storage.py
ContextSuite context=TestCpuCapServiceOfferings>:setup Error 0.00 test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup Error 0.18 test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup Error 0.00 test_snapshots.py
test_01_list_sec_storage_vm Failure 0.05 test_ssvm.py
test_02_list_cpvm_vm Failure 0.04 test_ssvm.py
test_03_ssvm_internals Failure 0.04 test_ssvm.py
test_04_cpvm_internals Failure 0.04 test_ssvm.py
test_05_stop_ssvm Failure 0.04 test_ssvm.py
test_06_stop_cpvm Failure 0.04 test_ssvm.py
test_07_reboot_ssvm Failure 0.04 test_ssvm.py
test_08_reboot_cpvm Failure 0.05 test_ssvm.py
test_09_destroy_ssvm Failure 0.04 test_ssvm.py
test_10_destroy_cpvm Failure 0.04 test_ssvm.py
ContextSuite context=TestVMWareStoragePolicies>:setup Error 0.00 test_storage_policy.py
test_02_create_template_with_checksum_sha1 Error 65.53 test_templates.py
test_03_create_template_with_checksum_sha256 Error 65.56 test_templates.py
test_04_create_template_with_checksum_md5 Error 65.52 test_templates.py
test_05_create_template_with_no_checksum Error 65.54 test_templates.py
test_02_deploy_vm_from_direct_download_template Error 1.28 test_templates.py
test_03_deploy_vm_wrong_checksum Error 1.35 test_templates.py
ContextSuite context=TestTemplates>:setup Error 17.43 test_templates.py
ContextSuite context=TestISOUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestLBRuleUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestNatRuleUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestPublicIPUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestSnapshotUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestVmUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestVolumeUsage>:setup Error 0.00 test_usage.py
ContextSuite context=TestVpnUsage>:setup Error 0.00 test_usage.py
ContextSuite context=Test01DeployVM>:setup Error 0.00 test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setup Error 0.00 test_vm_life_cycle.py
test_14_secure_to_secure_vm_migration Error 11.43 test_vm_life_cycle.py
test_15_secured_to_nonsecured_vm_migration Error 74.10 test_vm_life_cycle.py
test_16_nonsecured_to_secured_vm_migration Error 1.27 test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setup Error 1.96 test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setup Error 0.00 test_volumes.py
ContextSuite context=TestVolumes>:setup Error 0.00 test_volumes.py
ContextSuite context=TestVPCRedundancy>:setup Error 0.00 test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup Error 0.00 test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible Error 1514.47 test_hostha_kvm.py

qrry added a commit to qrry/cloudstack that referenced this pull request Dec 23, 2020
* master:
  server: add conditions for custom offerings (apache#4540)
  vr: Ensuring dnsmasq.leases file is populated (apache#4529)
  template: Ensuring template is cross zone if type changed to system (apache#4522)
  storage: Fix hypervisor type cast to string (apache#4516)
  db upgrade: fix sql exception: Access denied; you need (at least one of) the SUPER privilege(s) for this operation (apache#4533)
  CLOUDSTACK-10423:Potential sensitive information disclosure (apache#4536)
  jobs: The patch remove the password from resultObject and make it be humanreadable (apache#4538)
  listphysicalnetworks: Honouring keyword parameter (apache#4511)
  Fix NPE when Volume exists on secondary store but doesn't have a download URL (apache#4530)
  apidoc issue (apache#4532)
  db: Fix description of volume.stats.interval which is in milliseconds not seconds (apache#4526)
  kvm: set cpu topology only if cpucore per socket is positive value (apache#4527)
  xenserver: check and eject patch vbd for systemvms (apache#4525)
  Fix warning when setup cloudstack-common (apache#4523)
  kvm: FIX cpucorespersocket is not working on KVM (apache#4497)
  change debug to warn for unknown exceptions (apache#4521)
  Fix failure in validating IP address in case of multiple Management Servers (apache#4507)
  Update log output for FirstFitPlanner (apache#4515)
  ui: deprecate old UI and move to legacy to be served at /client/legacy (apache#4518)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants