Skip to content

Conversation

@davidjumani
Copy link
Contributor

@davidjumani davidjumani commented May 11, 2020

Support for CentOS 8 as mgmt/kvm host
Support for Ubuntu 20.04 as mgmt/kvm host
Support for XCP-ng 8.1 as Hypervisor.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Changes have been made in smoke tests to use test template for Xenserver/SCP-ng as XCP-ng 8 and above does not support built-in PV template. Since test template at present does not support iso9660 filesystem, ISO attach-detach test for the VM will skip mounting of the ISO.

- Splits commands to create user and grant access on database, the old
statement is no longer supported by MySQL 8.x
- `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove
that from db.properties conn parameters

For mysql-server 8.x setup the following changes were added/tested to
make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and
then restart the mysql-server process:

    server_id = 1
    sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
    innodb_rollback_on_timeout=1
    innodb_lock_wait_timeout=600
    max_connections=1000
    log-bin=mysql-bin
    binlog-format = 'ROW'

    default-authentication-plugin=mysql_native_password

Notice the last line above, this is to reset the old password based
authentication used by MySQL 5.x.

Developers can set empty password as follows:

    > sudo mysql -u root
    ALTER USER 'root'@'localhost' IDENTIFIED BY '';
@davidjumani
Copy link
Contributor Author

@weizhouapache We'll be adding our changes here, including the python3 support!

@rohityadavcloud
Copy link
Member

@davidjumani @shwstppr related PR for Ubuntu 20.04 with likely some overlapping parts - #4069

@weizhouapache
Copy link
Member

weizhouapache commented May 12, 2020

@davidjumani I suggest to create separated PR for python2 to python3 upgrade
it seems to be ready for review, as you have changed many files.

@davidjumani davidjumani force-pushed the adding-centos8-support branch from 5be4b2e to f5535b6 Compare May 12, 2020 08:25
@rohityadavcloud
Copy link
Member

@weizhouapache we want to keep them in this PR as CentOS8 ships by default with python3, it may be tricky in some cases to test this PR without adding some python3 support.

@weizhouapache
Copy link
Member

weizhouapache commented May 12, 2020

@weizhouapache we want to keep them in this PR as CentOS8 ships by default with python3, it may be tricky in some cases to test this PR without adding some python3 support.

@rhtyd @davidjumani
you can keep python3 changes in this branch, it is fine. It would be nice to create separated PR for python3 so we can test and merge it as soon as possible. it is important as it is required by both centos8 and ubuntu20.04 support.

we can upgrade to python3 without any changes for centos8 and ubuntu 20.04, as python3 can be tested on centos7 and ubuntu 18.04/ubuntu 16.04.

Comparing with a large PR with many changes, I prefer to create some small/medium PRs with one or two feature/improvement.

Copy link
Member

@rohityadavcloud rohityadavcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove changes to unnecessary files in tools and elsewhere. Please keep the scope limited to management/usage server scripts and kvm-agent.

davidjumani and others added 7 commits May 18, 2020 16:42
In libvirt repository, there are two related commits

2019-08-23 13:13 Daniel P. Berrangé            ● rpm: don't enable socket activation in upgrade if --listen present
2019-08-22 14:52 Daniel P. Berrangé            ● remote: forbid the --listen arg when systemd socket activation

In libvirt.spec.in

        /bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || :
        /bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :
Signed-off-by: Abhishek Kumar <[email protected]>
Signed-off-by: Abhishek Kumar <[email protected]>
@davidjumani davidjumani force-pushed the adding-centos8-support branch 2 times, most recently from f1aad4d to cae1374 Compare May 18, 2020 11:44
@davidjumani davidjumani force-pushed the adding-centos8-support branch from cae1374 to 67f40b5 Compare May 18, 2020 11:49
@DaanHoogland DaanHoogland changed the title [WIP] Adding Centos8 Support Adding Centos8 Support May 18, 2020
@vladimirpetrov
Copy link
Contributor

@blueorangutan help

@blueorangutan
Copy link

@vladimirpetrov I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [additional tests]
Mgmt OS options: ['centos6', 'centos7', 'centos8', 'ubuntu']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-centos8', 'kvm-ubuntu', 'xenserver-71', 'xenserver-65sp1', 'vmware-67u3', 'vmware-65u2', 'vmware-60u2', 'vmware-55u3', 'xcpng76', 'xcpng80', 'xcpng81', 'xenserver-74', 'xcpng74']
Additional tests: list of space separated tests with paths relative to the test/integration directory, for example: component/test_acl_listvm.py component/test_volumes.py
Note: when additional tests are passed, you need to specify mgmt server os and hypervisor or use the matrix command.

Blessed contributors for kicking Trillian test jobs: ['rhtyd', 'nvazquez', 'PaulAngus', 'borisstoyanov', 'DaanHoogland', 'shwstppr', 'andrijapanicsb', 'Spaceman1984', 'Pearl1594', 'davidjumani', 'harikrishna-patnala', 'vladimirpetrov', 'sureshanaparti']

@vladimirpetrov
Copy link
Contributor

@blueorangutan test centos8 kvm-centos8

@blueorangutan
Copy link

@vladimirpetrov a Trillian-Jenkins test job (centos8 mgmt + kvm-centos8) has been kicked to run smoke tests

@rohityadavcloud
Copy link
Member

Env issue failed, rekicked job at the backend (job/acs-pr-trillian/2449/console)

@blueorangutan
Copy link

Trillian test result (tid-2450)
Environment: kvm-centos8 (x2), Advanced Networking with Mgmt server 8
Total time taken: 52615 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2450-kvm-centos8.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_01_add_delete_kubernetes_supported_version Error 1807.51 test_kubernetes_supported_versions.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Failure 430.26 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Error 3922.06 test_vpc_redundant.py
test_hostha_kvm_host_fencing Error 170.85 test_hostha_kvm.py

@blueorangutan
Copy link

Trillian test result (tid-2452)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 8
Total time taken: 52647 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2452-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 83 look OK, 2 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_01_add_delete_kubernetes_supported_version Error 1808.71 test_kubernetes_supported_versions.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Failure 501.15 test_vpc_redundant.py

@blueorangutan
Copy link

Trillian test result (tid-2451)
Environment: xenserver-74 (x2), Advanced Networking with Mgmt server 8
Total time taken: 59393 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2451-xenserver-74.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_01_add_delete_kubernetes_supported_version Error 1807.42 test_kubernetes_supported_versions.py
test_01_scale_vm Failure 27.73 test_scale_vm.py
test_11_migrate_volume_and_change_offering Error 8.42 test_volumes.py

@rohityadavcloud
Copy link
Member

@blueorangutan test centos8 xenserver-71

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos8 mgmt + xenserver-71) has been kicked to run smoke tests

@blueorangutan
Copy link

Trillian test result (tid-2457)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 8
Total time taken: 55363 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2457-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 83 look OK, 2 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_01_add_delete_kubernetes_supported_version Error 1568.38 test_kubernetes_supported_versions.py
test_01_scale_vm Failure 25.70 test_scale_vm.py

Copy link
Contributor

@borisstoyanov borisstoyanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, based on marvin results. There are some failures, but I don't think they are related

@andrijapanicsb
Copy link
Contributor

@blueorangutan test centos7 kvm-centos7

@blueorangutan
Copy link

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@andrijapanicsb
Copy link
Contributor

LGTM

Any outstanding issues with tests, will have to be resolved for 4.15

@rohityadavcloud
Copy link
Member

Thanks @andrijapanicsb @borisstoyanov - agree failing tests are all intermittent failures. Merging this.

@rohityadavcloud rohityadavcloud merged commit d949302 into apache:master Aug 17, 2020
@blueorangutan
Copy link

Trillian test result (tid-2476)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49408 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2476-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_outofbandmanagement.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_01_add_delete_kubernetes_supported_version Error 1806.94 test_kubernetes_supported_versions.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Failure 439.99 test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers Error 361.84 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Error 3884.86 test_vpc_redundant.py
test_hostha_kvm_host_fencing Error 8.76 test_hostha_kvm.py

@andrijapanicsb
Copy link
Contributor

@weizhouapache were you able to test this PR against Ubuntu 20.04?. thx

@weizhouapache
Copy link
Member

@andrijapanicsb tested some actions and found some issues, see PR #4301 and #4303

@rohityadavcloud
Copy link
Member

Btw I'm running a custom build of 4.14 on Ubuntu 20.04 on raspberrypi4 using commits from Wei/this PR @andrijapanicsb

@weizhouapache
Copy link
Member

@rhtyd @PaulAngus maybe we can work together to make 4.14 on ubuntu 20.04.

I have backported some changes in this PR to 4.14, it is tested ok.
https://github.com/ustcweizhou/cloudstack/commits/4.14-ubuntu2004

@andrijapanicsb
Copy link
Contributor

not trying to be a smart one, but don't we only implement fixes on 4.14.x release? (i.e. no new features) - or am I confusing things here @weizhouapache ? Otherwise that sounds good (from a user perspective)

@weizhouapache
Copy link
Member

not trying to be a smart one, but don't we only implement fixes on 4.14.x release? (i.e. no new features) - or am I confusing things here @weizhouapache ? Otherwise that sounds good (from a user perspective)

@andrijapanicsb true. 4.14 accepts only bug fixes. so I will not create PR for it.
However, it is good to provide packages for cloudstack users who want to run cloudstack 4.14.X on ubuntu 20.04.
The packages are not official. it could be something like ubuntu/ppa: https://launchpad.net/ubuntu/+ppas

@vladimirpetrov
Copy link
Contributor

Management server with Ubuntu 20.04 - LGTM, here is what I've tested:

  • VM network - add/delete secondary IP, change IP address, change default NIC, add/remove NIC
  • Guest networks - create/remove/restart with clean up isolated, L2 and shared network, add/remove egress, firewall and port-forwarding rules
  • Images - register/delete ISO, register/delete template,
  • Volumes - create, attach/detach, migrate, create/restore snapshot, create template from volume, download volume, resize, delete
  • System VMs operations - start/stop, reboot, migrate, get diag. data, run diagnostics, destroy
  • Storages - add/remove primary and secondary storages

@andrijapanicsb
Copy link
Contributor

It's worth mentioning, that on AMD EPYC CPUs, we had to upgrade Ubuntu kernel to 5.8.1+ in order to resolve a Qemu bug with CPU registers and stuff....where the VM can't be started at all.

@andrijapanicsb
Copy link
Contributor

(same goes for CentOS 8, had to upgrade to Elrepo 5.8+ kernel to solve the same issues)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.