Agent 4.21.0.0 secondarystoragevm

[jonathaanbastin]

problem

If you upgrade the Ubuntu KVM cloustack-agent to 4.21.0.0 it longer power of or redeployes.

2025-09-01 15:42:44,488 INFO [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:b0554662) No existing libvirtd connection found. Opening a new one 2025-09-01 15:42:44,490 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:b0554662) Can not find a connection for Instance s-14814-VM. Assuming the default connection. 2025-09-01 15:42:44,607 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-4:[]) (logid:b0554662) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied. 2025-09-01 15:42:44,607 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-4:[]) (logid:b0554662) Shell script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.sh' is not available. Transformations will not be applied. 2025-09-01 15:42:44,607 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-4:[]) (logid:b0554662) Groovy scripting engine is not initialized. Data transformation skipped. 2025-09-01 15:43:11,433 WARN [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-2:[]) (logid:b0554662) Setting System VM's [VM {id: "14814", name: "s-14814-VM", uuid: "6f242aaa-bb01-408b-9991-5818c70ab7f7", type: "SecondaryStorageVm"}] current memory as max memory [524288]. 2025-09-01 15:43:11,435 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:b0554662) Trying to fetch storage pool 11838e6d-4476-32dd-9c19-f884dedd79b4 from libvirt 2025-09-01 15:43:11,473 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:b0554662) Trying to fetch storage pool 11838e6d-4476-32dd-9c19-f884dedd79b4 from libvirt 2025-09-01 15:43:11,528 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:b0554662) Trying to fetch storage pool 11838e6d-4476-32dd-9c19-f884dedd79b4 from libvirt 2025-09-01 15:43:11,538 WARN [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:b0554662) Caught: java.lang.NullPointerException: Cannot invoke "java.net.URI.getScheme()" because "uri" is null at com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:180) at com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:234) at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:258) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3816) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3395) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2240) at com.cloud.agent.Agent.processRequest(Agent.java:813) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)

versions

Cloudstack Management on 4.21.0.0
Ubuntu 24.04 on both Management and Host
Advanced networking selected without security groups
Secondary storage using NFS

The steps to reproduce the bug

1. Upgrade cloudstack-agent to 4.21.0.0 stop secondarystoragevm then start again
2. Also breaks on redeploy
   To fix the issue roll cloudstack-agent back to 4.20.0.1

What to do about it?

No response

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jonathaanbastin]

I believe this causes an issue with importing from external unmanaged VMware sources as well but still testing to confirm.

[amitmvyas]

Hi,

I have done the fresh installation of Management Server 4.21.0 version on Ubuntu 24.04, CS agent is also 4.21.0 on Ubuntu 24.04. SSVM doesn't comes up. If I downgrade the CS agent version to 4.20.0.1 then SSVM comes up, however GPU shows in Error state instead of free. Once I upgrade the CS agent on host to 4.21.0 then GPU shows as free. But CSProxy and SSVM is still showing 4.20.0.1 version.

[weizhouapache]

Hi,

I have done the fresh installation of Management Server 4.21.0 version on Ubuntu 24.04, CS agent is also 4.21.0 on Ubuntu 24.04. SSVM doesn't comes up. If I downgrade the CS agent version to 4.20.0.1 then SSVM comes up, however GPU shows in Error state instead of free. Once I upgrade the CS agent on host to 4.21.0 then GPU shows as free. But CSProxy and SSVM is still showing 4.20.0.1 version.

@amitmvyas
you need to stop/start or destroy or patch the system vms

[daviftorres]

I got this same error after upgrading 3 different non-prod environments. I even nuked a couple and installed 421 from scratch (no upgrade) and the issue remains.

2025-09-03 14:24:37,591 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Processing command: com.cloud.agent.api.StartCommand
2025-09-03 14:24:37,591 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Creating VM from specifications [VM {id: "3", name: "s-3-VM", uuid: "aef0e83b-f8a6-46b5-a77d-85e36c17f2f7", type: "SecondaryStorageVm"}]
2025-09-03 14:24:37,591 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Configuring VM with UUID [aef0e83b-f8a6-46b5-a77d-85e36c17f2f7].
2025-09-03 14:24:37,592 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Property [vm.memballoon.stats.period] was altered. Now using the value [5].
2025-09-03 14:24:37,592 WARN  [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Setting System VM's [VM {id: "3", name: "s-3-VM", uuid: "aef0e83b-f8a6-46b5-a77d-85e36c17f2f7", type: "SecondaryStorageVm"}] current memory as max memory [524288].
2025-09-03 14:24:37,592 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:9b1fb285) This host utilizes cgroupv2 (as the max shares value is [104000]), thus, the VM requested shares of [500] will be converted to consider the host limits; the new CPU shares value is [49].
2025-09-03 14:24:37,592 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Property [hypervisor.uri] has empty or null value. Using default value [null].
2025-09-03 14:24:37,592 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection at: qemu:///system
2025-09-03 14:24:37,593 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Trying to fetch storage pool 3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 from libvirt
2025-09-03 14:24:37,593 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection at: qemu:///system
2025-09-03 14:24:37,598 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully refreshed pool 3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 Capacity: (1.4292 TB) 1571370827776 Used: (431.42 GB) 463237808128 Available: (1.0078 TB) 1108133019648
2025-09-03 14:24:37,602 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection at: qemu:///system
2025-09-03 14:24:37,603 DEBUG [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Executing command [qemu-img info -U /mnt/3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85/84ee5be8-5b28-4f99-b453-afb3fc4f7ae3 ].
2025-09-03 14:24:37,611 DEBUG [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully executed process [13862] for command [qemu-img info -U /mnt/3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85/84ee5be8-5b28-4f99-b453-afb3fc4f7ae3 ].
2025-09-03 14:24:37,611 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Disk IO driver policy [null]. The host supports the io_uring policy [true]
2025-09-03 14:24:37,612 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Trying to fetch storage pool 3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 from libvirt
2025-09-03 14:24:37,612 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection at: qemu:///system
2025-09-03 14:24:37,616 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully refreshed pool 3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 Capacity: (1.4292 TB) 1571370827776 Used: (431.42 GB) 463237808128 Available: (1.0078 TB) 1108133019648
2025-09-03 14:24:37,616 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-5:[]) (logid:9b1fb285) nic=[Nic:Control-169.254.27.218-null]
2025-09-03 14:24:37,616 DEBUG [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Executing command [/bin/bash -c ip route | grep 169.254.0.0/16 ].
2025-09-03 14:24:37,623 DEBUG [utils.script.Script] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully executed process [13865] for command [/bin/bash -c ip route | grep 169.254.0.0/16 ].
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-5:[]) (logid:9b1fb285) nic=[Nic:Management-10.120.0.218-null]
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-5:[]) (logid:9b1fb285) nic=[Nic:Public-10.120.1.222-vlan://1201]
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-5:[]) (logid:9b1fb285) creating a vNet dev and bridge for public traffic per traffic label cloudbr1
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Executing command [/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 1201 -p eth3 -b breth3-1201 -o add ].
2025-09-03 14:24:37,642 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully executed process [13868] for command [/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 1201 -p eth3 -b breth3-1201 -o add ].
2025-09-03 14:24:37,642 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-5:[]) (logid:9b1fb285) nic=[Nic:Storage-172.25.157.232-null]
2025-09-03 14:24:37,642 DEBUG [kvm.storage.KVMStoragePoolManager] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Disconnecting disk 84ee5be8-5b28-4f99-b453-afb3fc4f7ae3
2025-09-03 14:24:37,642 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Trying to fetch storage pool 3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 from libvirt
2025-09-03 14:24:37,642 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection at: qemu:///system
2025-09-03 14:24:37,647 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully refreshed pool 3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 Capacity: (1.4292 TB) 1571370827776 Used: (431.42 GB) 463237808128 Available: (1.0078 TB) 1108133019648
2025-09-03 14:24:37,647 WARN  [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Caught: java.lang.NullPointerException: Cannot invoke "java.net.URI.getScheme()" because "uri" is null
        at com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:180)
        at com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:234)
        at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:258)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3816)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3395)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2240)
        at com.cloud.agent.Agent.processRequest(Agent.java:813)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-09-03 14:24:37,648 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:9b1fb285) Seq 1-2231252140385370214:  { Ans: , MgmtId: 90520744490122, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException: Cannot invoke "java.net.URI.getScheme()" because "uri" is null
        at com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:180)
        at com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:234)
        at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:258)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3816)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3395)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2240)
        at com.cloud.agent.Agent.processRequest(Agent.java:813)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-03 14:24:37,713 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Request:Seq 1-2231252140385370215:  { Cmd , MgmtId: 90520744490122, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.27.218","forceStop":"false","expungeVM":"false","volumesToDisconnect":[],"vmName":"s-3-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-03 14:24:37,713 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Processing command: com.cloud.agent.api.StopCommand
2025-09-03 14:24:37,713 DEBUG [resource.wrapper.LibvirtStopCommandWrapper] (AgentRequest-Handler-4:[]) (logid:9b1fb285) backing up the cmdline
2025-09-03 14:24:40,789 DEBUG [resource.wrapper.LibvirtStopCommandWrapper] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to backup cmdline file due to There was a problem while connecting to 169.254.27.218:3922
2025-09-03 14:24:40,790 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Property [hypervisor.uri] has empty or null value. Using default value [null].
2025-09-03 14:24:40,790 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Looking for libvirtd connection at: qemu:///system
2025-09-03 14:24:40,791 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Can not find KVM connection for Instance: s-3-VM, continuing.
2025-09-03 14:24:40,791 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Property [hypervisor.uri] has empty or null value. Using default value [null].
2025-09-03 14:24:40,791 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Looking for libvirtd connection at: lxc:///
2025-09-03 14:24:40,791 INFO  [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) No existing libvirtd connection found. Opening a new one
2025-09-03 14:24:40,791 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Property [libvirt.events.enabled] has empty or null value. Using default value [false].
2025-09-03 14:24:40,791 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Libvirt event listening is disabled, not setting up event loop
2025-09-03 14:24:40,792 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Can not find LXC connection for Instance: s-3-VM, continuing.
2025-09-03 14:24:40,792 WARN  [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Can not find a connection for Instance s-3-VM. Assuming the default connection.
2025-09-03 14:24:40,792 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Looking for libvirtd connection at: qemu:///system
2025-09-03 14:24:40,793 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 's-3-VM'
2025-09-03 14:24:40,794 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 's-3-VM'
2025-09-03 14:24:40,794 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 's-3-VM'
2025-09-03 14:24:40,794 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Executing command [/usr/share/cloudstack-common/scripts/vm/network/security_group.py destroy_network_rules_for_vm --vmname s-3-VM ].
2025-09-03 14:24:40,905 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Successfully executed process [13875] for command [/usr/share/cloudstack-common/scripts/vm/network/security_group.py destroy_network_rules_for_vm --vmname s-3-VM ].
2025-09-03 14:24:40,905 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) /usr/share/cloudstack-common/scripts/vm/network/security_group.py:828: SyntaxWarning: invalid escape sequence '\-'
  brName = execute("iptables-save |grep physdev-is-bridged |grep FORWARD |grep BF |grep '\-o' |awk '{print $4}' | head -1").strip()
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1273: SyntaxWarning: invalid escape sequence '\-'
  cmd = "iptables-save |grep physdev-is-bridged |grep FORWARD |grep BF |grep '\-o' | grep -w " + brname  + "|awk '{print $9}' | head -1"
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1331: SyntaxWarning: invalid escape sequence '\('
  refs = int(execute("""iptables -n -L %s | awk '/%s(.*)references/ {gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1332: SyntaxWarning: invalid escape sequence '\('
  refs_in = int(execute("""iptables -n -L %s-IN | awk '/%s-IN(.*)references/ {gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1333: SyntaxWarning: invalid escape sequence '\('
  refs_out = int(execute("""iptables -n -L %s-OUT | awk '/%s-OUT(.*)references/ {gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1334: SyntaxWarning: invalid escape sequence '\('
  refs6 = int(execute("""ip6tables -n -L %s | awk '/%s(.*)references/ {gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())

2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get vm :Domain not found: no domain with matching name 's-3-VM'
2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Try to stop the vm at first
2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-4:[]) (logid:9b1fb285) VM s-3-VM doesn't exist, no need to stop it
2025-09-03 14:24:40,906 WARN  [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2025-09-03 14:24:40,906 WARN  [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Shell script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.sh' is not available. Transformations will not be applied.
2025-09-03 14:24:40,906 WARN  [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Groovy scripting engine is not initialized. Data transformation skipped.
2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-4:[]) (logid:9b1fb285) No shell script provided for transformation. Data transformation skipped.
2025-09-03 14:24:40,906 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:9b1fb285) Seq 1-2231252140385370215:  { Ans: , MgmtId: 90520744490122, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }

[weizhouapache]

@jonathaanbastin @amitmvyas @daviftorres

just to confirm, you have a separated physical network for Storage traffic, right ?

[amitmvyas]

In my case, yes it's a different physical nic

[amitmvyas]

Hi,

I have done the fresh installation of Management Server 4.21.0 version on Ubuntu 24.04, CS agent is also 4.21.0 on Ubuntu 24.04. SSVM doesn't comes up. If I downgrade the CS agent version to 4.20.0.1 then SSVM comes up, however GPU shows in Error state instead of free. Once I upgrade the CS agent on host to 4.21.0 then GPU shows as free. But CSProxy and SSVM is still showing 4.20.0.1 version.

@amitmvyas
you need to stop/start or destroy or patch the system vms

I will give a try for this method but, I am pretty much sure that something is wrong with agent version 4.21.0 for SSVM which is causing the issue.