SSVM unable to create after upgrade from 4.20.1 to 4.21 #11611
Description
problem
After upgrading cloudstack from 4.20.1 to 4.21, SSVM is unable to be created after destroying. Live patching works fine and I can see that it has indeed been upgraded, but once you destroy the SSVM, it fails to create a new one. No issues with console proxy.
versions
cloudstack management 4.21 on ubuntu 22.04.
mysql database on 22.04
NFS secondary storage on 22.04.
cloudstack host 4.21 on ubuntu 24.04.
The steps to reproduce the bug
- upgrade both management server and host to cloudstack 4.21
- Log into UI
- Live patch SSVM, console proxy, VR - successful
- Destroy SSVM, fails to create new SSVM. Therefore left left without the SSVM for secondary storage.
...
What to do about it?
Log when creating SSVM:
2025-09-11 00:57:05,340 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-39:[ctx-232c698d, job-264/job-306, ctx-b6c34cdb]) (logid:2f566107) Trying to deploy VM [error decoding VM instance {"id":15,"instanceName":"s-15-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"c4ab0591-2337-45e9-a584-7e38263e3ebc"}] and details: Plan [error decoding {"_clusterId":1,"_dcId":1,"_physicalNetworkId":null,"_podId":1,"_poolId":1,"migrationPlan":false}]; avoid list [{}] and planner: [null].
2025-09-11 00:57:05,657 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-39:[ctx-232c698d, job-264/job-306, ctx-b6c34cdb]) (logid:2f566107) STATS: Failed to alloc resource from host: Host {"id":1,"name":"lab-host-01","type":"Routing","uuid":"3c2303ac-75cb-41ed-bcec-70851052f9e2"} reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (512.00 MB) 536870912
2025-09-11 00:57:09,135 INFO [c.c.a.ApiServer] (qtp1438988851-26:[ctx-a9b9a5c3, ctx-d79cbae3]) (logid:79677007) Unable to execute API command readyforshutdown due to missing parameter managementserverid
2025-09-11 00:57:11,200 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:[]) (logid:) Seq 1-2096707101517676724: Processing: { Ans: , MgmtId: 206863106017244, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException: Cannot invoke "java.net.URI.getScheme()" because "uri" is null
","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-11 00:57:14,148 INFO [c.c.a.ApiServer] (qtp1438988851-19:[ctx-1ad15139, ctx-f28d0b13]) (logid:f9a5adc5) Unable to execute API command readyforshutdown due to missing parameter managementserverid
2025-09-11 00:57:15,029 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-39:[ctx-232c698d, job-264/job-306, ctx-b6c34cdb]) (logid:2f566107) VM instance {"id":15,"instanceName":"s-15-VM","state":"Stopped","type":"SecondaryStorageVm","uuid":"c4ab0591-2337-45e9-a584-7e38263e3ebc"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: Host {"id":1,"name":"lab-host-01","type":"Routing","uuid":"3c2303ac-75cb-41ed-bcec-70851052f9e2"}
2025-09-11 00:57:15,083 ERROR [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-39:[ctx-232c698d, job-264/job-306, ctx-b6c34cdb]) (logid:2f566107) Unable to orchestrate start VM instance {"id":15,"instanceName":"s-15-VM","state":"Stopped","type":"SecondaryStorageVm","uuid":"c4ab0591-2337-45e9-a584-7e38263e3ebc"} due to [Unable to get answer that is of class com.cloud.agent.api.StartAnswer].
2025-09-11 00:57:15,087 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-39:[ctx-232c698d, job-264/job-306, ctx-b6c34cdb]) (logid:2f566107) Complete async job-306, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AHhVbmFibGUgdG8gb3JjaGVzdHJhdGUgdGhlIHN0YXJ0IG9mIFZNIGluc3RhbmNlIHsiaW5zdGFuY2VOYW1lIjoicy0xNS1WTSIsInV1aWQiOiJjNGFiMDU5MS0yMzM3LTQ1ZTktYTU4NC03ZTM4MjYzZTNlYmMifS51cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABNzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAAhCAAZmb3JtYXRJAApsaW5lTnVtYmVyTAAPY2xhc3NMb2FkZXJOYW1lcQB-AAVMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABUwACm1vZHVsZU5hbWVxAH4ABUwADW1vZHVsZVZlcnNpb25xAH4ABXhwAQAAFsZ0AANhcHB0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAEG9yY2hlc3RyYXRlU3RhcnRwcHNxAH4ADAD_____cHQAL2pkay5pbnRlcm5hbC5yZWZsZWN0LkdlbmVyYXRlZE1ldGhvZEFjY2Vzc29yMzA5cHQABmludm9rZXBwc3EAfgAMAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAUdAAJamF2YS5iYXNldAAHMTcuMC4xNnNxAH4ADAIAAAI5cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ABRxAH4AGHEAfgAZc3EAfgAMAQAAAGZxAH4ADnQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgAMAQAAFz9xAH4ADnEAfgAPcQB-ABBxAH4AIHBwc3EAfgAMAQAAAGNxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACsXEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMXEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA4cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABncQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADB0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA1cQB-AA5xAH4AM3EAfgAwdAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAucQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ALHEAfgAtcHBzcQB-AAwBAAACfXEAfgAOcQB-ACdxAH4AKHEAfgAtcHBzcQB-AAwCAAACG3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AMXEAfgAYcQB-ABlzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAABHBwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AGHEAfgAZc3EAfgAMAgAAAntwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBCcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAAA0hwdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AC1xAH4AGHEAfgAZc3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2025-09-11 00:57:15,172 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:[ctx-c3b65449]) (logid:da5d2ff6) Secondary Storage Vm creation failure in zone [lab-reno].
2025-09-11 00:57:15,180 WARN [c.c.a.AlertManagerImpl] (secstorage-1:[ctx-c3b65449]) (logid:da5d2ff6) alertType=[19] dataCenterId=[1] podId=[null] clusterId=[null] message=[Secondary Storage Vm creation failure in zone [lab-reno]. Error details: Unable to allocate capacity on zone [1] due to [null].].
2025-09-11 00:57:15,194 WARN [c.c.a.AlertManagerImpl] (secstorage-1:[ctx-c3b65449]) (logid:da5d2ff6) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Secondary Storage Vm creation failure in zone [lab-reno]. Error details: Unable to allocate capacity on zone [1] due to [null].] and content [Secondary Storage Vm creation failure (zone lab-reno)].
2025-09-11 00:57:15,196 WARN [c.c.v.SystemVmLoadScanner$1] (secstorage-1:[ctx-c3b65449]) (logid:da5d2ff6) Unexpected exception Unable to orchestrate the start of VM instance {"instanceName":"s-15-VM","uuid":"c4ab0591-2337-45e9-a584-7e38263e3ebc"}. com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate the start of VM instance {"instanceName":"s-15-VM","uuid":"c4ab0591-2337-45e9-a584-7e38263e3ebc"}.
2025-09-11 00:57:19,127 INFO [c.c.a.ApiServer] (qtp1438988851-26:[ctx-e465548e, ctx-15495d4f]) (logid:bfb8f311) Unable to execute API command readyforshutdown due to missing parameter managementserverid
2025-09-11 00:57:24,131 INFO [c.c.a.ApiServer] (qtp1438988851-19:[ctx-75111ec6, ctx-5fc78d99]) (logid:8c7dca11) Unable to execute API command readyforshutdown due to missing parameter managementserverid
2025-09-11 00:57:29,131 INFO [c.c.a.ApiServer] (qtp1438988851-26:[ctx-c78d0862, ctx-f92198e3]) (logid:2c543679) Unable to execute API command readyforshutdown due to missing parameter managementserverid