server: prevent duplicate HA works and alerts #10624

shwstppr · 2025-03-27T07:16:59Z

Description

Fixes #10367

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Fixes apache#10367 Signed-off-by: Abhishek Kumar <[email protected]>

codecov · 2025-03-27T07:25:15Z

Codecov Report

Attention: Patch coverage is 29.72973% with 26 lines in your changes missing coverage. Please review.

Project coverage is 15.17%. Comparing base (95c2481) to head (e7ba4de).
Report is 41 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...java/com/cloud/ha/HighAvailabilityManagerImpl.java	47.61%	7 Missing and 4 partials ⚠️
...java/com/cloud/ha/dao/HighAvailabilityDaoImpl.java	0.00%	11 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	20.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19   #10624      +/-   ##
============================================
+ Coverage     15.16%   15.17%   +0.01%     
- Complexity    11326    11343      +17     
============================================
  Files          5414     5412       -2     
  Lines        474804   475091     +287     
  Branches      57909    57973      +64     
============================================
+ Hits          72002    72096      +94     
- Misses       394749   394930     +181     
- Partials       8053     8065      +12

Flag	Coverage Δ
uitests	`4.29% <ø> (+<0.01%)`	⬆️
unittests	`15.89% <29.72%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

DaanHoogland · 2025-03-27T08:27:57Z

@blueorangutan package

blueorangutan · 2025-03-27T08:30:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

sureshanaparti

clgtm

shwstppr · 2025-03-27T08:56:24Z

@DaanHoogland @sureshanaparti just a heads up, I'm still investigating more on this. I'm looking into AgentManagerImpl to not call HA process using HighAvailabilityManagerImpl multiple times.

sureshanaparti · 2025-03-27T08:58:19Z

@DaanHoogland @sureshanaparti just a heads up, I'm still investigating more on this. I'm looking into AgentManagerImpl to not call HA process using HighAvailabilityManagerImpl multiple times.

@shwstppr will re-check once these changes are updated, thanks for letting us know.

blueorangutan · 2025-03-27T09:27:43Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12900

shwstppr · 2025-04-06T18:31:13Z

@blueorangutan package

blueorangutan · 2025-04-06T18:32:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-06T19:38:29Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12962

shwstppr · 2025-04-06T19:41:50Z

@blueorangutan test

blueorangutan · 2025-04-06T19:44:03Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-04-07T10:03:19Z

[SF] Trillian test result (tid-12884)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 48859 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10624-t12884-kvm-ol8.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

Pearl1594 · 2025-04-22T07:10:48Z

@shwstppr any update on this?

shwstppr · 2025-04-24T06:54:27Z

@Pearl1594 I'm looking into this. I'll update today or tomorrow.

server/src/main/java/com/cloud/ha/HighAvailabilityManagerImpl.java

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-05-02T11:38:25Z

@DaanHoogland @Pearl1594 @sureshanaparti this is ready for review now

@blueorangutan package

blueorangutan · 2025-05-02T11:40:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland

clgtm (only style remark)

DaanHoogland · 2025-05-02T11:46:30Z

server/src/main/java/com/cloud/ha/HighAvailabilityManagerImpl.java

        List<VMInstanceVO> reorderedVMList = new ArrayList<VMInstanceVO>();
-        if ((vms != null) && !vms.isEmpty()) {
+        int skippedHAVms = 0;
+        if (CollectionUtils.isNotEmpty(vms)) {


I think this block could go in its own method.

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2025-05-02T12:00:18Z

@blueorangutan package

blueorangutan · 2025-05-02T12:02:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-02T13:05:36Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13244

DaanHoogland · 2025-05-02T15:37:20Z

@blueorangutan test

blueorangutan · 2025-05-02T15:38:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-05-03T05:53:08Z

[SF] Trillian test result (tid-13196)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 48668 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10624-t13196-kvm-ol8.zip
Smoke tests completed. 132 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_secured_to_nonsecured_vm_migration	`Error`	404.78	test_vm_life_cycle.py

shwstppr · 2025-05-04T14:22:10Z

@blueorangutan test

blueorangutan · 2025-05-04T14:24:03Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-05-05T03:57:13Z

[SF] Trillian test result (tid-13199)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 46152 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10624-t13199-kvm-ol8.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

Pearl1594

code LGTM

DaanHoogland · 2025-05-06T08:41:30Z

tested with z(){ z|z& }; z& on the host running the VM. Between host down and vm starting only one alert per type was reported:

Signed-off-by: Abhishek Kumar <[email protected]>

server: prevent duplicate HA works and alerts

4e622f3

Fixes apache#10367 Signed-off-by: Abhishek Kumar <[email protected]>

sureshanaparti approved these changes Mar 27, 2025

View reviewed changes

sureshanaparti self-requested a review March 27, 2025 08:58

Pearl1594 added this to the 4.19.3 milestone Mar 28, 2025

Pearl1594 added this to ACS 4.20.1 Mar 28, 2025

Pearl1594 moved this to In Progress in ACS 4.20.1 Mar 28, 2025

DaanHoogland linked an issue Apr 8, 2025 that may be closed by this pull request

Multiple alerts for HA on VM and host #10367

Closed

rohityadavcloud requested review from DaanHoogland and Pearl1594 April 22, 2025 12:13

DaanHoogland reviewed Apr 24, 2025

View reviewed changes

server/src/main/java/com/cloud/ha/HighAvailabilityManagerImpl.java Outdated Show resolved Hide resolved

add checks, at agent side as well

44e1eb6

Signed-off-by: Abhishek Kumar <[email protected]>

boring-cyborg bot added component:agent component:orchestration labels May 2, 2025

DaanHoogland approved these changes May 2, 2025

View reviewed changes

shwstppr added 2 commits May 2, 2025 17:21

fix test

4542329

Signed-off-by: Abhishek Kumar <[email protected]>

refactor

e7ba4de

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr marked this pull request as ready for review May 5, 2025 06:12

DaanHoogland added status:needs-testing Severity:Critical Critical bug labels May 6, 2025

DaanHoogland self-assigned this May 6, 2025

Pearl1594 approved these changes May 6, 2025

View reviewed changes

DaanHoogland removed the status:needs-testing label May 6, 2025

DaanHoogland merged commit 919c979 into apache:4.19 May 6, 2025
25 of 26 checks passed

github-project-automation bot moved this from In Progress to Done in ACS 4.20.1 May 6, 2025

DaanHoogland deleted the fix-multiplehaworksalerts branch May 6, 2025 08:42

DaanHoogland mentioned this pull request May 6, 2025

Multiple alerts for HA on VM and host #10367

Closed

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jun 19, 2025

server: prevent duplicate HA works and alerts (apache#10624)

0d9bc77

Signed-off-by: Abhishek Kumar <[email protected]>

server: prevent duplicate HA works and alerts #10624

server: prevent duplicate HA works and alerts #10624

Uh oh!

Conversation

shwstppr commented Mar 27, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland commented Mar 27, 2025

Uh oh!

blueorangutan commented Mar 27, 2025

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Mar 27, 2025

Uh oh!

sureshanaparti commented Mar 27, 2025

Uh oh!

blueorangutan commented Mar 27, 2025

Uh oh!

shwstppr commented Apr 6, 2025

Uh oh!

blueorangutan commented Apr 6, 2025

Uh oh!

blueorangutan commented Apr 6, 2025

Uh oh!

shwstppr commented Apr 6, 2025

Uh oh!

blueorangutan commented Apr 6, 2025

Uh oh!

blueorangutan commented Apr 7, 2025

Uh oh!

Pearl1594 commented Apr 22, 2025

Uh oh!

shwstppr commented Apr 24, 2025

Uh oh!

Uh oh!

shwstppr commented May 2, 2025

Uh oh!

blueorangutan commented May 2, 2025

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland May 2, 2025

Choose a reason for hiding this comment

Uh oh!

shwstppr commented May 2, 2025

Uh oh!

blueorangutan commented May 2, 2025

Uh oh!

blueorangutan commented May 2, 2025

Uh oh!

DaanHoogland commented May 2, 2025

Uh oh!

blueorangutan commented May 2, 2025

Uh oh!

blueorangutan commented May 3, 2025

Uh oh!

shwstppr commented May 4, 2025

Uh oh!

blueorangutan commented May 4, 2025

Uh oh!

blueorangutan commented May 5, 2025

Uh oh!

Pearl1594 left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented May 6, 2025

codecov bot commented Mar 27, 2025 •

edited

Loading