Skip to content

Conversation

@zhangwl9
Copy link
Contributor

@zhangwl9 zhangwl9 commented Sep 26, 2025

Why are the changes needed?

Close #3781.

Brief change log

To optimize table cleanup operations (Dangling-Delete_file cleanup, data expiration, orphan_file cleanup,snapshotsFile cleanup), the last cleanup time is saved. This ensures that after an AMS service restart, if a cleanup operation was completed just before the restart, the next cleanup operation will only be performed after exceeding the respective execution interval.
The execution flow is as follows:

  • During the initialization of the ams service, retrieve the last execution time for each cleanup operation of every table to be optimized from the table_runtime_state table.

  • Check whether the time interval since the last cleanup exceeds the cleanup operation execution interval.

  • For optimization tables exceeding the execution interval, execute the cleanup operation.

  • Otherwise, schedule for the next round of execution.

  • Since each table's cleanup operation takes varying durations, after a table's cleanup completes, update the cleanup completion time in the table_runtime_state table.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added the module:ams-server Ams server module label Sep 26, 2025
@zhangwl9 zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch from bce7e0d to 8162476 Compare September 28, 2025 03:28
@codecov-commenter
Copy link

codecov-commenter commented Sep 28, 2025

Codecov Report

❌ Patch coverage is 60.78431% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.93%. Comparing base (fd44c44) to head (2aba806).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...amoro/server/scheduler/PeriodicTableScheduler.java 40.81% 28 Missing and 1 partial ⚠️
...apache/amoro/server/table/DefaultTableRuntime.java 88.46% 1 Missing and 2 partials ⚠️
...er/inline/DanglingDeleteFilesCleaningExecutor.java 0.00% 2 Missing ⚠️
.../server/scheduler/inline/DataExpiringExecutor.java 0.00% 2 Missing ⚠️
.../scheduler/inline/OrphanFilesCleaningExecutor.java 0.00% 2 Missing ⚠️
...er/scheduler/inline/SnapshotsExpiringExecutor.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3802      +/-   ##
============================================
- Coverage     29.45%   28.93%   -0.53%     
+ Complexity     4236     3870     -366     
============================================
  Files           670      631      -39     
  Lines         55245    50743    -4502     
  Branches       6983     6480     -503     
============================================
- Hits          16273    14682    -1591     
+ Misses        37778    35011    -2767     
+ Partials       1194     1050     -144     
Flag Coverage Δ
core 28.93% <60.78%> (+0.11%) ⬆️
trino ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@zhangwl9 zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch 4 times, most recently from 789c5bd to 7b1006d Compare November 5, 2025 02:59
@zhangwl9 zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch from 7b1006d to e69848b Compare November 11, 2025 10:46
@zhangwl9
Copy link
Contributor Author

zhangwl9 commented Nov 11, 2025

I have completed the logic modification to save cleanup times to the table_runtime_state table. Could you please review the code at your convenience? Thank you very much. @czy006 @baiyangtx

@zhangwl9 zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch from e69848b to 7b21fe0 Compare November 13, 2025 05:55
张文领 added 3 commits November 19, 2025 20:14
… each optimization table.

# Conflicts:
#	amoro-ams/src/main/resources/mysql/upgrade.sql
@klion26 klion26 force-pushed the AMORO-Save-lastTime-dev branch from a59efa7 to af05e8e Compare November 19, 2025 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Save the last completion time of the executor cleanup

2 participants