v8.5.3: log backup: update metafile format (#21412) #21577
Conversation
ti-chi-bot
added
area/br
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @ti-chi-bot, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request updates the documentation for TiDB's log backup feature, specifically detailing a significant change in the naming convention for backupmeta files. The new format incorporates multiple timestamp fields (flushTs, minDefaultTs, minTs, maxTs) and uses fixed-length hexadecimal encoding to ensure natural lexicographical sorting. This enhancement aims to improve the efficiency of file listing and range filtering operations for backup metadata.
Highlights
- Updated
backupmetafile naming convention: The format forbackupmetafiles has changed from{resolved_ts}-{uuid}.metato{flushTs}-{minDefaultTs}-{minTs}-{maxTs}.meta. - Detailed explanation of new timestamp fields: The documentation now clarifies the meaning of
flushTs,minDefaultTs,minTs, andmaxTswithin the new filename structure, providing more granular timestamp information. - Improved file sorting and filtering: The new naming convention uses fixed-length 16-digit hexadecimal strings for timestamps, ensuring lexicographical sorting for efficient batch listing and range filtering in external storage systems.
- Version applicability: This change in metadata file format applies to TiDB versions
v8.5.3andv9.0.0onwards.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request updates the documentation for the log backup metafile format. The changes are mostly accurate and improve the clarity of the documentation. However, a significant issue is that an older, conflicting description of the metafile format remains in another part of the document. This needs to be corrected to avoid user confusion. I have also provided a minor suggestion to improve stylistic consistency.
br/br-log-architecture.md
Outdated
| - `backupmeta` directory: stores backup metadata files. Starting from v8.5.3 and v9.0.0, the naming convention of these files changes from `{resolved_ts}-{uuid}.meta` to `{flushTs}-{minDefaultTs}-{minTs}-{maxTs}.meta`. The filename contains the following timestamp fields: | ||
| - `flushTs`: the timestamp when the backup file is periodically uploaded to the external storage. This value is obtained from PD and is globally unique. | ||
| - `minDefaultTs` (only applicable to Write CF files): the earliest transaction start time covered by this backup. | ||
| - `minTs` and `maxTs`: the minimum and maximum timestamps of all key-value data included in the backup file. | ||
|
|
||
| All these timestamps are encoded as fixed-length 16-digit hexadecimal strings, left-padded with zeros to ensure consistent length. This encoding design guarantees that filenames are naturally sorted in lexicographical order, making it efficient to perform batch listing and range filtering operations in external storage systems. |
There was a problem hiding this comment.
This new explanation of the backupmeta directory and the new .meta file format is very clear. However, there is a conflicting description of the .meta file at line 100, which still describes the old {resolved_ts}-{uuid}.meta format. To avoid confusing users, the content on lines 100-103 should be updated or removed as it seems redundant with this new, more detailed explanation section.
qiancai
changed the title
qiancai
added
the
needs-cherry-pick-master
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: qiancai The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
In response to a cherrypick label: new pull request created to branch |
Signed-off-by: ti-chi-bot <[email protected]>
…auto-embed * commit 'b570cc86270ba2e2b2b71d6eab8d717c2c8a1c56': (48 commits) *: remove TiDB roadmap from TOC, home, index, add warning (#21628) (#21637) Update the scope of system-variable ”tidb_redact_log“ (#21513) (#21634) *: add custom Japanese localization for cloud docs (#21612) Modify the introduction of `cop_pool_size` (#21462) (#21627) tiflash: Update tiflash upgrade guide (#21619) (#21625) Update tidb-performance-tuning-config.md (#21621) (#21624) Update tidb-cloud-poc.md (#21336) (#21411) cloud: update the pricing link for starter (#21620) ticdc: clarify descriptions of `index-value` and `columns` (#21616) (#21617) release-8.5: remove 'v' prefix from tidb-version variable (#21614) release-8.5: update version to 8.5.3 (#21613) cloud: fix note annotation syntax for proper rendering (#21611) ticdc: add key word for redo log (#21564) (#21607) ticdc: correct the tiup cluster command (#21135) (#21608) bump TiDB from v8.5.2 to v8.5.3 (#21578) v8.5.3: add release notes (#21399) (#21606) v8.5.3: log backup: update metafile format (#21412) (#21577) v8.5.3: update telemetry related doc files (#21404) (#21603) system-variables: add optimizer cost factors (#20748) (#21602) Add more description about SHARD_ROW_ID_BITS (#21467) (#21600) ...
3 participants
This is an automated cherry-pick of #21412
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions (in Chinese).
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?