From c3ac49025f12a5be765d3d83c08ad1f0a2e3cf16 Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Wed, 23 Mar 2022 15:28:29 +0800 Subject: [PATCH 01/13] Update tidb-lightning-faq.md --- tidb-lightning/tidb-lightning-faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tidb-lightning/tidb-lightning-faq.md b/tidb-lightning/tidb-lightning-faq.md index f12af703887db..45ce601ad200a 100644 --- a/tidb-lightning/tidb-lightning-faq.md +++ b/tidb-lightning/tidb-lightning-faq.md @@ -119,7 +119,7 @@ If `tidb-lightning` abnormally exited, the cluster might be stuck in the "import {{< copyable "shell-regular" >}} ```sh -tidb-lightning-ctl --fetch-mode +tidb-lightning-ctl --config tidb-lightning.toml --fetch-mode ``` You can force the cluster back to "normal mode" using the following command: @@ -127,7 +127,7 @@ You can force the cluster back to "normal mode" using the following command: {{< copyable "shell-regular" >}} ```sh -tidb-lightning-ctl --switch-mode=normal +tidb-lightning-ctl --config tidb-lightning.toml --fetch-mode ``` ## Can TiDB Lightning be used with 1-Gigabit network card? From c4e83db28e689e722b75b2e0fa2a7c08b15ac702 Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Mon, 6 Jun 2022 10:41:48 +0800 Subject: [PATCH 02/13] Update alert-rules.md --- alert-rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/alert-rules.md b/alert-rules.md index c06f3330addbc..abe9503b24b90 100644 --- a/alert-rules.md +++ b/alert-rules.md @@ -419,7 +419,7 @@ This section gives the alert rules for the TiKV component. * Solution: - Adjust the `block-cache-size` value of both `rockdb.defaultcf` and `rocksdb.writecf`. + Adjust the `block-cache-size` value of both `rocksdb.defaultcf` and `rocksdb.writecf`. #### `TiKV_GC_can_not_work` From e11f2901d79cdf88b23d59bafd4de3b2c154587c Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Mon, 6 Jun 2022 10:43:10 +0800 Subject: [PATCH 03/13] Revert "Update alert-rules.md" This reverts commit c4e83db28e689e722b75b2e0fa2a7c08b15ac702. --- alert-rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/alert-rules.md b/alert-rules.md index abe9503b24b90..c06f3330addbc 100644 --- a/alert-rules.md +++ b/alert-rules.md @@ -419,7 +419,7 @@ This section gives the alert rules for the TiKV component. * Solution: - Adjust the `block-cache-size` value of both `rocksdb.defaultcf` and `rocksdb.writecf`. + Adjust the `block-cache-size` value of both `rockdb.defaultcf` and `rocksdb.writecf`. #### `TiKV_GC_can_not_work` From 0db7df99f2ab4562d4961b7fd2803491c9d5915d Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Tue, 1 Aug 2023 11:26:57 +0800 Subject: [PATCH 04/13] Update dumpling-overview.md --- dumpling-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dumpling-overview.md b/dumpling-overview.md index de8bf19a336d9..f4cf5c6ace2a0 100644 --- a/dumpling-overview.md +++ b/dumpling-overview.md @@ -108,7 +108,7 @@ In the command above: + The `-t` option specifies the number of threads for the export. Increasing the number of threads improves the concurrency of Dumpling and the export speed, and also increases the database's memory consumption. Therefore, it is not recommended to set the number too large. Usually, it's less than 64. -+ The `-r` option specifies the maximum number of rows in a single file. With this option specified, Dumpling enables the in-table concurrency to speed up the export and reduce the memory usage. When the upstream database is TiDB v3.0 or later versions, a `-r` value greater than 0 indicates that the TiDB region information is used for splitting and the specific `-r` value does not affect the split algorithm. When the upstream database is MySQL and the primary key is of the `int` type, specifying `-r` can also enable the in-table concurrency. ++ The `-r` option enables the in-table concurrency to speed up the export. When the source database is TiDB, a `-r` value greater than 0 indicates that the TiDB region information is used for splitting, and reduces the memory usage. The specific `-r` value does not affect the split algorithm. When the source database is MySQL and the primary key is of the `int` type, specifying `-r` can also enable the in-table concurrency. + The `-F` option is used to specify the maximum size of a single file (the unit here is `MiB`; inputs like `5GiB` or `8KB` are also acceptable). It is recommended to keep its value to 256 MiB or less if you plan to use TiDB Lightning to load this file into a TiDB instance. > **Note:** From ae9709d906e7ffdaec0e461810a5427061603e7a Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Tue, 12 Sep 2023 16:26:10 +0800 Subject: [PATCH 05/13] update go to v1.21 --- hardware-and-software-requirements.md | 2 +- pd-control.md | 2 +- pd-recover.md | 2 +- tidb-control.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/hardware-and-software-requirements.md b/hardware-and-software-requirements.md index 15d824567af30..bb8e7ae2ecfd5 100644 --- a/hardware-and-software-requirements.md +++ b/hardware-and-software-requirements.md @@ -47,7 +47,7 @@ As an open-source distributed SQL database with high performance, TiDB can be de | Libraries required for compiling and running TiDB | Version | | :--- | :--- | -| Golang | 1.20 or later | +| Golang | 1.21 or later | | Rust | nightly-2022-07-31 or later | | GCC | 7.x | | LLVM | 13.0 or later | diff --git a/pd-control.md b/pd-control.md index 20e391c4b9852..53eb4a9321d6d 100644 --- a/pd-control.md +++ b/pd-control.md @@ -33,7 +33,7 @@ To obtain `pd-ctl` of the latest version, download the TiDB server installation ### Compile from source code -1. [Go](https://golang.org/) 1.20 or later is required because the Go modules are used. +1. [Go](https://golang.org/) 1.21 or later is required because the Go modules are used. 2. In the root directory of the [PD project](https://github.com/pingcap/pd), use the `make` or `make pd-ctl` command to compile and generate `bin/pd-ctl`. ## Usage diff --git a/pd-recover.md b/pd-recover.md index 1910e6c46ba24..24b3bd3049d80 100644 --- a/pd-recover.md +++ b/pd-recover.md @@ -10,7 +10,7 @@ PD Recover is a disaster recovery tool of PD, used to recover the PD cluster whi ## Compile from source code -+ [Go](https://golang.org/) 1.20 or later is required because the Go modules are used. ++ [Go](https://golang.org/) 1.21 or later is required because the Go modules are used. + In the root directory of the [PD project](https://github.com/pingcap/pd), use the `make pd-recover` command to compile and generate `bin/pd-recover`. > **Note:** diff --git a/tidb-control.md b/tidb-control.md index 921a27a761ded..2c79e6ff347f4 100644 --- a/tidb-control.md +++ b/tidb-control.md @@ -26,7 +26,7 @@ After installing TiUP, you can use `tiup ctl:v tidb` command to ### Compile from source code -- Compilation environment requirement: [Go](https://golang.org/) 1.20 or later +- Compilation environment requirement: [Go](https://golang.org/) 1.21 or later - Compilation procedures: Go to the root directory of the [TiDB Control project](https://github.com/pingcap/tidb-ctl), use the `make` command to compile, and generate `tidb-ctl`. - Compilation documentation: you can find the help files in the `doc` directory; if the help files are lost or you want to update them, use the `make doc` command to generate the help files. From f0b8b492a6596caf9b78d23428211449fe42b92e Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Tue, 12 Sep 2023 16:27:52 +0800 Subject: [PATCH 06/13] Revert "update go to v1.21" This reverts commit ae9709d906e7ffdaec0e461810a5427061603e7a. --- hardware-and-software-requirements.md | 2 +- pd-control.md | 2 +- pd-recover.md | 2 +- tidb-control.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/hardware-and-software-requirements.md b/hardware-and-software-requirements.md index bb8e7ae2ecfd5..15d824567af30 100644 --- a/hardware-and-software-requirements.md +++ b/hardware-and-software-requirements.md @@ -47,7 +47,7 @@ As an open-source distributed SQL database with high performance, TiDB can be de | Libraries required for compiling and running TiDB | Version | | :--- | :--- | -| Golang | 1.21 or later | +| Golang | 1.20 or later | | Rust | nightly-2022-07-31 or later | | GCC | 7.x | | LLVM | 13.0 or later | diff --git a/pd-control.md b/pd-control.md index 53eb4a9321d6d..20e391c4b9852 100644 --- a/pd-control.md +++ b/pd-control.md @@ -33,7 +33,7 @@ To obtain `pd-ctl` of the latest version, download the TiDB server installation ### Compile from source code -1. [Go](https://golang.org/) 1.21 or later is required because the Go modules are used. +1. [Go](https://golang.org/) 1.20 or later is required because the Go modules are used. 2. In the root directory of the [PD project](https://github.com/pingcap/pd), use the `make` or `make pd-ctl` command to compile and generate `bin/pd-ctl`. ## Usage diff --git a/pd-recover.md b/pd-recover.md index 24b3bd3049d80..1910e6c46ba24 100644 --- a/pd-recover.md +++ b/pd-recover.md @@ -10,7 +10,7 @@ PD Recover is a disaster recovery tool of PD, used to recover the PD cluster whi ## Compile from source code -+ [Go](https://golang.org/) 1.21 or later is required because the Go modules are used. ++ [Go](https://golang.org/) 1.20 or later is required because the Go modules are used. + In the root directory of the [PD project](https://github.com/pingcap/pd), use the `make pd-recover` command to compile and generate `bin/pd-recover`. > **Note:** diff --git a/tidb-control.md b/tidb-control.md index 2c79e6ff347f4..921a27a761ded 100644 --- a/tidb-control.md +++ b/tidb-control.md @@ -26,7 +26,7 @@ After installing TiUP, you can use `tiup ctl:v tidb` command to ### Compile from source code -- Compilation environment requirement: [Go](https://golang.org/) 1.21 or later +- Compilation environment requirement: [Go](https://golang.org/) 1.20 or later - Compilation procedures: Go to the root directory of the [TiDB Control project](https://github.com/pingcap/tidb-ctl), use the `make` command to compile, and generate `tidb-ctl`. - Compilation documentation: you can find the help files in the `doc` directory; if the help files are lost or you want to update them, use the `make doc` command to generate the help files. From 9e88a6e364585f84a6176eb2dabb9c3a6e01b6f5 Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Fri, 19 Apr 2024 10:14:53 +0800 Subject: [PATCH 07/13] add encoding='utf-8 for windows --- scripts/release_notes_update_pr_author_info_add_dup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/release_notes_update_pr_author_info_add_dup.py b/scripts/release_notes_update_pr_author_info_add_dup.py index 92039bf89380d..22f7c0cf56a49 100644 --- a/scripts/release_notes_update_pr_author_info_add_dup.py +++ b/scripts/release_notes_update_pr_author_info_add_dup.py @@ -174,7 +174,7 @@ def create_release_file(version, dup_notes_levels, dup_notes): release_file = os.path.join(ext_path, f'release-{version}.md') shutil.copyfile(template_file, release_file) # Replace the file content - with open(release_file, 'r+') as file: + with open(release_file, 'r+', encoding='utf-8') as file: content = file.read() content = content.replace('x.y.z', version) version_parts = version.split('.') From e25b4afaf0a5c4f98a688d4aecd6409c53298579 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Mon, 23 Dec 2024 12:10:58 +0800 Subject: [PATCH 08/13] Merge remote-tracking branch 'upstream/master' From 0c77222e5336b8a430a08f74d17f5bfbd3d5d8fe Mon Sep 17 00:00:00 2001 From: houfaxin Date: Tue, 21 Jan 2025 18:09:45 +0800 Subject: [PATCH 09/13] Merge remote-tracking branch 'upstream/master' From e6edea0666f177a89b4a32842a16bbd2f4632d72 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Thu, 7 Aug 2025 11:46:59 +0800 Subject: [PATCH 10/13] Update tidb-performance-tuning-config.md --- tidb-performance-tuning-config.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tidb-performance-tuning-config.md b/tidb-performance-tuning-config.md index ed1c5fab3f63a..6bb43748c1ba8 100644 --- a/tidb-performance-tuning-config.md +++ b/tidb-performance-tuning-config.md @@ -272,8 +272,8 @@ The following table compares throughput (operations per second) between the base | Item | Baseline (OPS) | Optimized (OPS) | Improvement | | ---------| ---- | ----| ----| -| load data | 2858.5 | 5074.3 | +77.59% | -| workloada | 2243.0 | 12804.3 | +470.86% | +| Load data | 2858.5 | 5074.3 | +77.59% | +| Workload | 2243.0 | 12804.3 | +470.86% | #### Performance analysis @@ -461,7 +461,7 @@ You can control the execution mode of DML statements using the [`tidb_dml_type`] To use the bulk DML execution mode, set `tidb_dml_type` to `"bulk"`. This mode optimizes bulk data loading without conflicts and reduces memory usage during large write operations. Before using this mode, ensure that: -- Auto-commit is enabled. +- Enable [`autocommit`](/system-variables.md#autocommit). - The [`pessimistic-auto-commit`](/tidb-configuration-file.md#pessimistic-auto-commit-new-in-v600) configuration item is set to `false`. ```sql From a58522c88a4c67ba73a2b1f3a22b999308cc9488 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Thu, 7 Aug 2025 11:48:58 +0800 Subject: [PATCH 11/13] Update tidb-performance-tuning-config.md --- tidb-performance-tuning-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-performance-tuning-config.md b/tidb-performance-tuning-config.md index 6bb43748c1ba8..8964686ff2e59 100644 --- a/tidb-performance-tuning-config.md +++ b/tidb-performance-tuning-config.md @@ -125,7 +125,7 @@ soft-pending-compaction-bytes-limit = "192GiB" | [`concurrent-send-snap-limit`](/tikv-configuration-file.md#concurrent-send-snap-limit), [`concurrent-recv-snap-limit`](/tikv-configuration-file.md#concurrent-recv-snap-limit), and [`snap-io-max-bytes-per-sec`](/tikv-configuration-file.md#snap-io-max-bytes-per-sec) | Set limits for concurrent snapshot transfer and I/O bandwidth during TiKV scaling operations. Higher limits reduce scaling time by allowing faster data migration. | Adjusting these limits affects the trade-off between scaling speed and online transaction performance. | | [`in-memory-peer-size-limit`](/tikv-configuration-file.md#in-memory-peer-size-limit-new-in-v840) and [`in-memory-instance-size-limit`](/tikv-configuration-file.md#in-memory-instance-size-limit-new-in-v840) | Control the memory allocation for pessimistic lock caching at the Region and TiKV instance levels. Storing locks in memory reduces disk I/O and improves transaction performance. | Monitor memory usage carefully. Higher limits improve performance but increase memory consumption. | | [`rocksdb.max-manifest-file-size`](/tikv-configuration-file.md#max-manifest-file-size) | Set the maximum size of the RocksDB Manifest file, which logs the metadata about SST files and database state changes. Increasing this size reduces the frequency of Manifest file rewrites, thereby minimizing their impact on foreground write performance. | The default value is `128MiB`. In environments with a large number of SST files (for example, hundreds of thousands), frequent Manifest rewrites can degrade write performance. Adjusting this parameter to a higher value, such as `256MiB` or larger, can help maintain optimal performance. | -| [`rocksdb.titan`](/tikv-configuration-file.md#rocksdbtitan), [`rocksdb.defaultcf.titan`](/tikv-configuration-file.md#rocksdbdefaultcftitan), [`min-blob-size`](/tikv-configuration-file.md#min-blob-size), and [`blob-file-compression`](/tikv-configuration-file.md#blob-file-compression) | Enable the Titan storage engine to reduce write amplification and alleviate disk I/O bottlenecks. Particularly useful when RocksDB compaction cannot keep up with write workloads, resulting in accumulated pending compaction bytes. | Enable it when write amplification is the primary bottleneck. Trade-offs include: 1. Potential performance impact on primary key range scans. 2. Increased space amplification (up to 2x in the worst case). 3. Additional memory usage for blob cache. | +| [`rocksdb.titan`](/tikv-configuration-file.md#rocksdbtitan), [`rocksdb.defaultcf.titan`](/tikv-configuration-file.md#rocksdbdefaultcftitan), [`min-blob-size`](/tikv-configuration-file.md#min-blob-size), and [`blob-file-compression`](/tikv-configuration-file.md#blob-file-compression) | Enable the Titan storage engine to reduce write amplification and alleviate disk I/O bottlenecks. Particularly useful when RocksDB compaction cannot keep up with write workloads, resulting in accumulated pending compaction bytes. | Enable it when write amplification is the primary bottleneck. Trade-offs include:
  • 1. Potential performance impact on primary key range scans.
  • 2. Increased space amplification (up to 2x in the worst case).
  • 3. Additional memory usage for blob cache.
| | [`storage.scheduler-pending-write-threshold`](/tikv-configuration-file.md#scheduler-pending-write-threshold) | Set the maximum size of the write queue in the TiKV scheduler. When the total size of pending write tasks exceeds this threshold, TiKV returns a `Server Is Busy` error for new write requests. | The default value is `100MiB`. In scenarios with high write concurrency or temporary write spikes, increasing this threshold (for example, to `512MiB`) can help accommodate the load. However, if the write queue continues to accumulate and exceeds this threshold persistently, it might indicate underlying performance issues that require further investigation. | | [`storage.flow-control.l0-files-threshold`](/tikv-configuration-file.md#l0-files-threshold) | Control when write flow control is triggered based on the number of kvDB L0 files. Increasing the threshold reduces write stalls during high write workloads. | Higher thresholds might lead to more aggressive compactions when many L0 files exist. | | [`storage.flow-control.soft-pending-compaction-bytes-limit`](/tikv-configuration-file.md#soft-pending-compaction-bytes-limit) | Control the threshold for pending compaction bytes to manage write flow control. The soft limit triggers partial write rejections. | The default soft limit is `192GiB`. In write-intensive scenarios, if compaction processes cannot keep up, pending compaction bytes accumulate, potentially triggering flow control. Adjusting the limit can provide more buffer space, but persistent accumulation indicates underlying issues that require further investigation. | From dac2b6e8479e4a9ea81b8fce8d987175eec174ea Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 7 Aug 2025 11:59:21 +0800 Subject: [PATCH 12/13] Update tidb-performance-tuning-config.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- tidb-performance-tuning-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-performance-tuning-config.md b/tidb-performance-tuning-config.md index 8964686ff2e59..8fde36fd92c95 100644 --- a/tidb-performance-tuning-config.md +++ b/tidb-performance-tuning-config.md @@ -461,7 +461,7 @@ You can control the execution mode of DML statements using the [`tidb_dml_type`] To use the bulk DML execution mode, set `tidb_dml_type` to `"bulk"`. This mode optimizes bulk data loading without conflicts and reduces memory usage during large write operations. Before using this mode, ensure that: -- Enable [`autocommit`](/system-variables.md#autocommit). +- [`autocommit`](/system-variables.md#autocommit) is enabled. - The [`pessimistic-auto-commit`](/tidb-configuration-file.md#pessimistic-auto-commit-new-in-v600) configuration item is set to `false`. ```sql From 8151c8461034131959523b7caadb47e9ec28ad84 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 7 Aug 2025 14:43:07 +0800 Subject: [PATCH 13/13] Update tidb-performance-tuning-config.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- tidb-performance-tuning-config.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-performance-tuning-config.md b/tidb-performance-tuning-config.md index 8fde36fd92c95..e77dcf450a5c9 100644 --- a/tidb-performance-tuning-config.md +++ b/tidb-performance-tuning-config.md @@ -273,7 +273,7 @@ The following table compares throughput (operations per second) between the base | Item | Baseline (OPS) | Optimized (OPS) | Improvement | | ---------| ---- | ----| ----| | Load data | 2858.5 | 5074.3 | +77.59% | -| Workload | 2243.0 | 12804.3 | +470.86% | +| Workloada | 2243.0 | 12804.3 | +470.86% | #### Performance analysis