RFC: reset TiKV to some certain mvcc snapshot #81

longfangsong · 2021-11-26T01:50:35Z

Signed-off-by: longfangsong [email protected]

Add flashback Signed-off-by: longfangsong <[email protected]>

BusyJay · 2021-11-26T04:31:30Z

You mean reset data to certain mvcc snapshot, right?

text/0081-flashback.md

nolouch · 2021-11-26T06:20:17Z

text/0081-flashback.md

+
+## Drawbacks
+
+This is a **very** unsafe operation, we should prevent the user call this on normal cluster.


I think we need some steps to ensure that this operation can be used in the right scene, and to inform the risk of the operation.

nolouch · 2021-11-26T06:20:59Z

text/0081-flashback.md

+
+### 1. Wait all committed raft log are applied
+
+We can just polling each region's `RegionInfo` in a method similar with [`Debugger::region_info`](https://github.com/tikv/tikv/blob/789c99666f2f9faaa6c6e5b021ac0cf7a76ae24e/src/server/debug.rs#L188) does, and wait for `commit_index` to be equal to `apply_index`.


may need some authentications for this debug service.

ref tikv/rfcs#81

Signed-off-by: longfangsong <[email protected]>

ref tikv/rfcs#81 Signed-off-by: longfangsong <[email protected]>

ref tikv/rfcs#81

text/0081-truncate.md

youjiali1995 · 2021-12-06T03:20:42Z

text/0081-truncate.md

+
+Then the sync state of the cluster (from the DR DC's PD) is not dependable. In this case, data can be partially sent to the TiKV nodes in DR, and break the ACID consistency.
+
+Though it is impossible to do lossless recovery in this situation, we can recover the data on TiKV nodes to a certain version (`resolved_ts` in this case), which is ACID consistency.


Whose resolved_ts? DR DC's or primary DC's?

text/0081-truncate.md

Co-authored-by: Lei Zhao <[email protected]> Update text/0081-truncate.md Co-authored-by: Lei Zhao <[email protected]> fix indent Signed-off-by: longfangsong <[email protected]>

…ing the truncate process Signed-off-by: longfangsong <[email protected]>

text/0081-truncate.md

Signed-off-by: longfangsong <[email protected]>

ref tikv/rfcs#81 Signed-off-by: longfangsong <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

* raftstore: support dynamically scaling pool size (#11159) (#11213) close #11159 Signed-off-by: Wenbo Zhang <[email protected]> * storage: reduce extra next/pre in row_scan (#11625) * reduce extra next/pre in row_scan close https://github.com/tikv/tikv/issues/11572 Signed-off-by: Jayice <[email protected]> * reduce extra next/pre in row_scan close #11572 Signed-off-by: Jayice <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: remove some unnecessary dependencies (#11592) close #11593 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Yilin Chen <[email protected]> * raftstore: make threads of snap-generator pool configurable (#11274) * add snap-generator-pool-size for configurable snapshot generator thread pool Signed-off-by: iosmanthus <[email protected]> * rename config macro Signed-off-by: iosmanthus <[email protected]> Co-authored-by: Andy Lok <[email protected]> * ctl:add prompt for ctl when required param is not specified (#11639) * add prompt for ctl close #11514 Signed-off-by: Jayice <[email protected]> * format code close #11514 Signed-off-by: Jayice <[email protected]> * format code close #11514 Signed-off-by: Jayice <[email protected]> * format code close #11514 Signed-off-by: Jayice <[email protected]> * optimize duplicate codes:pd check close #11514 Signed-off-by: Jayice <[email protected]> * move pd check in get_pd_rpc_client close #11514 Signed-off-by: Jayice <[email protected]> * simplify the codes check host|data-dir close #11514 Signed-off-by: Jayice <[email protected]> * ctl: validate column family name when scanning data using ctl (#11269) * ctl: should check column name validation when scan data using ctl close #11268 Signed-off-by: yuqi1129 <[email protected]> * fix format problem Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> * txn: Do assertions on data for mutations to reduce potential data inconsistency (#10977) * Add mutation checking logic Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * Fix usages of mutation Signed-off-by: MyonKeminta <[email protected]> * Update kvproto and renaming Signed-off-by: MyonKeminta <[email protected]> * Add a simple test Signed-off-by: MyonKeminta <[email protected]> * Fix incorrect assertion Signed-off-by: MyonKeminta <[email protected]> * Support check existance during pessimistic lock Signed-off-by: MyonKeminta <[email protected]> * Add assertion level support Signed-off-by: MyonKeminta <[email protected]> * Add tests Signed-off-by: MyonKeminta <[email protected]> * Add more test cases Signed-off-by: MyonKeminta <[email protected]> * Add test for check_existence of pessimistic_lock request Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * ref #11542 Signed-off-by: MyonKeminta <[email protected]> * Add a metric for counting different way of checking assertion, for reference about the performance impact Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * metrics: fix memory metrics (#11656) Signed-off-by: Shuaipeng Yu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * coprocesser: refactor encoding and add gbk_chinese_ci map (#11661) * close #11660 Signed-off-by: xiongjiwei <[email protected]> * close #11660 Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216) * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * fix enable_ttl Signed-off-by: andylokandy <[email protected]> * simplify test Signed-off-by: andylokandy <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> Co-authored-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cdc: load uninlined value more effectively (#11615) Signed-off-by: qupeng <[email protected]> * cdc: load old value with prefix seek if possible (#11643) * close #11642 Signed-off-by: qupeng <[email protected]> * a more comment Signed-off-by: qupeng <[email protected]> * ctl: split into multiple modules (#11658) * ctl: split into multiple modules ref #10938 Signed-off-by: andylokandy <[email protected]> * refactor if let to match Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * Fix build issues on non-linux platforms (#11671) * close #11676 Signed-off-by: devillove084 <[email protected]> * all: Migrate from std:: types to primitives. (#11677) * Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref #11678 ```release-note NONE ``` Signed-off-by: Harold Dost <[email protected]> * metrics alert: fix apply cpu alert (#11672) Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * resrc_mtr: introduce scanned keys statistics (#11085) close #11690 Signed-off-by: mornyx <[email protected]> * file_system: support collect the io flow by proc (#11667) * file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * fix clippy Signed-off-by: MuZhou233 <[email protected]> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <[email protected]> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * Integrate thread_io Signed-off-by: MuZhou233 <[email protected]> * record bytes in request Signed-off-by: MuZhou233 <[email protected]> * fix logical bug Signed-off-by: MuZhou233 <[email protected]> * Update test write_bytes Signed-off-by: MuZhou233 <[email protected]> * Delete unused & Rename func Signed-off-by: MuZhou233 <[email protected]> * Add test for read & Update test for write Signed-off-by: MuZhou233 <[email protected]> * Add buffered thread io Signed-off-by: MuZhou233 <[email protected]> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <[email protected]> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <[email protected]> * Change fetch logic. close #10867 Signed-off-by: MuZhou233 <[email protected]> * Add AtomicIOBytes Signed-off-by: MuZhou233 <[email protected]> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <[email protected]> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <[email protected]> * Adjust code style & fix Signed-off-by: MuZhou233 <[email protected]> * improve write_bytes test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check epoch and validness of lock table when writing pessimistic locks (#11591) ref #11452 Signed-off-by: Yilin Chen <[email protected]> * rsmeter: introduce datasink (#11688) * add datasink trait Signed-off-by: Zhenchi <[email protected]> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build ref #11691 Signed-off-by: Zhenchi <[email protected]> * datasink -> data_sink Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * clippy: Fixing up clippy errors. (#11193) * clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref #4301 Signed-off-by: Harold Dost <[email protected]> * Test Revert of Iterations ref #4301 Signed-off-by: Harold Dost <[email protected]> * clippy: Enable check for rust-2018-idioms ref #4301 Signed-off-by: Harold Dost <[email protected]> * rsmeter: add centralized place to construct protobuf data (#11696) ref #11691 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * tikv_util: update procfs to 0.12.0 (#11703) Signed-off-by: linning <[email protected]> * Use generic for api version to reduce runtime branching (#11687) * Use generic for api version to reduce runtime branching ref #10938 Signed-off-by: andylokandy <[email protected]> * fix test Signed-off-by: andylokandy <[email protected]> * add test Signed-off-by: andylokandy <[email protected]> * fix clippy Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * fmt Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * rename vars Signed-off-by: andylokandy <[email protected]> * improve vars Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * sst_importer: check api version when importing (#11664) * sst_importer: check api version when importing ref #10938 Signed-off-by: Peng Guanwen <[email protected]> * Fix bugs Signed-off-by: Peng Guanwen <[email protected]> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <[email protected]> * remove redundant log Signed-off-by: Peng Guanwen <[email protected]> * Show the entire key Signed-off-by: Peng Guanwen <[email protected]> * Fix lint issue Signed-off-by: Peng Guanwen <[email protected]> * resolve conflict Signed-off-by: andylokandy <[email protected]> Co-authored-by: Andy Lok <[email protected]> * backup: pipeline scan and save step (#11528) * br: sperate io and scan threads Signed-off-by: Yu Juncen <[email protected]> * br/stream: make clippy happy Signed-off-by: yujuncen <[email protected]> * br/stream: fix tests Signed-off-by: Yu Juncen <[email protected]> * backup: fix a dummy bug Signed-off-by: Yu Juncen <[email protected]> * backup: add config of io-threads Signed-off-by: yujuncen <[email protected]> * backup: ref #11350: some minior change Signed-off-by: yujuncen <[email protected]> * backup: added some metrics Signed-off-by: yujuncen <[email protected]> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <[email protected]> * backup: added some slow log and metrics Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: fix build Signed-off-by: yujuncen <[email protected]> * Revert "backup: fix build" This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e. Signed-off-by: yujuncen <[email protected]> * Revert "backup: address comments" This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa. Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: always set IO type to Export Signed-off-by: yujuncen <[email protected]> * backup:make clippy happy Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * metrics: fix the metrics cannot be displayed (#11710) ref #11662 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * backup: return api-version to BR when backup (#11704) * Update kvproto Signed-off-by: Peng Guanwen <[email protected]> * Return api-version for br ref #10938 Signed-off-by: Peng Guanwen <[email protected]> * Reformat code Signed-off-by: Peng Guanwen <[email protected]> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * format code Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * file_system: limit thread_io target os (#11715) * limit thread_io target os Signed-off-by: MuZhou233 <[email protected]> * limit thread_io target os. typo. close #11698 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Yilin Chen <[email protected]> * file_system: bypass file allocate implementation in fs2 (#11700) Close #10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <[email protected]> * metrics: remove min legend of some Grafana panels (#11487) Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref #11119 Signed-off-by: tabokie <[email protected]> * coprocessor: not return rows when there is no input for simple aggregation (#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <[email protected]> * close #11735, and address comments Signed-off-by: xufei <[email protected]> * backup: reduce the default thread pool size of backup, enable auto tune by default (#11699) * backup: clamp auto tune values Signed-off-by: yujuncen <[email protected]> * ref #11000: change the default backup threads. Signed-off-by: yujuncen <[email protected]> * backup: run tasks in background Signed-off-by: yujuncen <[email protected]> * backup: run rustfmt Signed-off-by: yujuncen <[email protected]> * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen <[email protected]> * br-stream: fix build Signed-off-by: yujuncen <[email protected]> * Backup: add S3 metrics && add s3_multi_part_size config (#11666) * s3: add request metrics for s3 storage Signed-off-by: 3pointer <[email protected]> * s3: add grafana json Signed-off-by: 3pointer <[email protected]> * br: add config for s3 multi part upload Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * hidden the new config Signed-off-by: 3pointer <[email protected]> * only hidden config in config-template.toml Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * close #11727 && format Signed-off-by: 3pointer <[email protected]> * fix Signed-off-by: 3pointer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: support multiple datasinks (#11707) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <[email protected]> * wrap config notifier Signed-off-by: Zhenchi <[email protected]> * add unit tests Signed-off-by: Zhenchi <[email protected]> * polish ref #11691 Signed-off-by: Zhenchi <[email protected]> * address comments Signed-off-by: Zhenchi <[email protected]> * fix test Signed-off-by: Zhenchi <[email protected]> * fix outdated comments Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * engine: properly estimate store size (#11728) * properly estimate store size, ref #11119 Signed-off-by: tabokie <[email protected]> * address comments Signed-off-by: tabokie <[email protected]> * implement global config client for tikv (#11685) * implement global config client close #11686 Signed-off-by: lemonhx <[email protected]> * implement unit test for global config in pd client Signed-off-by: lemonhx <[email protected]> * resolved formatting issues Signed-off-by: lemonhx <[email protected]> * impl pd client asyncly Signed-off-by: lemonhx <[email protected]> * according to reviewer's opinion removing store method implementation Signed-off-by: lemonhx <[email protected]> * Merge branch 'master' of https://github.com/tikv/tikv into global_conf Signed-off-by: lemonhx <[email protected]> * write test case for watch global config when grpc server is closed. Signed-off-by: lemonhx <[email protected]> * resove issues Signed-off-by: lemonhx <[email protected]> * clippy Signed-off-by: lemonhx <[email protected]> * watch global config rentry future in test deadlock just removed Signed-off-by: lemonhx <[email protected]> Co-authored-by: Zhenchi <[email protected]> * storage: skip scanning lock when using rc (#11701) * skip scanning lock when using rc, ref #11485 Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> * add test Signed-off-by: you06 <[email protected]> * address comment Signed-off-by: you06 <[email protected]> * add an integration test Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: pushdown substring to tikv (#11494) * corp: pushdown substring to tikv Signed-off-by: guo-shaoge <[email protected]> * fix case. close #11495 Signed-off-by: guo-shaoge <[email protected]> * fix comment Signed-off-by: guo-shaoge <[email protected]> * rsmeter: distinguish between collectors and observers within recorder (#11712) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <[email protected]> * wrap config notifier Signed-off-by: Zhenchi <[email protected]> * add unit tests Signed-off-by: Zhenchi <[email protected]> * polish ref #11691 Signed-off-by: Zhenchi <[email protected]> * rsmeter: distinguish between collectors and observers within recorder Signed-off-by: Zhenchi <[email protected]> * fmt & try to resolve #11689 Signed-off-by: Zhenchi <[email protected]> * add unit tests ref #11691 Signed-off-by: Zhenchi <[email protected]> * retrigger test Signed-off-by: Zhenchi <[email protected]> * address comments Signed-off-by: Zhenchi <[email protected]> * fix test Signed-off-by: Zhenchi <[email protected]> * fix outdated comments Signed-off-by: Zhenchi <[email protected]> * Update components/resource_metering/src/recorder/sub_recorder/mod.rs Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> * address comment Signed-off-by: Zhenchi <[email protected]> * remove enabled Signed-off-by: Zhenchi <[email protected]> * fix Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Wenxuan <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> * config: validate online configurable thread pools (tikv#11159) (#11714) ref #11159 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * res_meter: upgrade kvproto (#11749) * Upgrade kvproto close #11748 Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Use master kvproto Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: add pubsub datasink (#11719) * rsmeter: add pubsub datasink close #11691 Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> * Update components/resource_metering/src/reporter/pubsub.rs Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * status_server: Support online config update for configuration hosting platform (#11693) close #11692 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * res_meter: optimize top k (#11753) * Keep top k on recorder Signed-off-by: mornyx <[email protected]> * Modify comments Signed-off-by: mornyx <[email protected]> * Fix pd.rs Signed-off-by: mornyx <[email protected]> * Refactor Signed-off-by: mornyx <[email protected]> * Remove import Signed-off-by: mornyx <[email protected]> * max_resource_groups 200 -> 100 Signed-off-by: mornyx <[email protected]> * Fix ut Signed-off-by: mornyx <[email protected]> * Use iterator to avoid clone Signed-off-by: mornyx <[email protected]> * Optimize close #11754 Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftclient: delay flush (#11705) * raftclient: delay flush Signed-off-by: Jay Lee <[email protected]> * remove raftstore wait Signed-off-by: Jay Lee <[email protected]> * Ref #11309. tikv#11310 introduces delay in raftstore thread to reduce the flush rate. It can only reduce flush rate from one thread. If there are many raftstore threads or IO threads, messages can still be flush frequently. This PR reduces the flush rate by introducing delay in raft client. So delay will work at connection level and achieve the maximun batch affect. Signed-off-by: Jay Lee <[email protected]> * remove useless field Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * fix unstable raw get case Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Liqi Geng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cloud: support azure blob storage as external storage backend (#11732) * support azure storage blob for external storage Signed-off-by: Leavrth <[email protected]> * modify the priority of the azure login methods Signed-off-by: Leavrth <[email protected]> * fix feature match branch Signed-off-by: Leavrth <[email protected]> * implement token update retriable Signed-off-by: Leavrth <[email protected]> * fix some problems Signed-off-by: Leavrth <[email protected]> * close #11731 Signed-off-by: Leavrth <[email protected]> * rustfmt Signed-off-by: Leavrth <[email protected]> * make clippy success Signed-off-by: Leavrth <[email protected]> * simplify uploader Signed-off-by: Leavrth <[email protected]> * commit some suggestions Signed-off-by: Leavrth <[email protected]> * coprocesser: update encoding lib (#11506) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * coprocesser: implement upper and lower function on GBK charset (#11756) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: support archiving and rotation (#11651) (#11657) * logger: support archiving and rotation (#11651) close #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: add testcases to test dt_from_file_name (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: simplify DateTime format (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * diagnostics: refactor (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: run clippy for integration tests (#11768) * run clippy --fix with nightly-2021-12-17 Signed-off-by: tabokie <[email protected]> * manual clippy Signed-off-by: tabokie <[email protected]> * format and clippy Signed-off-by: tabokie <[email protected]> * enable default features when running clippy, ref #11119 Signed-off-by: tabokie <[email protected]> * fix build Signed-off-by: tabokie <[email protected]> * revert box_collection lint Signed-off-by: tabokie <[email protected]> * Update tests/integrations/raftstore/test_bootstrap.rs Signed-off-by: Xinye Tao <[email protected]> Co-authored-by: Lei Zhao <[email protected]> Co-authored-by: Lei Zhao <[email protected]> * logger: fix config-template's log unit (#11651) (#11777) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: improve testcase (#11651) (#11778) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * raftstore: propose in-memory pessimistic locks before leader transfer (#11577) * *: check epoch and validness of lock table when writing pessimistic locks ref #11452 Signed-off-by: Yilin Chen <[email protected]> * raftstore: propose in-memory pessimistic locks before leader transfer ref #11452 Signed-off-by: Yilin Chen <[email protected]> * mark delete in scheduler and delete after apply Signed-off-by: Yilin Chen <[email protected]> * Add tests for two different orders between proposing locks and write command Signed-off-by: Yilin Chen <[email protected]> * do not clear pessimistic locks before transfer leader Signed-off-by: Yilin Chen <[email protected]> * delete memory locks in the apply thread and add more term and version check Signed-off-by: Yilin Chen <[email protected]> * remove potentially incorrect assert Signed-off-by: Yilin Chen <[email protected]> * check term and version when reading the lock table Signed-off-by: Yilin Chen <[email protected]> * revert a log change Signed-off-by: Yilin Chen <[email protected]> * change term when becoming follower Signed-off-by: Yilin Chen <[email protected]> * fix lint Signed-off-by: Yilin Chen <[email protected]> * do not check version when reading locks (temporarily) Signed-off-by: Yilin Chen <[email protected]> * fix another test Signed-off-by: Yilin Chen <[email protected]> * only treat MsgTransferLeader with context with valid msg Signed-off-by: Yilin Chen <[email protected]> * fix a tiny comment error Signed-off-by: Yilin Chen <[email protected]> * make clippy happy Signed-off-by: Yilin Chen <[email protected]> * res_meter: ignore the read keys test temporarily (#11779) ref #11765 Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: change critical to fatal (#11651) (#11780) * logger: change critical to fatal (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: fix test_parse_log_level (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * update CHANGELOG.md (#11790) * update from tikv-server v5.0.1(20210423) from https://github.com/tikv/tikv/releases Signed-off-by: xiejiandong <[email protected]> * update CHANGELOG.md close #11167 Signed-off-by: xiejiandong <[email protected]> * config: relax the bounds of online configurable worker number (#11651) (#11798) close #11776 Signed-off-by: Wenbo Zhang <[email protected]> * logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792) close #11789 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: unify thread related native APIs (#11785) * *: unify thread related native APIs Now there are duplicated codes calling syscalls to fetch thread informations. This PR provides unified APIs and remove duplicated codes. Close #11784. Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * declare platform compatability Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * Fix method on_tick will not update metric when enable_auto_tune is false (#11791) * Fix method on_tick will not update metric when enable_auto_tune is false ref #11787 Signed-off-by: yuqi1129 <[email protected]> * resolve format problem in discussion Signed-off-by: yuqi1129 <[email protected]> * resolve format problem in discussion again Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: add a new tick to renew lease in advance (#6427) * renew lease in raft tick Signed-off-by: 5kbpers <[email protected]> * check lease before sending read index Signed-off-by: 5kbpers <[email protected]> * renew lease on raft tick to cover lease till next tick Signed-off-by: linning <[email protected]> * refactor Signed-off-by: linning <[email protected]> * don't use inspect_lease Signed-off-by: linning <[email protected]> * fix ci Signed-off-by: linning <[email protected]> * add test Signed-off-by: linning <[email protected]> * remove unneed comment Signed-off-by: linning <[email protected]> * close #5388 Signed-off-by: 5kbpers <[email protected]> * check pending_read_count before hibernating Signed-off-by: 5kbpers <[email protected]> * increase the range to check lease expired Signed-off-by: 5kbpers <[email protected]> * renew leader lease after not hibernate Signed-off-by: 5kbpers <[email protected]> * fix hibernate states metrics Signed-off-by: 5kbpers <[email protected]> * check writes and pre_read_index before proposing read index Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * use a individual tick to renewing lease Signed-off-by: 5kbpers <[email protected]> * fix condition Signed-off-by: 5kbpers <[email protected]> * stablize test_node_renew_lease Signed-off-by: 5kbpers <[email protected]> * cleanup unused changes Signed-off-by: 5kbpers <[email protected]> * add config item for interval of tick Signed-off-by: 5kbpers <[email protected]> * rename tick Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_inconsistent_configuration Signed-off-by: 5kbpers <[email protected]> * change deafault value of check_leader_lease_interval Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_read_hibernated_region Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * count renew lease read index into raft metrics Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_renew_lease Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix lint error Signed-off-by: 5kbpers <[email protected]> * add a 100ms buffer for renew_bound Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * make clippy happy Signed-off-by: 5kbpers <[email protected]> Co-authored-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cloud: Add retry for azure blob server busy error (#11813) * add retry for azure blob server busy error Signed-off-by: Leavrth <[email protected]> * close#11812 Signed-off-by: Leavrth <[email protected]> * use to_string Signed-off-by: Leavrth <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * tikv_util: make cgroup parsing more robust (#11786) * make cgroup parsing more robust Signed-off-by: tabokie <[email protected]> * fix error message Signed-off-by: tabokie <[email protected]> * parse cpu quota in floats Signed-off-by: tabokie <[email protected]> * *: make it compile on macos (#11825) * *: make it compile on macos close #11823 Signed-off-by: Jay Lee <[email protected]> * remove extra comment Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * resolved_ts: add more metrics (#11809) * add metrics Signed-off-by: 5kbpers <[email protected]> * add slow log Signed-off-by: 5kbpers <[email protected]> * remove duplicate metric Signed-off-by: 5kbpers <[email protected]> * add resolved ts metrics for leader peers Signed-off-by: 5kbpers <[email protected]> * close #11767 Signed-off-by: 5kbpers <[email protected]> * init endpoint with store meta Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: fix perf regression caused by arc swap (#11833) * rsmeter: fix perf regression caused by arc swap Signed-off-by: Zhenchi <[email protected]> * use atomic cell instead Signed-off-by: Zhenchi <[email protected]> * debug assert Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * load base split: more accurate sample for large key ranges batch (#11039) * more accurate sample for large key ranges Signed-off-by: lhy1024 <[email protected]> * update Signed-off-by: lhy1024 <[email protected]> * ref #11521 Signed-off-by: lhy1024 <[email protected]> * address comment Signed-off-by: lhy1024 <[email protected]> * fix test Signed-off-by: lhy1024 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftclient: enable delay only in high load (#11772) and improve CPU efficiency under high load. But it turns out this can lead to significant regression in normal load. So this PR adds CPU stats and only enabling delay in high load. This is also the same strategy used in v4.x. Close #11769. Signed-off-by: Jay Lee <[email protected]> * tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231) Signed-off-by: Yao Zongyou <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Connor <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: skip deleting snapshot files in peer pending_remove is true (#11782) * raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746 This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash. Signed-off-by: tonyxuqqi <[email protected]> * address code review feedback -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> Co-authored-by: qi.xu <[email protected]> * load_statis: fix compilation on macos (#11851) close #11772 Signed-off-by: Jay Lee <[email protected]> * config: override rocksdb config when flow control enabled (#11840) Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rocksdb: consider env sharing when configuring background threads (#11760) #9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c). This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled. * consider env sharing when configuring background threads ref #11119 Signed-off-by: tabokie <[email protected]> * server/config: keep compatible using option (#11862) * server/config: keep compatible using option Using `Option` for `heavy_load_wait_duration` to keep compatible with versions prior to v5.3. Close #11861. Signed-off-by: Jay Lee <[email protected]> * fix format Signed-off-by: Jay Lee <[email protected]> * fix clippy Signed-off-by: Jay Lee <[email protected]> Co-authored-by: zhouqiang <[email protected]> * raftstore: fix missing workers' stop on shutdown (#11864) ref #11159 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: move in-memory pessimistic locks to split regions (#11655) ref #11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: Remove the entrance of enabling io snooper. (#11394) * *: Remove the entrance of enabling io snooper. close #10867 Signed-off-by: MuZhou233 <[email protected]> * fix Signed-off-by: MuZhou233 <[email protected]> * fix test Signed-off-by: MuZhou233 <[email protected]> * fix test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: change PeerTicks to enum (#11849) * raftstore: change PeerTicks to enum close #11848 Signed-off-by: Yilin Chen <[email protected]> * use bool array for registry Signed-off-by: Yilin Chen <[email protected]> * rename PeerTicks to PeerTick Signed-off-by: Yilin Chen <[email protected]> * add associated const for number of PeerTick types Signed-off-by: Yilin Chen <[email protected]> * change naming Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check memory limit when inserting in-memory pessimistic locks (#11841) * *: check memory limit when inserting in-memory pessimistic locks ref #11452 Signed-off-by: Yilin Chen <[email protected]> * fix merged upstream Signed-off-by: Yilin Chen <[email protected]> * fix test Signed-off-by: Yilin Chen <[email protected]> * fix typo Signed-off-by: Yilin Chen <[email protected]> * *: fix invalid failpoint caused by typo (#11709) * fix invalid failpoint caused by typo close #11734 Signed-off-by: Ryan Leung <[email protected]> * address the comment Signed-off-by: Ryan Leung <[email protected]> * address the comment Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Extract significant router (#11750) * ref#11409 add significant router Signed-off-by: Connor1996 <[email protected]> * make format Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * raftstore: move scan delete to raft log gc worker (#11853) * raftstore: move scan delete to raft log gc worker When clearing raft metas, raftstore will scan raft logs and delete them one by one. Seeking can be slow if there are a lot of tombstone keys. This PR moves the operation to raft log gc worker to reduce the impact. The final solution should be also moving remaining IO operations to async write IO threads. Close #10210. Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * speed up destroy Signed-off-by: Jay Lee <[email protected]> * fix compile Signed-off-by: Jay Lee <[email protected]> * further speed up Signed-off-by: Jay Lee <[email protected]> * revert test case configuration Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * doc: update new rules for linking issue and commit message (#11832) close #11097, close #11831 Signed-off-by: zhangyangyu <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Xiaoguang Sun <[email protected]> * *: update rust toolchain to 2022-01-07 (#11875) * update rust toolchain to 2022-01-17 Signed-off-by: tabokie <[email protected]> * address comment: clean up unnecessary `as_ref()` Signed-off-by: tabokie <[email protected]> * try fixing the underflow Signed-off-by: tabokie <[email protected]> * mute underflow warning for raft metrics Signed-off-by: tabokie <[email protected]> * clean up unused data members Signed-off-by: tabokie <[email protected]> * format Signed-off-by: tabokie <[email protected]> * step back to 2022-01-07 Signed-off-by: tabokie <[email protected]> * readme: change images based on github theme (#11795) Signed-off-by: sloorush <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: don't remove other peer's read delegate (#11882) * don't remove read delegate besides peer_destroy Signed-off-by: linning <[email protected]> * add test case Signed-off-by: linning <[email protected]> * make clippy happy Signed-off-by: linning <[email protected]> * address comment Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: use eprintln! if the logger is not initialized (#11869) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: fix the calculation of total column size in analyze (#11884) Signed-off-by: xuyifan <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * engine: update raft-engine for data consistency fix (#11885) Fix https://github.com/tikv/raft-engine/issues/142. When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart. * update raft-engine Signed-off-by: tabokie <[email protected]> * update one more commit Signed-off-by: tabokie <[email protected]> * check docker build during clippy, ref #11312 (#11819) Signed-off-by: tabokie <[email protected]> * gc_worker: fix incorrect scheduled_tasks counting (#11904) * gc_worker: fix incorrect scheduled_tasks counting close #11903 Signed-off-by: Yilin Chen <[email protected]> * remove check_is_busy totally Signed-off-by: Yilin Chen <[email protected]> * do not use wildcard match in error handling Signed-off-by: Yilin Chen <[email protected]> * raftstore: Introduce raft log fetcher (#11900) * ref#11409 introduce raft log fetcher Signed-off-by: Connor1996 <[email protected]> * update kvproto Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * fix test build Signed-off-by: Connor1996 <[email protected]> * fix clippy Signed-off-by: Connor1996 <[email protected]> * limit capacity Signed-off-by: Connor1996 <[email protected]> * update kvproto Signed-off-by: Connor1996 <[email protected]> * call stop on worker Signed-off-by: Connor1996 <[email protected]> * rename worker Signed-off-by: Connor1996 <[email protected]> * status_server: add pprof flamegraph response header (#10951) * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <[email protected]> * test(status_server): add Content-Type asset to test_pprof_profile_service Signed-off-by: Suhaha <[email protected]> * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <[email protected]> * tweak(status_server): add pprof flamegraph response header close #11917 Signed-off-by: Suhaha <[email protected]> * close #11917 Signed-off-by: Suhaha <[email protected]> Co-authored-by: goroutine <[email protected]> Co-authored-by: Connor <[email protected]> * raftstore: renew leader lease in advance when handle read request (#9307) * renew lease advance Signed-off-by: linning <[email protected]> * add log Signed-off-by: linning <[email protected]> * make clippy happy Signed-off-by: linning <[email protected]> * ref #11579 Signed-off-by: 5kbpers <[email protected]> * reset raft tick Signed-off-by: 5kbpers <[email protected]> * set has ready Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * add renew_leader_lease_advance_duration config Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * fix panic Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * disable renewing for test_lease_unsafe_during_leader_transfers Signed-off-by: 5kbpers <[email protected]> Co-authored-by: 5kbpers <[email protected]> * pprof: ignore cpu profiling on non-x86 arch (#11925) * Disable cpu profiling on non-x86_64 arch Signed-off-by: mornyx <[email protected]> * Fix warns Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * github: add new pr requirement for linking issue and commit message (#11887) Signed-off-by: zhangyangyu <[email protected]> Co-authored-by: Xiaoguang Sun <[email protected]> * engine: enable raft engine by default (#11928) * enable raft engine by default and synchronize docs Signed-off-by: tabokie <[email protected]> * update raft engine Signed-off-by: tabokie <[email protected]> * update raft-engine and fix tests Signed-off-by: tabokie <[email protected]> * gc_worker: Limit the key range to scan for GcKeys tasks (#11922) close tikv/tikv#11752, close tikv/tikv#11902 Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: TransferLeader support multiple target peers (#11063) ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602 Signed-off-by: MrCroxx <[email protected]> * raftstore: remove the leaders field of `StoreMeta` (#11934) close tikv/tikv#11933 Remove `StoreMeta.leaders` ``` ### Related changes ### Check List  Tests  - Manual test (add detailed scripts or steps below) `cargo check` ### Release note  ```release-note None Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Fetch raft log in async manner (#11409) close tikv/tikv#10408, close tikv/tikv#11320 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: add quarter function (#11935) ref tikv/tikv#5751 Signed-off-by: zhongyong jin <[email protected]> * Enable full debug info for dev and test (#11949) ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572 ``` ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch --> ### Check List  Tests  - Manual test (add detailed scripts or steps below) - No code Manual test: Linux box with 8 cores (3:50 -> 4:00): ``` # Before this PR: $ make clean cargo clean rm -rf bin dist $ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 3m 50s real 3m50,487s user 38m11,859s sys 3m9,540s # After this PR: $ git diff diff --git a/Cargo.toml b/Cargo.toml index 71f5329d3..67cb9d183 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"] [profile.dev] opt-level = 0 -debug = 1 # required for line numbers in tests, see tikv #5049 +debug = true codegen-units = 4 lto = false incremental = true @@ -293,7 +293,7 @@ codegen-units = 4 [profile.test] opt-level = 0 -debug = 1 # enable line numbers by default for easy test debugging +debug = true codegen-units = 16 lto = false incremental = true mattias@msig:~/repos/tikv$ time make clean cargo clean rm -rf bin dist real 0m0,975s user 0m0,148s sys 0m0,828s mattias@msig:~/repos/tikv$ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 00s real 4m0,201s user 39m45,037s sys 3m16,397s ``` Macbook Air M1: ``` # Before: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 01s make build 1107.42s user 116.20s system 506% cpu 4:01.46 total # After: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 10s make build 1179.39s user 120.74s system 518% cpu 4:10.98 total ``` Side effects - Performance regression, Only when building, to the benefit of full debug info by default. ### Release note  ```release-note None Signed-off-by: Mattias Jonsson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * correct a metric about compaction filter (#11938) correct a metric about compaction filter Signed-off-by: qupeng <[email protected]> * raft: Fix possible panic on entries fetched callback (#11961) close tikv/tikv#11951 Signed-off-by: Connor1996 <[email protected]> * split_controller: refine the sample function and add some comments (#11952) Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * metrics: fix grid position in TiKV Details (#11936) ref tikv/tikv#11119 Fix grid positions in TiKV Details dashboard. Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cdc: capture old value from txn layer dynamically (#11896) close tikv/tikv#10091 Signed-off-by: hi-rustin <[email protected]> Signed-off-by: qupeng <[email protected]> Co-authored-by: hi-rustin <[email protected]> * split_controller: reorganize the structs and add more comments (#11967) Reorganize the structs and add more comments. Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support Bit column push downBit column (#11968) close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884 Signed-off-by: xiejiandong <[email protected]> * copr: support push down mod, sysdate to TiKV (#11970) close tikv/tikv#11916, ref tikv/tikv#11916 Signed-off-by: xiejiandong <[email protected]> * raftstore: propose in-memory pessimistic locks before prepare merge (#11758) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> * stats_monitor: reformat stats_monitor tick (#11972) Signed-off-by: lhy1024 <[email protected]> * copr: support extract scalar value from bit (#11980) close tikv/tikv#11893 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * diagnosis: support get cpu time of threads for macOS (#11978) close tikv/tikv#11977 Add the variant for macOS into `components/tikv_util/src/sys/thread.rs` Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Jay <[email protected]> * rust: update toolchain to fix missing rls (#11954) close tikv/tikv#11953 Signed-off-by: Connor1996 <[email protected]> Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Yilin Chen <[email protected]> * cdc: separate resolved region outliers (#11991) Separate broadcasing outlier regions and normal regions, so 1) downstreams know where they should send resolve lock requests, and 2) resolved ts of normal regions does not fallback. close pingcap/tiflow#4516 close pingcap/tiflow#4311 ref pingcap/tiflow#4146 Signed-off-by: Neil Shen <[email protected]> * *: collect key ranges for the read request in ResourceMeteringTag (#11995) close tikv/tikv#11988 *: collect key ranges for the read request in ResourceMeteringTag Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: move sample_threshold check out of the split_key iteration (#11986) split_controller: move sample_threshold check out of the split_key iteration Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support insertutf8 and lowerutf8 (#11987) ref tikv/tikv#5751 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support rest greatest/least functions (#12003) ref tikv/tikv#5751 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: force compact with gentleness (#12006) * force compact with gentleness Signed-off-by: tabokie <[email protected]> * address comment Signed-off-by: tabokie <[email protected]> * split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010) split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: fix the incorrect sampled key ranges number check (#12013) close tikv/tikv#12012 split_controller: fix the incorrect sampled key ranges number check Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Support adjusting max_inflight_msgs dynamically (#11866) close tikv/tikv#11865 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: fix missing flag solvement for bit column in decoding (#12016) ref tikv/tikv#7, close pingcap/tidb#32506 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * file_system: overhaul proc-based io tracing utilities (#11873) ref tikv/tikv#10867, fix tikv/tikv#11775 Signed-off-by: tabokie <[email protected]> * raftstore: Wait for the apply index equals to the commit index (#11716) ref tikv/tikv#10483 Signed-off-by: v01dstar <[email protected]> * copr: revert 11968/11980/12016 (#12030) ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506 Signed-off-by: yisaer <[email protected]> * build: fix a Mac build issue (#12027) ref tikv/tikv#10867, ref tikv/tikv#11873 Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: Extra physical table id column (#11931) close tikv/tikv#11888 Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer. Signed-off-by: Mattias Jonsson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: hotfix panic from tokio-timer (#12004) close tikv/tikv#11940, ref tikv/tikv#11940 *: hotfix panic from tokio-timer by enlarging the length of the level vector Signed-off-by: you06 <[email protected]> * build: add arm64 check on SSE in makefile (#12035) close tikv/tikv#12034 Signed-off-by: Jin Dong <[email protected]> Co-authored-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * replication mode: sync state to pd (#11751) Signed-off-by: disksing <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> * copr: support push bit column down (#12037) ref pingcap/tidb#30738 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: add more sample function test cases (#12058) split_controller: add more sample function test cases Signed-off-by: JmPotato <[email protected]> * copr: fix greatest/least time/date args type (#12056) Signed-off-by: yisaer <[email protected]> * cdc: advancing resolved ts correctly for clusters with tiflash (#12050) close pingcap/tiflow#4461 cdc: advancing resolved ts correctly for clusters with tiflash Signed-off-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: bump master branch version to v6.0.0-alpha (#12077) close tikv/tikv#12075 Signed-off-by: Yilin Chen <[email protected]> * raftstore: check uninitialized destroy for merge (#12055) close tikv/tikv#12048 When a peer is destroyed without being initialized, it will store itself to peer list and the region epoch is missing. In merge if such state is detected, it should abort merging instead of panicking. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: Add TruncateManager and TruncateWorker (#11553) ref tikv/rfcs#81 Signed-off-by: longfangsong <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * TiKV supports buckets (#11763) ref tikv/tikv#11759 Signed-off-by: qi.xu <[email protected]> Co-authored-by: qi.xu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * api_version: Codec for RawKV key (#12036) ref tikv/tikv#11965 Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2. Signed-off-by: pingyu <[email protected]> Co-authored-by: Andy Lok <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: fix some typos (#12066) Signed-off-by: cuishuang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: fix stale message cause panic (#12054) close tikv/tikv#12023 raftstore: fix stale message cause panic Signed-off-by: linning <[email protected]> * raftstore: ignore async fetch result when the peer is pending removed (#12038) close tikv/tikv#11973, close tikv/tikv#12026 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: remove part about wechat in doc (#12101) close tikv/tikv#12100 Signed-off-by: jackwener <[email protected]> * engine: upgrade raft engine (#12095) ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119 Signed-off-by: Randy <[email protected]> Co-authored-by: Xinye Tao <[email protected]> * *: Fix possible undefined behavior for transmuting vec (#12096) close tikv/tikv#12070 Signed-off-by: Connor1996 <zbk602423539@gm…

* storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216) * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * fix enable_ttl Signed-off-by: andylokandy <[email protected]> * simplify test Signed-off-by: andylokandy <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> Co-authored-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cdc: load uninlined value more effectively (#11615) Signed-off-by: qupeng <[email protected]> * cdc: load old value with prefix seek if possible (#11643) * close #11642 Signed-off-by: qupeng <[email protected]> * a more comment Signed-off-by: qupeng <[email protected]> * ctl: split into multiple modules (#11658) * ctl: split into multiple modules ref #10938 Signed-off-by: andylokandy <[email protected]> * refactor if let to match Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * Fix build issues on non-linux platforms (#11671) * close #11676 Signed-off-by: devillove084 <[email protected]> * all: Migrate from std:: types to primitives. (#11677) * Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref #11678 ```release-note NONE ``` Signed-off-by: Harold Dost <[email protected]> * metrics alert: fix apply cpu alert (#11672) Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * resrc_mtr: introduce scanned keys statistics (#11085) close #11690 Signed-off-by: mornyx <[email protected]> * file_system: support collect the io flow by proc (#11667) * file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * fix clippy Signed-off-by: MuZhou233 <[email protected]> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <[email protected]> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * Integrate thread_io Signed-off-by: MuZhou233 <[email protected]> * record bytes in request Signed-off-by: MuZhou233 <[email protected]> * fix logical bug Signed-off-by: MuZhou233 <[email protected]> * Update test write_bytes Signed-off-by: MuZhou233 <[email protected]> * Delete unused & Rename func Signed-off-by: MuZhou233 <[email protected]> * Add test for read & Update test for write Signed-off-by: MuZhou233 <[email protected]> * Add buffered thread io Signed-off-by: MuZhou233 <[email protected]> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <[email protected]> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <[email protected]> * Change fetch logic. close #10867 Signed-off-by: MuZhou233 <[email protected]> * Add AtomicIOBytes Signed-off-by: MuZhou233 <[email protected]> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <[email protected]> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <[email protected]> * Adjust code style & fix Signed-off-by: MuZhou233 <[email protected]> * improve write_bytes test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check epoch and validness of lock table when writing pessimistic locks (#11591) ref #11452 Signed-off-by: Yilin Chen <[email protected]> * rsmeter: introduce datasink (#11688) * add datasink trait Signed-off-by: Zhenchi <[email protected]> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build ref #11691 Signed-off-by: Zhenchi <[email protected]> * datasink -> data_sink Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * clippy: Fixing up clippy errors. (#11193) * clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref #4301 Signed-off-by: Harold Dost <[email protected]> * Test Revert of Iterations ref #4301 Signed-off-by: Harold Dost <[email protected]> * clippy: Enable check for rust-2018-idioms ref #4301 Signed-off-by: Harold Dost <[email protected]> * rsmeter: add centralized place to construct protobuf data (#11696) ref #11691 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * tikv_util: update procfs to 0.12.0 (#11703) Signed-off-by: linning <[email protected]> * Use generic for api version to reduce runtime branching (#11687) * Use generic for api version to reduce runtime branching ref #10938 Signed-off-by: andylokandy <[email protected]> * fix test Signed-off-by: andylokandy <[email protected]> * add test Signed-off-by: andylokandy <[email protected]> * fix clippy Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * fmt Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * rename vars Signed-off-by: andylokandy <[email protected]> * improve vars Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * sst_importer: check api version when importing (#11664) * sst_importer: check api version when importing ref #10938 Signed-off-by: Peng Guanwen <[email protected]> * Fix bugs Signed-off-by: Peng Guanwen <[email protected]> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <[email protected]> * remove redundant log Signed-off-by: Peng Guanwen <[email protected]> * Show the entire key Signed-off-by: Peng Guanwen <[email protected]> * Fix lint issue Signed-off-by: Peng Guanwen <[email protected]> * resolve conflict Signed-off-by: andylokandy <[email protected]> Co-authored-by: Andy Lok <[email protected]> * backup: pipeline scan and save step (#11528) * br: sperate io and scan threads Signed-off-by: Yu Juncen <[email protected]> * br/stream: make clippy happy Signed-off-by: yujuncen <[email protected]> * br/stream: fix tests Signed-off-by: Yu Juncen <[email protected]> * backup: fix a dummy bug Signed-off-by: Yu Juncen <[email protected]> * backup: add config of io-threads Signed-off-by: yujuncen <[email protected]> * backup: ref #11350: some minior change Signed-off-by: yujuncen <[email protected]> * backup: added some metrics Signed-off-by: yujuncen <[email protected]> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <[email protected]> * backup: added some slow log and metrics Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: fix build Signed-off-by: yujuncen <[email protected]> * Revert "backup: fix build" This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e. Signed-off-by: yujuncen <[email protected]> * Revert "backup: address comments" This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa. Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: always set IO type to Export Signed-off-by: yujuncen <[email protected]> * backup:make clippy happy Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * metrics: fix the metrics cannot be displayed (#11710) ref #11662 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * backup: return api-version to BR when backup (#11704) * Update kvproto Signed-off-by: Peng Guanwen <[email protected]> * Return api-version for br ref #10938 Signed-off-by: Peng Guanwen <[email protected]> * Reformat code Signed-off-by: Peng Guanwen <[email protected]> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * format code Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * file_system: limit thread_io target os (#11715) * limit thread_io target os Signed-off-by: MuZhou233 <[email protected]> * limit thread_io target os. typo. close #11698 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Yilin Chen <[email protected]> * file_system: bypass file allocate implementation in fs2 (#11700) Close #10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <[email protected]> * metrics: remove min legend of some Grafana panels (#11487) Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref #11119 Signed-off-by: tabokie <[email protected]> * coprocessor: not return rows when there is no input for simple aggregation (#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <[email protected]> * close #11735, and address comments Signed-off-by: xufei <[email protected]> * backup: reduce the default thread pool size of backup, enable auto tune by default (#11699) * backup: clamp auto tune values Signed-off-by: yujuncen <[email protected]> * ref #11000: change the default backup threads. Signed-off-by: yujuncen <[email protected]> * backup: run tasks in background Signed-off-by: yujuncen <[email protected]> * backup: run rustfmt Signed-off-by: yujuncen <[email protected]> * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen <[email protected]> * br-stream: fix build Signed-off-by: yujuncen <[email protected]> * Backup: add S3 metrics && add s3_multi_part_size config (#11666) * s3: add request metrics for s3 storage Signed-off-by: 3pointer <[email protected]> * s3: add grafana json Signed-off-by: 3pointer <[email protected]> * br: add config for s3 multi part upload Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * hidden the new config Signed-off-by: 3pointer <[email protected]> * only hidden config in config-template.toml Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * close #11727 && format Signed-off-by: 3pointer <[email protected]> * fix Signed-off-by: 3pointer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: support multiple datasinks (#11707) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <[email protected]> * wrap config notifier Signed-off-by: Zhenchi <[email protected]> * add unit tests Signed-off-by: Zhenchi <[email protected]> * polish ref #11691 Signed-off-by: Zhenchi <[email protected]> * address comments Signed-off-by: Zhenchi <[email protected]> * fix test Signed-off-by: Zhenchi <[email protected]> * fix outdated comments Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * engine: properly estimate store size (#11728) * properly estimate store size, ref #11119 Signed-off-by: tabokie <[email protected]> * address comments Signed-off-by: tabokie <[email protected]> * implement global config client for tikv (#11685) * implement global config client close #11686 Signed-off-by: lemonhx <[email protected]> * implement unit test for global config in pd client Signed-off-by: lemonhx <[email protected]> * resolved formatting issues Signed-off-by: lemonhx <[email protected]> * impl pd client asyncly Signed-off-by: lemonhx <[email protected]> * according to reviewer's opinion removing store method implementation Signed-off-by: lemonhx <[email protected]> * Merge branch 'master' of https://github.com/tikv/tikv into global_conf Signed-off-by: lemonhx <[email protected]> * write test case for watch global config when grpc server is closed. Signed-off-by: lemonhx <[email protected]> * resove issues Signed-off-by: lemonhx <[email protected]> * clippy Signed-off-by: lemonhx <[email protected]> * watch global config rentry future in test deadlock just removed Signed-off-by: lemonhx <[email protected]> Co-authored-by: Zhenchi <[email protected]> * storage: skip scanning lock when using rc (#11701) * skip scanning lock when using rc, ref #11485 Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> * add test Signed-off-by: you06 <[email protected]> * address comment Signed-off-by: you06 <[email protected]> * add an integration test Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: pushdown substring to tikv (#11494) * corp: pushdown substring to tikv Signed-off-by: guo-shaoge <[email protected]> * fix case. close #11495 Signed-off-by: guo-shaoge <[email protected]> * fix comment Signed-off-by: guo-shaoge <[email protected]> * rsmeter: distinguish between collectors and observers within recorder (#11712) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <[email protected]> * wrap config notifier Signed-off-by: Zhenchi <[email protected]> * add unit tests Signed-off-by: Zhenchi <[email protected]> * polish ref #11691 Signed-off-by: Zhenchi <[email protected]> * rsmeter: distinguish between collectors and observers within recorder Signed-off-by: Zhenchi <[email protected]> * fmt & try to resolve #11689 Signed-off-by: Zhenchi <[email protected]> * add unit tests ref #11691 Signed-off-by: Zhenchi <[email protected]> * retrigger test Signed-off-by: Zhenchi <[email protected]> * address comments Signed-off-by: Zhenchi <[email protected]> * fix test Signed-off-by: Zhenchi <[email protected]> * fix outdated comments Signed-off-by: Zhenchi <[email protected]> * Update components/resource_metering/src/recorder/sub_recorder/mod.rs Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> * address comment Signed-off-by: Zhenchi <[email protected]> * remove enabled Signed-off-by: Zhenchi <[email protected]> * fix Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Wenxuan <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> * config: validate online configurable thread pools (tikv#11159) (#11714) ref #11159 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * res_meter: upgrade kvproto (#11749) * Upgrade kvproto close #11748 Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Use master kvproto Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: add pubsub datasink (#11719) * rsmeter: add pubsub datasink close #11691 Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> * Update components/resource_metering/src/reporter/pubsub.rs Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * status_server: Support online config update for configuration hosting platform (#11693) close #11692 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * res_meter: optimize top k (#11753) * Keep top k on recorder Signed-off-by: mornyx <[email protected]> * Modify comments Signed-off-by: mornyx <[email protected]> * Fix pd.rs Signed-off-by: mornyx <[email protected]> * Refactor Signed-off-by: mornyx <[email protected]> * Remove import Signed-off-by: mornyx <[email protected]> * max_resource_groups 200 -> 100 Signed-off-by: mornyx <[email protected]> * Fix ut Signed-off-by: mornyx <[email protected]> * Use iterator to avoid clone Signed-off-by: mornyx <[email protected]> * Optimize close #11754 Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftclient: delay flush (#11705) * raftclient: delay flush Signed-off-by: Jay Lee <[email protected]> * remove raftstore wait Signed-off-by: Jay Lee <[email protected]> * Ref #11309. tikv#11310 introduces delay in raftstore thread to reduce the flush rate. It can only reduce flush rate from one thread. If there are many raftstore threads or IO threads, messages can still be flush frequently. This PR reduces the flush rate by introducing delay in raft client. So delay will work at connection level and achieve the maximun batch affect. Signed-off-by: Jay Lee <[email protected]> * remove useless field Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * fix unstable raw get case Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Liqi Geng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cloud: support azure blob storage as external storage backend (#11732) * support azure storage blob for external storage Signed-off-by: Leavrth <[email protected]> * modify the priority of the azure login methods Signed-off-by: Leavrth <[email protected]> * fix feature match branch Signed-off-by: Leavrth <[email protected]> * implement token update retriable Signed-off-by: Leavrth <[email protected]> * fix some problems Signed-off-by: Leavrth <[email protected]> * close #11731 Signed-off-by: Leavrth <[email protected]> * rustfmt Signed-off-by: Leavrth <[email protected]> * make clippy success Signed-off-by: Leavrth <[email protected]> * simplify uploader Signed-off-by: Leavrth <[email protected]> * commit some suggestions Signed-off-by: Leavrth <[email protected]> * coprocesser: update encoding lib (#11506) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * coprocesser: implement upper and lower function on GBK charset (#11756) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: support archiving and rotation (#11651) (#11657) * logger: support archiving and rotation (#11651) close #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: add testcases to test dt_from_file_name (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: simplify DateTime format (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * diagnostics: refactor (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: run clippy for integration tests (#11768) * run clippy --fix with nightly-2021-12-17 Signed-off-by: tabokie <[email protected]> * manual clippy Signed-off-by: tabokie <[email protected]> * format and clippy Signed-off-by: tabokie <[email protected]> * enable default features when running clippy, ref #11119 Signed-off-by: tabokie <[email protected]> * fix build Signed-off-by: tabokie <[email protected]> * revert box_collection lint Signed-off-by: tabokie <[email protected]> * Update tests/integrations/raftstore/test_bootstrap.rs Signed-off-by: Xinye Tao <[email protected]> Co-authored-by: Lei Zhao <[email protected]> Co-authored-by: Lei Zhao <[email protected]> * logger: fix config-template's log unit (#11651) (#11777) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: improve testcase (#11651) (#11778) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * raftstore: propose in-memory pessimistic locks before leader transfer (#11577) * *: check epoch and validness of lock table when writing pessimistic locks ref #11452 Signed-off-by: Yilin Chen <[email protected]> * raftstore: propose in-memory pessimistic locks before leader transfer ref #11452 Signed-off-by: Yilin Chen <[email protected]> * mark delete in scheduler and delete after apply Signed-off-by: Yilin Chen <[email protected]> * Add tests for two different orders between proposing locks and write command Signed-off-by: Yilin Chen <[email protected]> * do not clear pessimistic locks before transfer leader Signed-off-by: Yilin Chen <[email protected]> * delete memory locks in the apply thread and add more term and version check Signed-off-by: Yilin Chen <[email protected]> * remove potentially incorrect assert Signed-off-by: Yilin Chen <[email protected]> * check term and version when reading the lock table Signed-off-by: Yilin Chen <[email protected]> * revert a log change Signed-off-by: Yilin Chen <[email protected]> * change term when becoming follower Signed-off-by: Yilin Chen <[email protected]> * fix lint Signed-off-by: Yilin Chen <[email protected]> * do not check version when reading locks (temporarily) Signed-off-by: Yilin Chen <[email protected]> * fix another test Signed-off-by: Yilin Chen <[email protected]> * only treat MsgTransferLeader with context with valid msg Signed-off-by: Yilin Chen <[email protected]> * fix a tiny comment error Signed-off-by: Yilin Chen <[email protected]> * make clippy happy Signed-off-by: Yilin Chen <[email protected]> * res_meter: ignore the read keys test temporarily (#11779) ref #11765 Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: change critical to fatal (#11651) (#11780) * logger: change critical to fatal (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: fix test_parse_log_level (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * update CHANGELOG.md (#11790) * update from tikv-server v5.0.1(20210423) from https://github.com/tikv/tikv/releases Signed-off-by: xiejiandong <[email protected]> * update CHANGELOG.md close #11167 Signed-off-by: xiejiandong <[email protected]> * config: relax the bounds of online configurable worker number (#11651) (#11798) close #11776 Signed-off-by: Wenbo Zhang <[email protected]> * logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792) close #11789 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: unify thread related native APIs (#11785) * *: unify thread related native APIs Now there are duplicated codes calling syscalls to fetch thread informations. This PR provides unified APIs and remove duplicated codes. Close #11784. Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * declare platform compatability Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * Fix method on_tick will not update metric when enable_auto_tune is false (#11791) * Fix method on_tick will not update metric when enable_auto_tune is false ref #11787 Signed-off-by: yuqi1129 <[email protected]> * resolve format problem in discussion Signed-off-by: yuqi1129 <[email protected]> * resolve format problem in discussion again Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: add a new tick to renew lease in advance (#6427) * renew lease in raft tick Signed-off-by: 5kbpers <[email protected]> * check lease before sending read index Signed-off-by: 5kbpers <[email protected]> * renew lease on raft tick to cover lease till next tick Signed-off-by: linning <[email protected]> * refactor Signed-off-by: linning <[email protected]> * don't use inspect_lease Signed-off-by: linning <[email protected]> * fix ci Signed-off-by: linning <[email protected]> * add test Signed-off-by: linning <[email protected]> * remove unneed comment Signed-off-by: linning <[email protected]> * close #5388 Signed-off-by: 5kbpers <[email protected]> * check pending_read_count before hibernating Signed-off-by: 5kbpers <[email protected]> * increase the range to check lease expired Signed-off-by: 5kbpers <[email protected]> * renew leader lease after not hibernate Signed-off-by: 5kbpers <[email protected]> * fix hibernate states metrics Signed-off-by: 5kbpers <[email protected]> * check writes and pre_read_index before proposing read index Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * use a individual tick to renewing lease Signed-off-by: 5kbpers <[email protected]> * fix condition Signed-off-by: 5kbpers <[email protected]> * stablize test_node_renew_lease Signed-off-by: 5kbpers <[email protected]> * cleanup unused changes Signed-off-by: 5kbpers <[email protected]> * add config item for interval of tick Signed-off-by: 5kbpers <[email protected]> * rename tick Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_inconsistent_configuration Signed-off-by: 5kbpers <[email protected]> * change deafault value of check_leader_lease_interval Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_read_hibernated_region Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * count renew lease read index into raft metrics Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_renew_lease Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix lint error Signed-off-by: 5kbpers <[email protected]> * add a 100ms buffer for renew_bound Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * make clippy happy Signed-off-by: 5kbpers <[email protected]> Co-authored-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cloud: Add retry for azure blob server busy error (#11813) * add retry for azure blob server busy error Signed-off-by: Leavrth <[email protected]> * close#11812 Signed-off-by: Leavrth <[email protected]> * use to_string Signed-off-by: Leavrth <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * tikv_util: make cgroup parsing more robust (#11786) * make cgroup parsing more robust Signed-off-by: tabokie <[email protected]> * fix error message Signed-off-by: tabokie <[email protected]> * parse cpu quota in floats Signed-off-by: tabokie <[email protected]> * *: make it compile on macos (#11825) * *: make it compile on macos close #11823 Signed-off-by: Jay Lee <[email protected]> * remove extra comment Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * resolved_ts: add more metrics (#11809) * add metrics Signed-off-by: 5kbpers <[email protected]> * add slow log Signed-off-by: 5kbpers <[email protected]> * remove duplicate metric Signed-off-by: 5kbpers <[email protected]> * add resolved ts metrics for leader peers Signed-off-by: 5kbpers <[email protected]> * close #11767 Signed-off-by: 5kbpers <[email protected]> * init endpoint with store meta Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: fix perf regression caused by arc swap (#11833) * rsmeter: fix perf regression caused by arc swap Signed-off-by: Zhenchi <[email protected]> * use atomic cell instead Signed-off-by: Zhenchi <[email protected]> * debug assert Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * load base split: more accurate sample for large key ranges batch (#11039) * more accurate sample for large key ranges Signed-off-by: lhy1024 <[email protected]> * update Signed-off-by: lhy1024 <[email protected]> * ref #11521 Signed-off-by: lhy1024 <[email protected]> * address comment Signed-off-by: lhy1024 <[email protected]> * fix test Signed-off-by: lhy1024 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftclient: enable delay only in high load (#11772) and improve CPU efficiency under high load. But it turns out this can lead to significant regression in normal load. So this PR adds CPU stats and only enabling delay in high load. This is also the same strategy used in v4.x. Close #11769. Signed-off-by: Jay Lee <[email protected]> * tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231) Signed-off-by: Yao Zongyou <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Connor <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: skip deleting snapshot files in peer pending_remove is true (#11782) * raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746 This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash. Signed-off-by: tonyxuqqi <[email protected]> * address code review feedback -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> Co-authored-by: qi.xu <[email protected]> * load_statis: fix compilation on macos (#11851) close #11772 Signed-off-by: Jay Lee <[email protected]> * config: override rocksdb config when flow control enabled (#11840) Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rocksdb: consider env sharing when configuring background threads (#11760) #9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c). This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled. * consider env sharing when configuring background threads ref #11119 Signed-off-by: tabokie <[email protected]> * server/config: keep compatible using option (#11862) * server/config: keep compatible using option Using `Option` for `heavy_load_wait_duration` to keep compatible with versions prior to v5.3. Close #11861. Signed-off-by: Jay Lee <[email protected]> * fix format Signed-off-by: Jay Lee <[email protected]> * fix clippy Signed-off-by: Jay Lee <[email protected]> Co-authored-by: zhouqiang <[email protected]> * raftstore: fix missing workers' stop on shutdown (#11864) ref #11159 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: move in-memory pessimistic locks to split regions (#11655) ref #11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: Remove the entrance of enabling io snooper. (#11394) * *: Remove the entrance of enabling io snooper. close #10867 Signed-off-by: MuZhou233 <[email protected]> * fix Signed-off-by: MuZhou233 <[email protected]> * fix test Signed-off-by: MuZhou233 <[email protected]> * fix test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: change PeerTicks to enum (#11849) * raftstore: change PeerTicks to enum close #11848 Signed-off-by: Yilin Chen <[email protected]> * use bool array for registry Signed-off-by: Yilin Chen <[email protected]> * rename PeerTicks to PeerTick Signed-off-by: Yilin Chen <[email protected]> * add associated const for number of PeerTick types Signed-off-by: Yilin Chen <[email protected]> * change naming Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check memory limit when inserting in-memory pessimistic locks (#11841) * *: check memory limit when inserting in-memory pessimistic locks ref #11452 Signed-off-by: Yilin Chen <[email protected]> * fix merged upstream Signed-off-by: Yilin Chen <[email protected]> * fix test Signed-off-by: Yilin Chen <[email protected]> * fix typo Signed-off-by: Yilin Chen <[email protected]> * *: fix invalid failpoint caused by typo (#11709) * fix invalid failpoint caused by typo close #11734 Signed-off-by: Ryan Leung <[email protected]> * address the comment Signed-off-by: Ryan Leung <[email protected]> * address the comment Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Extract significant router (#11750) * ref#11409 add significant router Signed-off-by: Connor1996 <[email protected]> * make format Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * raftstore: move scan delete to raft log gc worker (#11853) * raftstore: move scan delete to raft log gc worker When clearing raft metas, raftstore will scan raft logs and delete them one by one. Seeking can be slow if there are a lot of tombstone keys. This PR moves the operation to raft log gc worker to reduce the impact. The final solution should be also moving remaining IO operations to async write IO threads. Close #10210. Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * speed up destroy Signed-off-by: Jay Lee <[email protected]> * fix compile Signed-off-by: Jay Lee <[email protected]> * further speed up Signed-off-by: Jay Lee <[email protected]> * revert test case configuration Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * doc: update new rules for linking issue and commit message (#11832) close #11097, close #11831 Signed-off-by: zhangyangyu <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Xiaoguang Sun <[email protected]> * *: update rust toolchain to 2022-01-07 (#11875) * update rust toolchain to 2022-01-17 Signed-off-by: tabokie <[email protected]> * address comment: clean up unnecessary `as_ref()` Signed-off-by: tabokie <[email protected]> * try fixing the underflow Signed-off-by: tabokie <[email protected]> * mute underflow warning for raft metrics Signed-off-by: tabokie <[email protected]> * clean up unused data members Signed-off-by: tabokie <[email protected]> * format Signed-off-by: tabokie <[email protected]> * step back to 2022-01-07 Signed-off-by: tabokie <[email protected]> * readme: change images based on github theme (#11795) Signed-off-by: sloorush <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: don't remove other peer's read delegate (#11882) * don't remove read delegate besides peer_destroy Signed-off-by: linning <[email protected]> * add test case Signed-off-by: linning <[email protected]> * make clippy happy Signed-off-by: linning <[email protected]> * address comment Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: use eprintln! if the logger is not initialized (#11869) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: fix the calculation of total column size in analyze (#11884) Signed-off-by: xuyifan <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * engine: update raft-engine for data consistency fix (#11885) Fix https://github.com/tikv/raft-engine/issues/142. When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart. * update raft-engine Signed-off-by: tabokie <[email protected]> * update one more commit Signed-off-by: tabokie <[email protected]> * check docker build during clippy, ref #11312 (#11819) Signed-off-by: tabokie <[email protected]> * gc_worker: fix incorrect scheduled_tasks counting (#11904) * gc_worker: fix incorrect scheduled_tasks counting close #11903 Signed-off-by: Yilin Chen <[email protected]> * remove check_is_busy totally Signed-off-by: Yilin Chen <[email protected]> * do not use wildcard match in error handling Signed-off-by: Yilin Chen <[email protected]> * raftstore: Introduce raft log fetcher (#11900) * ref#11409 introduce raft log fetcher Signed-off-by: Connor1996 <[email protected]> * update kvproto Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * fix test build Signed-off-by: Connor1996 <[email protected]> * fix clippy Signed-off-by: Connor1996 <[email protected]> * limit capacity Signed-off-by: Connor1996 <[email protected]> * update kvproto Signed-off-by: Connor1996 <[email protected]> * call stop on worker Signed-off-by: Connor1996 <[email protected]> * rename worker Signed-off-by: Connor1996 <[email protected]> * status_server: add pprof flamegraph response header (#10951) * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <[email protected]> * test(status_server): add Content-Type asset to test_pprof_profile_service Signed-off-by: Suhaha <[email protected]> * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <[email protected]> * tweak(status_server): add pprof flamegraph response header close #11917 Signed-off-by: Suhaha <[email protected]> * close #11917 Signed-off-by: Suhaha <[email protected]> Co-authored-by: goroutine <[email protected]> Co-authored-by: Connor <[email protected]> * raftstore: renew leader lease in advance when handle read request (#9307) * renew lease advance Signed-off-by: linning <[email protected]> * add log Signed-off-by: linning <[email protected]> * make clippy happy Signed-off-by: linning <[email protected]> * ref #11579 Signed-off-by: 5kbpers <[email protected]> * reset raft tick Signed-off-by: 5kbpers <[email protected]> * set has ready Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * add renew_leader_lease_advance_duration config Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * fix panic Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * disable renewing for test_lease_unsafe_during_leader_transfers Signed-off-by: 5kbpers <[email protected]> Co-authored-by: 5kbpers <[email protected]> * pprof: ignore cpu profiling on non-x86 arch (#11925) * Disable cpu profiling on non-x86_64 arch Signed-off-by: mornyx <[email protected]> * Fix warns Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * github: add new pr requirement for linking issue and commit message (#11887) Signed-off-by: zhangyangyu <[email protected]> Co-authored-by: Xiaoguang Sun <[email protected]> * engine: enable raft engine by default (#11928) * enable raft engine by default and synchronize docs Signed-off-by: tabokie <[email protected]> * update raft engine Signed-off-by: tabokie <[email protected]> * update raft-engine and fix tests Signed-off-by: tabokie <[email protected]> * gc_worker: Limit the key range to scan for GcKeys tasks (#11922) close tikv/tikv#11752, close tikv/tikv#11902 Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: TransferLeader support multiple target peers (#11063) ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602 Signed-off-by: MrCroxx <[email protected]> * raftstore: remove the leaders field of `StoreMeta` (#11934) close tikv/tikv#11933 Remove `StoreMeta.leaders` ``` ### Related changes ### Check List  Tests  - Manual test (add detailed scripts or steps below) `cargo check` ### Release note  ```release-note None Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Fetch raft log in async manner (#11409) close tikv/tikv#10408, close tikv/tikv#11320 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: add quarter function (#11935) ref tikv/tikv#5751 Signed-off-by: zhongyong jin <[email protected]> * Enable full debug info for dev and test (#11949) ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572 ``` ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch --> ### Check List  Tests  - Manual test (add detailed scripts or steps below) - No code Manual test: Linux box with 8 cores (3:50 -> 4:00): ``` # Before this PR: $ make clean cargo clean rm -rf bin dist $ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 3m 50s real 3m50,487s user 38m11,859s sys 3m9,540s # After this PR: $ git diff diff --git a/Cargo.toml b/Cargo.toml index 71f5329d3..67cb9d183 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"] [profile.dev] opt-level = 0 -debug = 1 # required for line numbers in tests, see tikv #5049 +debug = true codegen-units = 4 lto = false incremental = true @@ -293,7 +293,7 @@ codegen-units = 4 [profile.test] opt-level = 0 -debug = 1 # enable line numbers by default for easy test debugging +debug = true codegen-units = 16 lto = false incremental = true mattias@msig:~/repos/tikv$ time make clean cargo clean rm -rf bin dist real 0m0,975s user 0m0,148s sys 0m0,828s mattias@msig:~/repos/tikv$ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 00s real 4m0,201s user 39m45,037s sys 3m16,397s ``` Macbook Air M1: ``` # Before: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 01s make build 1107.42s user 116.20s system 506% cpu 4:01.46 total # After: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 10s make build 1179.39s user 120.74s system 518% cpu 4:10.98 total ``` Side effects - Performance regression, Only when building, to the benefit of full debug info by default. ### Release note  ```release-note None Signed-off-by: Mattias Jonsson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * correct a metric about compaction filter (#11938) correct a metric about compaction filter Signed-off-by: qupeng <[email protected]> * raft: Fix possible panic on entries fetched callback (#11961) close tikv/tikv#11951 Signed-off-by: Connor1996 <[email protected]> * split_controller: refine the sample function and add some comments (#11952) Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * metrics: fix grid position in TiKV Details (#11936) ref tikv/tikv#11119 Fix grid positions in TiKV Details dashboard. Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cdc: capture old value from txn layer dynamically (#11896) close tikv/tikv#10091 Signed-off-by: hi-rustin <[email protected]> Signed-off-by: qupeng <[email protected]> Co-authored-by: hi-rustin <[email protected]> * split_controller: reorganize the structs and add more comments (#11967) Reorganize the structs and add more comments. Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support Bit column push downBit column (#11968) close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884 Signed-off-by: xiejiandong <[email protected]> * copr: support push down mod, sysdate to TiKV (#11970) close tikv/tikv#11916, ref tikv/tikv#11916 Signed-off-by: xiejiandong <[email protected]> * raftstore: propose in-memory pessimistic locks before prepare merge (#11758) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> * stats_monitor: reformat stats_monitor tick (#11972) Signed-off-by: lhy1024 <[email protected]> * copr: support extract scalar value from bit (#11980) close tikv/tikv#11893 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * diagnosis: support get cpu time of threads for macOS (#11978) close tikv/tikv#11977 Add the variant for macOS into `components/tikv_util/src/sys/thread.rs` Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Jay <[email protected]> * rust: update toolchain to fix missing rls (#11954) close tikv/tikv#11953 Signed-off-by: Connor1996 <[email protected]> Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Yilin Chen <[email protected]> * cdc: separate resolved region outliers (#11991) Separate broadcasing outlier regions and normal regions, so 1) downstreams know where they should send resolve lock requests, and 2) resolved ts of normal regions does not fallback. close pingcap/tiflow#4516 close pingcap/tiflow#4311 ref pingcap/tiflow#4146 Signed-off-by: Neil Shen <[email protected]> * *: collect key ranges for the read request in ResourceMeteringTag (#11995) close tikv/tikv#11988 *: collect key ranges for the read request in ResourceMeteringTag Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: move sample_threshold check out of the split_key iteration (#11986) split_controller: move sample_threshold check out of the split_key iteration Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support insertutf8 and lowerutf8 (#11987) ref tikv/tikv#5751 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support rest greatest/least functions (#12003) ref tikv/tikv#5751 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: force compact with gentleness (#12006) * force compact with gentleness Signed-off-by: tabokie <[email protected]> * address comment Signed-off-by: tabokie <[email protected]> * split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010) split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: fix the incorrect sampled key ranges number check (#12013) close tikv/tikv#12012 split_controller: fix the incorrect sampled key ranges number check Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Support adjusting max_inflight_msgs dynamically (#11866) close tikv/tikv#11865 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: fix missing flag solvement for bit column in decoding (#12016) ref tikv/tikv#7, close pingcap/tidb#32506 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * file_system: overhaul proc-based io tracing utilities (#11873) ref tikv/tikv#10867, fix tikv/tikv#11775 Signed-off-by: tabokie <[email protected]> * raftstore: Wait for the apply index equals to the commit index (#11716) ref tikv/tikv#10483 Signed-off-by: v01dstar <[email protected]> * copr: revert 11968/11980/12016 (#12030) ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506 Signed-off-by: yisaer <[email protected]> * build: fix a Mac build issue (#12027) ref tikv/tikv#10867, ref tikv/tikv#11873 Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: Extra physical table id column (#11931) close tikv/tikv#11888 Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer. Signed-off-by: Mattias Jonsson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: hotfix panic from tokio-timer (#12004) close tikv/tikv#11940, ref tikv/tikv#11940 *: hotfix panic from tokio-timer by enlarging the length of the level vector Signed-off-by: you06 <[email protected]> * build: add arm64 check on SSE in makefile (#12035) close tikv/tikv#12034 Signed-off-by: Jin Dong <[email protected]> Co-authored-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * replication mode: sync state to pd (#11751) Signed-off-by: disksing <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> * copr: support push bit column down (#12037) ref pingcap/tidb#30738 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: add more sample function test cases (#12058) split_controller: add more sample function test cases Signed-off-by: JmPotato <[email protected]> * copr: fix greatest/least time/date args type (#12056) Signed-off-by: yisaer <[email protected]> * cdc: advancing resolved ts correctly for clusters with tiflash (#12050) close pingcap/tiflow#4461 cdc: advancing resolved ts correctly for clusters with tiflash Signed-off-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: bump master branch version to v6.0.0-alpha (#12077) close tikv/tikv#12075 Signed-off-by: Yilin Chen <[email protected]> * raftstore: check uninitialized destroy for merge (#12055) close tikv/tikv#12048 When a peer is destroyed without being initialized, it will store itself to peer list and the region epoch is missing. In merge if such state is detected, it should abort merging instead of panicking. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: Add TruncateManager and TruncateWorker (#11553) ref tikv/rfcs#81 Signed-off-by: longfangsong <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * TiKV supports buckets (#11763) ref tikv/tikv#11759 Signed-off-by: qi.xu <[email protected]> Co-authored-by: qi.xu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * api_version: Codec for RawKV key (#12036) ref tikv/tikv#11965 Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2. Signed-off-by: pingyu <[email protected]> Co-authored-by: Andy Lok <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: fix some typos (#12066) Signed-off-by: cuishuang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: fix stale message cause panic (#12054) close tikv/tikv#12023 raftstore: fix stale message cause panic Signed-off-by: linning <[email protected]> * raftstore: ignore async fetch result when the peer is pending removed (#12038) close tikv/tikv#11973, close tikv/tikv#12026 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: remove part about wechat in doc (#12101) close tikv/tikv#12100 Signed-off-by: jackwener <[email protected]> * engine: upgrade raft engine (#12095) ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119 Signed-off-by: Randy <[email protected]> Co-authored-by: Xinye Tao <[email protected]> * *: Fix possible undefined behavior for transmuting vec (#12096) close tikv/tikv#12070 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * github: Remove the pingcap/tidb-ansible from the PR template (#12102) close tikv/tikv#12103 Signed-off-by: hi-rustin <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: extract reschedule to inject a fail point (#12108) Signed-off-by: Zhenchi <[email protected]> * raftstore: reactivate in-memory pessimistic locking when leader transfer fails (#11883) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check memory locks for replica read only on the leader (#12115) close tikv/tikv#12109 Consider a reader sends read index to the leader it supposed to be, but when the leader receives the read index message, it has stepped down to a follower. Without this commit, a peer will check the locks and re-fill the read index context with the result no matter what role it is. So, when the read index request is redirected to the new leader again, it no longer carries the context of lock checking. This commmit changes to only do the check when the peer is a leader. Then, the read index request will remain unchanged before being redirected to the leader. If the lease of the leader expires, it is still safe to check on the uncertained leader. If the heartbeat check passes and no new leader is elected, then the check works fine. If there is a new leader, when the old leader becomes a follower, it will clear its pending read index. Then, the reader has to resend the read index again to the correct leader. So, generally it is safe as long as we guarantee the check only happens on the leader. Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * apiv2 rawkv: logical deletion flag (#12090) ref tikv/tikv#11965 Signed-off-by: haojinming <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: add projection (#10689) close tikv/tikv#12114 Signed-off-by: ichn-hu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Alex Chi <[email protected]> * raftstore: adjust raft max msg size dynamically (#12018) close tikv/tikv#12017 Signed-off-by: glorv <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: update deps (#12098) - Upgrade grpcio to 0.10 - Upgrade openssl to 1.1.1m - Remove dependency on prost Signed-off-by: Jay Lee <[email protected]> * tests: make raftstore tests about transactions more stable (#12122) close tikv/tikv#12120 A new fail point "after_propose" is added to insert callback from the test. Then we can ensure we release other key fail points after they take effect. The other change is to retry request automatically to avoid stale command errors. Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * metrics: support multi k8s in grafana dashboards (#12008) close tikv/tikv#12104 Signed-off-by: just1900 <[email protected]> * storage: set in-memory pessimistic lock feature gate to 6.0.0 (#12078) ref tikv/tikv#11452 The in-memory pessimistic lock feature should be only enabled after all TiKV instances have been upgraded to the new version. Otherwise, transferring leader may be blocked because of the new protocol and procedure when in-memory pessimistic locks exist. So, we need to use a feature gate to make sure this feature is only enabled after all TiKVs have been upgraded to the new version. Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore,cdc: pass leader transferee to cdc observer (#12124) Signed-off-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * config: simplify handling of nullable configurations (#12133) Support deriving `O…

…roxy (#72) * ctl: validate column family name when scanning data using ctl (#11269) * ctl: should check column name validation when scan data using ctl close #11268 Signed-off-by: yuqi1129 <[email protected]> * fix format problem Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> * txn: Do assertions on data for mutations to reduce potential data inconsistency (#10977) * Add mutation checking logic Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * Fix usages of mutation Signed-off-by: MyonKeminta <[email protected]> * Update kvproto and renaming Signed-off-by: MyonKeminta <[email protected]> * Add a simple test Signed-off-by: MyonKeminta <[email protected]> * Fix incorrect assertion Signed-off-by: MyonKeminta <[email protected]> * Support check existance during pessimistic lock Signed-off-by: MyonKeminta <[email protected]> * Add assertion level support Signed-off-by: MyonKeminta <[email protected]> * Add tests Signed-off-by: MyonKeminta <[email protected]> * Add more test cases Signed-off-by: MyonKeminta <[email protected]> * Add test for check_existence of pessimistic_lock request Signed-off-by: MyonKeminta <[email protected]> * Fix build Signed-off-by: MyonKeminta <[email protected]> * ref #11542 Signed-off-by: MyonKeminta <[email protected]> * Add a metric for counting different way of checking assertion, for reference about the performance impact Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * metrics: fix memory metrics (#11656) Signed-off-by: Shuaipeng Yu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * coprocesser: refactor encoding and add gbk_chinese_ci map (#11661) * close #11660 Signed-off-by: xiongjiwei <[email protected]> * close #11660 Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216) * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> * fix enable_ttl Signed-off-by: andylokandy <[email protected]> * simplify test Signed-off-by: andylokandy <[email protected]> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <[email protected]> Co-authored-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cdc: load uninlined value more effectively (#11615) Signed-off-by: qupeng <[email protected]> * cdc: load old value with prefix seek if possible (#11643) * close #11642 Signed-off-by: qupeng <[email protected]> * a more comment Signed-off-by: qupeng <[email protected]> * ctl: split into multiple modules (#11658) * ctl: split into multiple modules ref #10938 Signed-off-by: andylokandy <[email protected]> * refactor if let to match Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * Fix build issues on non-linux platforms (#11671) * close #11676 Signed-off-by: devillove084 <[email protected]> * all: Migrate from std:: types to primitives. (#11677) * Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref #11678 ```release-note NONE ``` Signed-off-by: Harold Dost <[email protected]> * metrics alert: fix apply cpu alert (#11672) Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * resrc_mtr: introduce scanned keys statistics (#11085) close #11690 Signed-off-by: mornyx <[email protected]> * file_system: support collect the io flow by proc (#11667) * file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * fix clippy Signed-off-by: MuZhou233 <[email protected]> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <[email protected]> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <[email protected]> * fix lint Signed-off-by: MuZhou233 <[email protected]> * Integrate thread_io Signed-off-by: MuZhou233 <[email protected]> * record bytes in request Signed-off-by: MuZhou233 <[email protected]> * fix logical bug Signed-off-by: MuZhou233 <[email protected]> * Update test write_bytes Signed-off-by: MuZhou233 <[email protected]> * Delete unused & Rename func Signed-off-by: MuZhou233 <[email protected]> * Add test for read & Update test for write Signed-off-by: MuZhou233 <[email protected]> * Add buffered thread io Signed-off-by: MuZhou233 <[email protected]> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <[email protected]> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <[email protected]> * Change fetch logic. close #10867 Signed-off-by: MuZhou233 <[email protected]> * Add AtomicIOBytes Signed-off-by: MuZhou233 <[email protected]> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <[email protected]> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <[email protected]> * Adjust code style & fix Signed-off-by: MuZhou233 <[email protected]> * improve write_bytes test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check epoch and validness of lock table when writing pessimistic locks (#11591) ref #11452 Signed-off-by: Yilin Chen <[email protected]> * rsmeter: introduce datasink (#11688) * add datasink trait Signed-off-by: Zhenchi <[email protected]> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build ref #11691 Signed-off-by: Zhenchi <[email protected]> * datasink -> data_sink Signed-off-by: Zhenchi <[email protected]> * fmt Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * clippy: Fixing up clippy errors. (#11193) * clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref #4301 Signed-off-by: Harold Dost <[email protected]> * Test Revert of Iterations ref #4301 Signed-off-by: Harold Dost <[email protected]> * clippy: Enable check for rust-2018-idioms ref #4301 Signed-off-by: Harold Dost <[email protected]> * rsmeter: add centralized place to construct protobuf data (#11696) ref #11691 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * tikv_util: update procfs to 0.12.0 (#11703) Signed-off-by: linning <[email protected]> * Use generic for api version to reduce runtime branching (#11687) * Use generic for api version to reduce runtime branching ref #10938 Signed-off-by: andylokandy <[email protected]> * fix test Signed-off-by: andylokandy <[email protected]> * add test Signed-off-by: andylokandy <[email protected]> * fix clippy Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * fmt Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * address comment Signed-off-by: andylokandy <[email protected]> * rename vars Signed-off-by: andylokandy <[email protected]> * improve vars Signed-off-by: andylokandy <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * sst_importer: check api version when importing (#11664) * sst_importer: check api version when importing ref #10938 Signed-off-by: Peng Guanwen <[email protected]> * Fix bugs Signed-off-by: Peng Guanwen <[email protected]> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <[email protected]> * remove redundant log Signed-off-by: Peng Guanwen <[email protected]> * Show the entire key Signed-off-by: Peng Guanwen <[email protected]> * Fix lint issue Signed-off-by: Peng Guanwen <[email protected]> * resolve conflict Signed-off-by: andylokandy <[email protected]> Co-authored-by: Andy Lok <[email protected]> * backup: pipeline scan and save step (#11528) * br: sperate io and scan threads Signed-off-by: Yu Juncen <[email protected]> * br/stream: make clippy happy Signed-off-by: yujuncen <[email protected]> * br/stream: fix tests Signed-off-by: Yu Juncen <[email protected]> * backup: fix a dummy bug Signed-off-by: Yu Juncen <[email protected]> * backup: add config of io-threads Signed-off-by: yujuncen <[email protected]> * backup: ref #11350: some minior change Signed-off-by: yujuncen <[email protected]> * backup: added some metrics Signed-off-by: yujuncen <[email protected]> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <[email protected]> * backup: added some slow log and metrics Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: fix build Signed-off-by: yujuncen <[email protected]> * Revert "backup: fix build" This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e. Signed-off-by: yujuncen <[email protected]> * Revert "backup: address comments" This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa. Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * backup: always set IO type to Export Signed-off-by: yujuncen <[email protected]> * backup:make clippy happy Signed-off-by: yujuncen <[email protected]> * backup: address comments Signed-off-by: yujuncen <[email protected]> * metrics: fix the metrics cannot be displayed (#11710) ref #11662 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * backup: return api-version to BR when backup (#11704) * Update kvproto Signed-off-by: Peng Guanwen <[email protected]> * Return api-version for br ref #10938 Signed-off-by: Peng Guanwen <[email protected]> * Reformat code Signed-off-by: Peng Guanwen <[email protected]> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * format code Signed-off-by: Peng Guanwen <[email protected]> Co-authored-by: Andy Lok <[email protected]> * file_system: limit thread_io target os (#11715) * limit thread_io target os Signed-off-by: MuZhou233 <[email protected]> * limit thread_io target os. typo. close #11698 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Yilin Chen <[email protected]> * file_system: bypass file allocate implementation in fs2 (#11700) Close #10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <[email protected]> * metrics: remove min legend of some Grafana panels (#11487) Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref #11119 Signed-off-by: tabokie <[email protected]> * coprocessor: not return rows when there is no input for simple aggregation (#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <[email protected]> * close #11735, and address comments Signed-off-by: xufei <[email protected]> * backup: reduce the default thread pool size of backup, enable auto tune by default (#11699) * backup: clamp auto tune values Signed-off-by: yujuncen <[email protected]> * ref #11000: change the default backup threads. Signed-off-by: yujuncen <[email protected]> * backup: run tasks in background Signed-off-by: yujuncen <[email protected]> * backup: run rustfmt Signed-off-by: yujuncen <[email protected]> * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen <[email protected]> * br-stream: fix build Signed-off-by: yujuncen <[email protected]> * Backup: add S3 metrics && add s3_multi_part_size config (#11666) * s3: add request metrics for s3 storage Signed-off-by: 3pointer <[email protected]> * s3: add grafana json Signed-off-by: 3pointer <[email protected]> * br: add config for s3 multi part upload Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * update comment Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * hidden the new config Signed-off-by: 3pointer <[email protected]> * only hidden config in config-template.toml Signed-off-by: 3pointer <[email protected]> * address comment Signed-off-by: 3pointer <[email protected]> * close #11727 && format Signed-off-by: 3pointer <[email protected]> * fix Signed-off-by: 3pointer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: support multiple datasinks (#11707) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <[email protected]> * wrap config notifier Signed-off-by: Zhenchi <[email protected]> * add unit tests Signed-off-by: Zhenchi <[email protected]> * polish ref #11691 Signed-off-by: Zhenchi <[email protected]> * address comments Signed-off-by: Zhenchi <[email protected]> * fix test Signed-off-by: Zhenchi <[email protected]> * fix outdated comments Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * engine: properly estimate store size (#11728) * properly estimate store size, ref #11119 Signed-off-by: tabokie <[email protected]> * address comments Signed-off-by: tabokie <[email protected]> * implement global config client for tikv (#11685) * implement global config client close #11686 Signed-off-by: lemonhx <[email protected]> * implement unit test for global config in pd client Signed-off-by: lemonhx <[email protected]> * resolved formatting issues Signed-off-by: lemonhx <[email protected]> * impl pd client asyncly Signed-off-by: lemonhx <[email protected]> * according to reviewer's opinion removing store method implementation Signed-off-by: lemonhx <[email protected]> * Merge branch 'master' of https://github.com/tikv/tikv into global_conf Signed-off-by: lemonhx <[email protected]> * write test case for watch global config when grpc server is closed. Signed-off-by: lemonhx <[email protected]> * resove issues Signed-off-by: lemonhx <[email protected]> * clippy Signed-off-by: lemonhx <[email protected]> * watch global config rentry future in test deadlock just removed Signed-off-by: lemonhx <[email protected]> Co-authored-by: Zhenchi <[email protected]> * storage: skip scanning lock when using rc (#11701) * skip scanning lock when using rc, ref #11485 Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> * add test Signed-off-by: you06 <[email protected]> * address comment Signed-off-by: you06 <[email protected]> * add an integration test Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: pushdown substring to tikv (#11494) * corp: pushdown substring to tikv Signed-off-by: guo-shaoge <[email protected]> * fix case. close #11495 Signed-off-by: guo-shaoge <[email protected]> * fix comment Signed-off-by: guo-shaoge <[email protected]> * rsmeter: distinguish between collectors and observers within recorder (#11712) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <[email protected]> * wrap config notifier Signed-off-by: Zhenchi <[email protected]> * add unit tests Signed-off-by: Zhenchi <[email protected]> * polish ref #11691 Signed-off-by: Zhenchi <[email protected]> * rsmeter: distinguish between collectors and observers within recorder Signed-off-by: Zhenchi <[email protected]> * fmt & try to resolve #11689 Signed-off-by: Zhenchi <[email protected]> * add unit tests ref #11691 Signed-off-by: Zhenchi <[email protected]> * retrigger test Signed-off-by: Zhenchi <[email protected]> * address comments Signed-off-by: Zhenchi <[email protected]> * fix test Signed-off-by: Zhenchi <[email protected]> * fix outdated comments Signed-off-by: Zhenchi <[email protected]> * Update components/resource_metering/src/recorder/sub_recorder/mod.rs Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> * address comment Signed-off-by: Zhenchi <[email protected]> * remove enabled Signed-off-by: Zhenchi <[email protected]> * fix Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Wenxuan <[email protected]> Co-authored-by: Yexiang Zhang <[email protected]> * config: validate online configurable thread pools (tikv#11159) (#11714) ref #11159 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * res_meter: upgrade kvproto (#11749) * Upgrade kvproto close #11748 Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Fix test compilation Signed-off-by: mornyx <[email protected]> * Use master kvproto Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: add pubsub datasink (#11719) * rsmeter: add pubsub datasink close #11691 Signed-off-by: Zhenchi <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> * Update components/resource_metering/src/reporter/pubsub.rs Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * fix build Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * status_server: Support online config update for configuration hosting platform (#11693) close #11692 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * res_meter: optimize top k (#11753) * Keep top k on recorder Signed-off-by: mornyx <[email protected]> * Modify comments Signed-off-by: mornyx <[email protected]> * Fix pd.rs Signed-off-by: mornyx <[email protected]> * Refactor Signed-off-by: mornyx <[email protected]> * Remove import Signed-off-by: mornyx <[email protected]> * max_resource_groups 200 -> 100 Signed-off-by: mornyx <[email protected]> * Fix ut Signed-off-by: mornyx <[email protected]> * Use iterator to avoid clone Signed-off-by: mornyx <[email protected]> * Optimize close #11754 Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftclient: delay flush (#11705) * raftclient: delay flush Signed-off-by: Jay Lee <[email protected]> * remove raftstore wait Signed-off-by: Jay Lee <[email protected]> * Ref #11309. tikv#11310 introduces delay in raftstore thread to reduce the flush rate. It can only reduce flush rate from one thread. If there are many raftstore threads or IO threads, messages can still be flush frequently. This PR reduces the flush rate by introducing delay in raft client. So delay will work at connection level and achieve the maximun batch affect. Signed-off-by: Jay Lee <[email protected]> * remove useless field Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * fix unstable raw get case Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Liqi Geng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cloud: support azure blob storage as external storage backend (#11732) * support azure storage blob for external storage Signed-off-by: Leavrth <[email protected]> * modify the priority of the azure login methods Signed-off-by: Leavrth <[email protected]> * fix feature match branch Signed-off-by: Leavrth <[email protected]> * implement token update retriable Signed-off-by: Leavrth <[email protected]> * fix some problems Signed-off-by: Leavrth <[email protected]> * close #11731 Signed-off-by: Leavrth <[email protected]> * rustfmt Signed-off-by: Leavrth <[email protected]> * make clippy success Signed-off-by: Leavrth <[email protected]> * simplify uploader Signed-off-by: Leavrth <[email protected]> * commit some suggestions Signed-off-by: Leavrth <[email protected]> * coprocesser: update encoding lib (#11506) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * coprocesser: implement upper and lower function on GBK charset (#11756) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: support archiving and rotation (#11651) (#11657) * logger: support archiving and rotation (#11651) close #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: add testcases to test dt_from_file_name (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: simplify DateTime format (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * diagnostics: refactor (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: run clippy for integration tests (#11768) * run clippy --fix with nightly-2021-12-17 Signed-off-by: tabokie <[email protected]> * manual clippy Signed-off-by: tabokie <[email protected]> * format and clippy Signed-off-by: tabokie <[email protected]> * enable default features when running clippy, ref #11119 Signed-off-by: tabokie <[email protected]> * fix build Signed-off-by: tabokie <[email protected]> * revert box_collection lint Signed-off-by: tabokie <[email protected]> * Update tests/integrations/raftstore/test_bootstrap.rs Signed-off-by: Xinye Tao <[email protected]> Co-authored-by: Lei Zhao <[email protected]> Co-authored-by: Lei Zhao <[email protected]> * logger: fix config-template's log unit (#11651) (#11777) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: improve testcase (#11651) (#11778) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * raftstore: propose in-memory pessimistic locks before leader transfer (#11577) * *: check epoch and validness of lock table when writing pessimistic locks ref #11452 Signed-off-by: Yilin Chen <[email protected]> * raftstore: propose in-memory pessimistic locks before leader transfer ref #11452 Signed-off-by: Yilin Chen <[email protected]> * mark delete in scheduler and delete after apply Signed-off-by: Yilin Chen <[email protected]> * Add tests for two different orders between proposing locks and write command Signed-off-by: Yilin Chen <[email protected]> * do not clear pessimistic locks before transfer leader Signed-off-by: Yilin Chen <[email protected]> * delete memory locks in the apply thread and add more term and version check Signed-off-by: Yilin Chen <[email protected]> * remove potentially incorrect assert Signed-off-by: Yilin Chen <[email protected]> * check term and version when reading the lock table Signed-off-by: Yilin Chen <[email protected]> * revert a log change Signed-off-by: Yilin Chen <[email protected]> * change term when becoming follower Signed-off-by: Yilin Chen <[email protected]> * fix lint Signed-off-by: Yilin Chen <[email protected]> * do not check version when reading locks (temporarily) Signed-off-by: Yilin Chen <[email protected]> * fix another test Signed-off-by: Yilin Chen <[email protected]> * only treat MsgTransferLeader with context with valid msg Signed-off-by: Yilin Chen <[email protected]> * fix a tiny comment error Signed-off-by: Yilin Chen <[email protected]> * make clippy happy Signed-off-by: Yilin Chen <[email protected]> * res_meter: ignore the read keys test temporarily (#11779) ref #11765 Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: change critical to fatal (#11651) (#11780) * logger: change critical to fatal (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * logger: fix test_parse_log_level (#11651) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> * update CHANGELOG.md (#11790) * update from tikv-server v5.0.1(20210423) from https://github.com/tikv/tikv/releases Signed-off-by: xiejiandong <[email protected]> * update CHANGELOG.md close #11167 Signed-off-by: xiejiandong <[email protected]> * config: relax the bounds of online configurable worker number (#11651) (#11798) close #11776 Signed-off-by: Wenbo Zhang <[email protected]> * logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792) close #11789 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: unify thread related native APIs (#11785) * *: unify thread related native APIs Now there are duplicated codes calling syscalls to fetch thread informations. This PR provides unified APIs and remove duplicated codes. Close #11784. Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * declare platform compatability Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * Fix method on_tick will not update metric when enable_auto_tune is false (#11791) * Fix method on_tick will not update metric when enable_auto_tune is false ref #11787 Signed-off-by: yuqi1129 <[email protected]> * resolve format problem in discussion Signed-off-by: yuqi1129 <[email protected]> * resolve format problem in discussion again Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: add a new tick to renew lease in advance (#6427) * renew lease in raft tick Signed-off-by: 5kbpers <[email protected]> * check lease before sending read index Signed-off-by: 5kbpers <[email protected]> * renew lease on raft tick to cover lease till next tick Signed-off-by: linning <[email protected]> * refactor Signed-off-by: linning <[email protected]> * don't use inspect_lease Signed-off-by: linning <[email protected]> * fix ci Signed-off-by: linning <[email protected]> * add test Signed-off-by: linning <[email protected]> * remove unneed comment Signed-off-by: linning <[email protected]> * close #5388 Signed-off-by: 5kbpers <[email protected]> * check pending_read_count before hibernating Signed-off-by: 5kbpers <[email protected]> * increase the range to check lease expired Signed-off-by: 5kbpers <[email protected]> * renew leader lease after not hibernate Signed-off-by: 5kbpers <[email protected]> * fix hibernate states metrics Signed-off-by: 5kbpers <[email protected]> * check writes and pre_read_index before proposing read index Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * use a individual tick to renewing lease Signed-off-by: 5kbpers <[email protected]> * fix condition Signed-off-by: 5kbpers <[email protected]> * stablize test_node_renew_lease Signed-off-by: 5kbpers <[email protected]> * cleanup unused changes Signed-off-by: 5kbpers <[email protected]> * add config item for interval of tick Signed-off-by: 5kbpers <[email protected]> * rename tick Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_inconsistent_configuration Signed-off-by: 5kbpers <[email protected]> * change deafault value of check_leader_lease_interval Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_read_hibernated_region Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * count renew lease read index into raft metrics Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * disable the tick for test_renew_lease Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix lint error Signed-off-by: 5kbpers <[email protected]> * add a 100ms buffer for renew_bound Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * make clippy happy Signed-off-by: 5kbpers <[email protected]> Co-authored-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cloud: Add retry for azure blob server busy error (#11813) * add retry for azure blob server busy error Signed-off-by: Leavrth <[email protected]> * close#11812 Signed-off-by: Leavrth <[email protected]> * use to_string Signed-off-by: Leavrth <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * tikv_util: make cgroup parsing more robust (#11786) * make cgroup parsing more robust Signed-off-by: tabokie <[email protected]> * fix error message Signed-off-by: tabokie <[email protected]> * parse cpu quota in floats Signed-off-by: tabokie <[email protected]> * *: make it compile on macos (#11825) * *: make it compile on macos close #11823 Signed-off-by: Jay Lee <[email protected]> * remove extra comment Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * resolved_ts: add more metrics (#11809) * add metrics Signed-off-by: 5kbpers <[email protected]> * add slow log Signed-off-by: 5kbpers <[email protected]> * remove duplicate metric Signed-off-by: 5kbpers <[email protected]> * add resolved ts metrics for leader peers Signed-off-by: 5kbpers <[email protected]> * close #11767 Signed-off-by: 5kbpers <[email protected]> * init endpoint with store meta Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rsmeter: fix perf regression caused by arc swap (#11833) * rsmeter: fix perf regression caused by arc swap Signed-off-by: Zhenchi <[email protected]> * use atomic cell instead Signed-off-by: Zhenchi <[email protected]> * debug assert Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Wenxuan <[email protected]> * load base split: more accurate sample for large key ranges batch (#11039) * more accurate sample for large key ranges Signed-off-by: lhy1024 <[email protected]> * update Signed-off-by: lhy1024 <[email protected]> * ref #11521 Signed-off-by: lhy1024 <[email protected]> * address comment Signed-off-by: lhy1024 <[email protected]> * fix test Signed-off-by: lhy1024 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftclient: enable delay only in high load (#11772) and improve CPU efficiency under high load. But it turns out this can lead to significant regression in normal load. So this PR adds CPU stats and only enabling delay in high load. This is also the same strategy used in v4.x. Close #11769. Signed-off-by: Jay Lee <[email protected]> * tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231) Signed-off-by: Yao Zongyou <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Connor <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: skip deleting snapshot files in peer pending_remove is true (#11782) * raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746 This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash. Signed-off-by: tonyxuqqi <[email protected]> * address code review feedback -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <[email protected]> Co-authored-by: qi.xu <[email protected]> * load_statis: fix compilation on macos (#11851) close #11772 Signed-off-by: Jay Lee <[email protected]> * config: override rocksdb config when flow control enabled (#11840) Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * rocksdb: consider env sharing when configuring background threads (#11760) #9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c). This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled. * consider env sharing when configuring background threads ref #11119 Signed-off-by: tabokie <[email protected]> * server/config: keep compatible using option (#11862) * server/config: keep compatible using option Using `Option` for `heavy_load_wait_duration` to keep compatible with versions prior to v5.3. Close #11861. Signed-off-by: Jay Lee <[email protected]> * fix format Signed-off-by: Jay Lee <[email protected]> * fix clippy Signed-off-by: Jay Lee <[email protected]> Co-authored-by: zhouqiang <[email protected]> * raftstore: fix missing workers' stop on shutdown (#11864) ref #11159 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: move in-memory pessimistic locks to split regions (#11655) ref #11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: Remove the entrance of enabling io snooper. (#11394) * *: Remove the entrance of enabling io snooper. close #10867 Signed-off-by: MuZhou233 <[email protected]> * fix Signed-off-by: MuZhou233 <[email protected]> * fix test Signed-off-by: MuZhou233 <[email protected]> * fix test Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: change PeerTicks to enum (#11849) * raftstore: change PeerTicks to enum close #11848 Signed-off-by: Yilin Chen <[email protected]> * use bool array for registry Signed-off-by: Yilin Chen <[email protected]> * rename PeerTicks to PeerTick Signed-off-by: Yilin Chen <[email protected]> * add associated const for number of PeerTick types Signed-off-by: Yilin Chen <[email protected]> * change naming Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check memory limit when inserting in-memory pessimistic locks (#11841) * *: check memory limit when inserting in-memory pessimistic locks ref #11452 Signed-off-by: Yilin Chen <[email protected]> * fix merged upstream Signed-off-by: Yilin Chen <[email protected]> * fix test Signed-off-by: Yilin Chen <[email protected]> * fix typo Signed-off-by: Yilin Chen <[email protected]> * *: fix invalid failpoint caused by typo (#11709) * fix invalid failpoint caused by typo close #11734 Signed-off-by: Ryan Leung <[email protected]> * address the comment Signed-off-by: Ryan Leung <[email protected]> * address the comment Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Extract significant router (#11750) * ref#11409 add significant router Signed-off-by: Connor1996 <[email protected]> * make format Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * raftstore: move scan delete to raft log gc worker (#11853) * raftstore: move scan delete to raft log gc worker When clearing raft metas, raftstore will scan raft logs and delete them one by one. Seeking can be slow if there are a lot of tombstone keys. This PR moves the operation to raft log gc worker to reduce the impact. The final solution should be also moving remaining IO operations to async write IO threads. Close #10210. Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * speed up destroy Signed-off-by: Jay Lee <[email protected]> * fix compile Signed-off-by: Jay Lee <[email protected]> * further speed up Signed-off-by: Jay Lee <[email protected]> * revert test case configuration Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> * address comment Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * doc: update new rules for linking issue and commit message (#11832) close #11097, close #11831 Signed-off-by: zhangyangyu <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Mini256 <[email protected]> Co-authored-by: Xiaoguang Sun <[email protected]> * *: update rust toolchain to 2022-01-07 (#11875) * update rust toolchain to 2022-01-17 Signed-off-by: tabokie <[email protected]> * address comment: clean up unnecessary `as_ref()` Signed-off-by: tabokie <[email protected]> * try fixing the underflow Signed-off-by: tabokie <[email protected]> * mute underflow warning for raft metrics Signed-off-by: tabokie <[email protected]> * clean up unused data members Signed-off-by: tabokie <[email protected]> * format Signed-off-by: tabokie <[email protected]> * step back to 2022-01-07 Signed-off-by: tabokie <[email protected]> * readme: change images based on github theme (#11795) Signed-off-by: sloorush <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: don't remove other peer's read delegate (#11882) * don't remove read delegate besides peer_destroy Signed-off-by: linning <[email protected]> * add test case Signed-off-by: linning <[email protected]> * make clippy happy Signed-off-by: linning <[email protected]> * address comment Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * logger: use eprintln! if the logger is not initialized (#11869) ref #11651 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: fix the calculation of total column size in analyze (#11884) Signed-off-by: xuyifan <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * engine: update raft-engine for data consistency fix (#11885) Fix https://github.com/tikv/raft-engine/issues/142. When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart. * update raft-engine Signed-off-by: tabokie <[email protected]> * update one more commit Signed-off-by: tabokie <[email protected]> * check docker build during clippy, ref #11312 (#11819) Signed-off-by: tabokie <[email protected]> * gc_worker: fix incorrect scheduled_tasks counting (#11904) * gc_worker: fix incorrect scheduled_tasks counting close #11903 Signed-off-by: Yilin Chen <[email protected]> * remove check_is_busy totally Signed-off-by: Yilin Chen <[email protected]> * do not use wildcard match in error handling Signed-off-by: Yilin Chen <[email protected]> * raftstore: Introduce raft log fetcher (#11900) * ref#11409 introduce raft log fetcher Signed-off-by: Connor1996 <[email protected]> * update kvproto Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * fix test build Signed-off-by: Connor1996 <[email protected]> * fix clippy Signed-off-by: Connor1996 <[email protected]> * limit capacity Signed-off-by: Connor1996 <[email protected]> * update kvproto Signed-off-by: Connor1996 <[email protected]> * call stop on worker Signed-off-by: Connor1996 <[email protected]> * rename worker Signed-off-by: Connor1996 <[email protected]> * status_server: add pprof flamegraph response header (#10951) * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <[email protected]> * test(status_server): add Content-Type asset to test_pprof_profile_service Signed-off-by: Suhaha <[email protected]> * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <[email protected]> * tweak(status_server): add pprof flamegraph response header close #11917 Signed-off-by: Suhaha <[email protected]> * close #11917 Signed-off-by: Suhaha <[email protected]> Co-authored-by: goroutine <[email protected]> Co-authored-by: Connor <[email protected]> * raftstore: renew leader lease in advance when handle read request (#9307) * renew lease advance Signed-off-by: linning <[email protected]> * add log Signed-off-by: linning <[email protected]> * make clippy happy Signed-off-by: linning <[email protected]> * ref #11579 Signed-off-by: 5kbpers <[email protected]> * reset raft tick Signed-off-by: 5kbpers <[email protected]> * set has ready Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * add renew_leader_lease_advance_duration config Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * fix panic Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * disable renewing for test_lease_unsafe_during_leader_transfers Signed-off-by: 5kbpers <[email protected]> Co-authored-by: 5kbpers <[email protected]> * pprof: ignore cpu profiling on non-x86 arch (#11925) * Disable cpu profiling on non-x86_64 arch Signed-off-by: mornyx <[email protected]> * Fix warns Signed-off-by: mornyx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * github: add new pr requirement for linking issue and commit message (#11887) Signed-off-by: zhangyangyu <[email protected]> Co-authored-by: Xiaoguang Sun <[email protected]> * engine: enable raft engine by default (#11928) * enable raft engine by default and synchronize docs Signed-off-by: tabokie <[email protected]> * update raft engine Signed-off-by: tabokie <[email protected]> * update raft-engine and fix tests Signed-off-by: tabokie <[email protected]> * gc_worker: Limit the key range to scan for GcKeys tasks (#11922) close tikv/tikv#11752, close tikv/tikv#11902 Signed-off-by: MyonKeminta <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: TransferLeader support multiple target peers (#11063) ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602 Signed-off-by: MrCroxx <[email protected]> * raftstore: remove the leaders field of `StoreMeta` (#11934) close tikv/tikv#11933 Remove `StoreMeta.leaders` ``` ### Related changes ### Check List  Tests  - Manual test (add detailed scripts or steps below) `cargo check` ### Release note  ```release-note None Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Fetch raft log in async manner (#11409) close tikv/tikv#10408, close tikv/tikv#11320 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: add quarter function (#11935) ref tikv/tikv#5751 Signed-off-by: zhongyong jin <[email protected]> * Enable full debug info for dev and test (#11949) ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572 ``` ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch --> ### Check List  Tests  - Manual test (add detailed scripts or steps below) - No code Manual test: Linux box with 8 cores (3:50 -> 4:00): ``` # Before this PR: $ make clean cargo clean rm -rf bin dist $ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 3m 50s real 3m50,487s user 38m11,859s sys 3m9,540s # After this PR: $ git diff diff --git a/Cargo.toml b/Cargo.toml index 71f5329d3..67cb9d183 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"] [profile.dev] opt-level = 0 -debug = 1 # required for line numbers in tests, see tikv #5049 +debug = true codegen-units = 4 lto = false incremental = true @@ -293,7 +293,7 @@ codegen-units = 4 [profile.test] opt-level = 0 -debug = 1 # enable line numbers by default for easy test debugging +debug = true codegen-units = 16 lto = false incremental = true mattias@msig:~/repos/tikv$ time make clean cargo clean rm -rf bin dist real 0m0,975s user 0m0,148s sys 0m0,828s mattias@msig:~/repos/tikv$ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 00s real 4m0,201s user 39m45,037s sys 3m16,397s ``` Macbook Air M1: ``` # Before: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 01s make build 1107.42s user 116.20s system 506% cpu 4:01.46 total # After: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 10s make build 1179.39s user 120.74s system 518% cpu 4:10.98 total ``` Side effects - Performance regression, Only when building, to the benefit of full debug info by default. ### Release note  ```release-note None Signed-off-by: Mattias Jonsson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * correct a metric about compaction filter (#11938) correct a metric about compaction filter Signed-off-by: qupeng <[email protected]> * raft: Fix possible panic on entries fetched callback (#11961) close tikv/tikv#11951 Signed-off-by: Connor1996 <[email protected]> * split_controller: refine the sample function and add some comments (#11952) Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * metrics: fix grid position in TiKV Details (#11936) ref tikv/tikv#11119 Fix grid positions in TiKV Details dashboard. Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * cdc: capture old value from txn layer dynamically (#11896) close tikv/tikv#10091 Signed-off-by: hi-rustin <[email protected]> Signed-off-by: qupeng <[email protected]> Co-authored-by: hi-rustin <[email protected]> * split_controller: reorganize the structs and add more comments (#11967) Reorganize the structs and add more comments. Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support Bit column push downBit column (#11968) close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884 Signed-off-by: xiejiandong <[email protected]> * copr: support push down mod, sysdate to TiKV (#11970) close tikv/tikv#11916, ref tikv/tikv#11916 Signed-off-by: xiejiandong <[email protected]> * raftstore: propose in-memory pessimistic locks before prepare merge (#11758) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> * stats_monitor: reformat stats_monitor tick (#11972) Signed-off-by: lhy1024 <[email protected]> * copr: support extract scalar value from bit (#11980) close tikv/tikv#11893 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * diagnosis: support get cpu time of threads for macOS (#11978) close tikv/tikv#11977 Add the variant for macOS into `components/tikv_util/src/sys/thread.rs` Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Jay <[email protected]> * rust: update toolchain to fix missing rls (#11954) close tikv/tikv#11953 Signed-off-by: Connor1996 <[email protected]> Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Yilin Chen <[email protected]> * cdc: separate resolved region outliers (#11991) Separate broadcasing outlier regions and normal regions, so 1) downstreams know where they should send resolve lock requests, and 2) resolved ts of normal regions does not fallback. close pingcap/tiflow#4516 close pingcap/tiflow#4311 ref pingcap/tiflow#4146 Signed-off-by: Neil Shen <[email protected]> * *: collect key ranges for the read request in ResourceMeteringTag (#11995) close tikv/tikv#11988 *: collect key ranges for the read request in ResourceMeteringTag Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: move sample_threshold check out of the split_key iteration (#11986) split_controller: move sample_threshold check out of the split_key iteration Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support insertutf8 and lowerutf8 (#11987) ref tikv/tikv#5751 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: support rest greatest/least functions (#12003) ref tikv/tikv#5751 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: force compact with gentleness (#12006) * force compact with gentleness Signed-off-by: tabokie <[email protected]> * address comment Signed-off-by: tabokie <[email protected]> * split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010) split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: fix the incorrect sampled key ranges number check (#12013) close tikv/tikv#12012 split_controller: fix the incorrect sampled key ranges number check Signed-off-by: JmPotato <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: Support adjusting max_inflight_msgs dynamically (#11866) close tikv/tikv#11865 Signed-off-by: Wenbo Zhang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: fix missing flag solvement for bit column in decoding (#12016) ref tikv/tikv#7, close pingcap/tidb#32506 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * file_system: overhaul proc-based io tracing utilities (#11873) ref tikv/tikv#10867, fix tikv/tikv#11775 Signed-off-by: tabokie <[email protected]> * raftstore: Wait for the apply index equals to the commit index (#11716) ref tikv/tikv#10483 Signed-off-by: v01dstar <[email protected]> * copr: revert 11968/11980/12016 (#12030) ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506 Signed-off-by: yisaer <[email protected]> * build: fix a Mac build issue (#12027) ref tikv/tikv#10867, ref tikv/tikv#11873 Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: Extra physical table id column (#11931) close tikv/tikv#11888 Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer. Signed-off-by: Mattias Jonsson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: hotfix panic from tokio-timer (#12004) close tikv/tikv#11940, ref tikv/tikv#11940 *: hotfix panic from tokio-timer by enlarging the length of the level vector Signed-off-by: you06 <[email protected]> * build: add arm64 check on SSE in makefile (#12035) close tikv/tikv#12034 Signed-off-by: Jin Dong <[email protected]> Co-authored-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * replication mode: sync state to pd (#11751) Signed-off-by: disksing <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> * copr: support push bit column down (#12037) ref pingcap/tidb#30738 Signed-off-by: yisaer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * split_controller: add more sample function test cases (#12058) split_controller: add more sample function test cases Signed-off-by: JmPotato <[email protected]> * copr: fix greatest/least time/date args type (#12056) Signed-off-by: yisaer <[email protected]> * cdc: advancing resolved ts correctly for clusters with tiflash (#12050) close pingcap/tiflow#4461 cdc: advancing resolved ts correctly for clusters with tiflash Signed-off-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: bump master branch version to v6.0.0-alpha (#12077) close tikv/tikv#12075 Signed-off-by: Yilin Chen <[email protected]> * raftstore: check uninitialized destroy for merge (#12055) close tikv/tikv#12048 When a peer is destroyed without being initialized, it will store itself to peer list and the region epoch is missing. In merge if such state is detected, it should abort merging instead of panicking. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: Add TruncateManager and TruncateWorker (#11553) ref tikv/rfcs#81 Signed-off-by: longfangsong <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * TiKV supports buckets (#11763) ref tikv/tikv#11759 Signed-off-by: qi.xu <[email protected]> Co-authored-by: qi.xu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * api_version: Codec for RawKV key (#12036) ref tikv/tikv#11965 Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2. Signed-off-by: pingyu <[email protected]> Co-authored-by: Andy Lok <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: fix some typos (#12066) Signed-off-by: cuishuang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * raftstore: fix stale message cause panic (#12054) close tikv/tikv#12023 raftstore: fix stale message cause panic Signed-off-by: linning <[email protected]> * raftstore: ignore async fetch result when the peer is pending removed (#12038) close tikv/tikv#11973, close tikv/tikv#12026 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: remove part about wechat in doc (#12101) close tikv/tikv#12100 Signed-off-by: jackwener <[email protected]> * engine: upgrade raft engine (#12095) ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119 Signed-off-by: Randy <[email protected]> Co-authored-by: Xinye Tao <[email protected]> * *: Fix possible undefined behavior for transmuting vec (#12096) close tikv/tikv#12070 Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * github: Remove the pingcap/tidb-ansible from the PR template (#12102) close tikv/tikv#12103 Signed-off-by: hi-rustin <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * copr: extract reschedule to inject a fail point (#12108) Signed-off-by: Zhenchi <[email protected]> * raftstore: reactivate in-memory pessimistic locking when leader transfer fails (#11883) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> * *: check memory locks for replica read only on the leader (#12115) close tikv/tikv#12109 Consider a reader sends read index to the leader it supposed to be, but when the leader receives the read index message, it has stepped down to a follower. Without this commit, a peer will check the locks and re-fill the read index context with the result no matter what role it is. So, when the read index request is redirected to the new leader again, it no longer carries the context of lock checking. This commmit changes to only do the check when the peer is a leader. Then, the read index request will remain unchanged before being redirected to the leader. If the lease of the leader expires, it is still safe to check on the uncertained leader. If the heartbeat check passes and no new leader is elected, then the check works fine. If there is a new leader, when the old leader becomes a follower, it will clear its pending read index. Then, the reader has to resend th…

longfangsong force-pushed the patch-1 branch from 1217a83 to d1c8eb9 Compare November 26, 2021 01:53

longfangsong changed the title ~~Create 0081-flashback.md~~ RFC: Return TiKV to some certain version Nov 26, 2021

longfangsong mentioned this pull request Nov 26, 2021

Add reset to version rpc pingcap/kvproto#837

Merged

Create 0081-flashback.md

40ac22e

Add flashback Signed-off-by: longfangsong <[email protected]>

longfangsong force-pushed the patch-1 branch from d1c8eb9 to 40ac22e Compare November 26, 2021 02:28

cfzjywxk reviewed Nov 26, 2021

View reviewed changes

text/0081-flashback.md Outdated Show resolved Hide resolved

nolouch reviewed Nov 26, 2021

View reviewed changes

longfangsong added a commit to longfangsong/tikv that referenced this pull request Dec 3, 2021

init TruncateManager and TruncateWorker

987915d

ref tikv/rfcs#81

longfangsong mentioned this pull request Dec 3, 2021

*: Add TruncateManager and TruncateWorker tikv/tikv#11553

Merged

rename flashback into truncate

9dc2a77

Signed-off-by: longfangsong <[email protected]>

longfangsong added a commit to longfangsong/tikv that referenced this pull request Dec 3, 2021

init TruncateManager and TruncateWorker

2898db5

ref tikv/rfcs#81 Signed-off-by: longfangsong <[email protected]>

longfangsong changed the title ~~RFC: Return TiKV to some certain version~~ RFC: Return TiKV to some certain mvcc snapshot Dec 3, 2021

longfangsong mentioned this pull request Dec 3, 2021

Tracking issue for return TiKV to some certain mvcc snapshot tikv/tikv#11555

Open

3 tasks

longfangsong added a commit to longfangsong/tikv that referenced this pull request Dec 3, 2021

init TruncateManager and TruncateWorker

1625187

ref tikv/rfcs#81

youjiali1995 reviewed Dec 6, 2021

View reviewed changes

Update text/0081-truncate.md

a2cc55c

Co-authored-by: Lei Zhao <[email protected]> Update text/0081-truncate.md Co-authored-by: Lei Zhao <[email protected]> fix indent Signed-off-by: longfangsong <[email protected]>

longfangsong force-pushed the patch-1 branch from 382a10d to a2cc55c Compare December 6, 2021 03:47

add some note about no more write requests should be sent to TiKV dur…

95b955c

…ing the truncate process Signed-off-by: longfangsong <[email protected]>

sticnarf reviewed Dec 7, 2021

View reviewed changes

text/0081-truncate.md Outdated Show resolved Hide resolved

text/0081-truncate.md Outdated Show resolved Hide resolved

BusyJay changed the title ~~RFC: Return TiKV to some certain mvcc snapshot~~ RFC: reset TiKV to some certain mvcc snapshot Dec 7, 2021

address comments

a5c7eb9

Signed-off-by: longfangsong <[email protected]>

longfangsong force-pushed the patch-1 branch from e580ded to a5c7eb9 Compare December 10, 2021 02:11

ti-chi-bot added a commit to tikv/tikv that referenced this pull request Mar 3, 2022

*: Add TruncateManager and TruncateWorker (#11553)

dbf58ac

ref tikv/rfcs#81 Signed-off-by: longfangsong <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

This was referenced Jun 15, 2022

add reset_to_version command back tikv/tikv#12823

Merged

tikv-ctl: Add reset_to_version command back tikv/tikv#12824

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: reset TiKV to some certain mvcc snapshot #81

RFC: reset TiKV to some certain mvcc snapshot #81

longfangsong commented Nov 26, 2021

BusyJay commented Nov 26, 2021

nolouch Nov 26, 2021 •

edited

Loading

nolouch Nov 26, 2021 •

edited

Loading

youjiali1995 Dec 6, 2021


		## Drawbacks

		This is a very unsafe operation, we should prevent the user call this on normal cluster.


		### 1. Wait all committed raft log are applied

		We can just polling each region's `RegionInfo` in a method similar with [`Debugger::region_info`](https://github.com/tikv/tikv/blob/789c99666f2f9faaa6c6e5b021ac0cf7a76ae24e/src/server/debug.rs#L188) does, and wait for `commit_index` to be equal to `apply_index`.


		Then the sync state of the cluster (from the DR DC's PD) is not dependable. In this case, data can be partially sent to the TiKV nodes in DR, and break the ACID consistency.

		Though it is impossible to do lossless recovery in this situation, we can recover the data on TiKV nodes to a certain version (`resolved_ts` in this case), which is ACID consistency.

RFC: reset TiKV to some certain mvcc snapshot #81

Are you sure you want to change the base?

RFC: reset TiKV to some certain mvcc snapshot #81

Conversation

longfangsong commented Nov 26, 2021

BusyJay commented Nov 26, 2021

nolouch Nov 26, 2021 • edited Loading

Choose a reason for hiding this comment

nolouch Nov 26, 2021 • edited Loading

Choose a reason for hiding this comment

youjiali1995 Dec 6, 2021

Choose a reason for hiding this comment

nolouch Nov 26, 2021 •

edited

Loading

nolouch Nov 26, 2021 •

edited

Loading