Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release 6.5 #36

Closed
wants to merge 69 commits into from
Closed

Release 6.5 #36

wants to merge 69 commits into from

Conversation

lhy1024
Copy link
Owner

@lhy1024 lhy1024 commented Jan 29, 2024

What problem does this PR solve?

Issue Number: Close #xxx

What is changed and how does it work?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Code changes

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

Release note

None.

ti-chi-bot and others added 30 commits December 5, 2022 19:52
@ti-chi-bot @Connor1996
Fix unsafe recovery auto detect mode (tikv#5754) (tikv#5756)
5d6c912 
close tikv#5753, ref tikv#5754

Fix unsafe recovery auto-detect mode by passing detected failed stores to TiKV

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: Connor1996 <[email protected]>
@ti-chi-bot @rleungx
*: fix AtomicCheckAndPutRegion deadlock (tikv#5758) (tikv#5759)
d1a4433 
ref tikv#5755, close tikv#5757, ref tikv#5758

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
@ti-chi-bot @HunDunDM
checker: when there are too many orphan peers, try to delete them (ti…
979c460 
…kv#5787) (tikv#5818)

ref tikv#5786, ref tikv#5787

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: HunDunDM <[email protected]>
@ti-chi-bot @bufferflies
core: scoreV2 conside the extra file (tikv#5819) (tikv#5842)
d1c0d84 
Signed-off-by: bufferflies <[email protected]>

Signed-off-by: bufferflies <[email protected]>
Co-authored-by: bufferflies <[email protected]>
@ti-chi-bot @HunDunDM
operator: When replacing-down-peer, it should not stop at the step of…
5ade6b9 
… leave-joint-state (tikv#5789) (tikv#5795)

ref tikv#5568, close tikv#5788, ref tikv#5789

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: HunDunDM <[email protected]>
@ti-chi-bot @HunDunDM
region_scatterer: fix the bug that could generate schedule with too m…
b04f0a9 
…any peers (tikv#5920) (tikv#5923)

ref tikv#4570, close tikv#5909, ref tikv#5920

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: HunDunDM <[email protected]>
@ti-chi-bot @HunDunDM
cluster: no more redundant etcd writes when ReportMinResolvedTS is …
d34913b 
…called (tikv#5966) (tikv#5979)

close tikv#5965, ref tikv#5966

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: HunDunDM <[email protected]>
@ti-chi-bot @nolouch
dashboard: default close telemetry (tikv#5974) (tikv#5983)
1985ea3 
close tikv#5973, ref tikv#5974

dashboard: default close telemetry

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>

Co-authored-by: ShuNing <[email protected]>
Co-authored-by: nolouch <[email protected]>
@ti-chi-bot @HuSharp
metrics/grafana: fix Error showing "Current ID assignment" (tikv#5899) (
f07f6ff 
tikv#6016)

close tikv#5898, ref tikv#5899

The old version of idalloc is a globally unique path, and keyspace needs to use id allocator to allocate keyspaceID, so in this pr the id allocator is changed to a general purpose, but not sync metrics.
tikv#5284

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
@ti-chi-bot @HunDunDM
region_scatter: fix a bug that the leaders may be unbalanced after sc…
e4a9312 
…atter region (tikv#6054) (tikv#6061)

close tikv#6017, ref tikv#6054

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: HunDunDM <[email protected]>
@nolouch
*: upgrade the dependencies (tikv#6082)
297cee8 
close tikv#6081

*: upgrade the dependencies

Signed-off-by: nolouch <[email protected]>
@ti-chi-bot @lhy1024
moving_filter: fix data race with cache result (tikv#6080) (tikv#6087)
5a6f460 
ref tikv#5798, close tikv#6069, ref tikv#6080

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
@ti-chi-bot @rleungx
*: fix member race problem (tikv#6070) (tikv#6091)
51f382c 
ref tikv#5310, ref tikv#6069, ref tikv#6070

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
@ti-chi-bot @HunDunDM
region_scatter: fix a bug that PD may panic when scatter region if re…
bd80719 
…gion is unhealthy (tikv#6128) (tikv#6129)

close tikv#6099, ref tikv#6128

Signed-off-by: HunDunDM <[email protected]>

Co-authored-by: HunDunDM <[email protected]>
@ti-chi-bot @bufferflies
rule: it can transfer leader between the different rule (tikv#6200) (t…
7910e2c 
…ikv#6202)

close tikv#6195, ref tikv#6200

1. role can be transfer when their's rule role is same.

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: bufferflies <[email protected]>

Co-authored-by: buffer <[email protected]>
Co-authored-by: bufferflies <[email protected]>
@ti-chi-bot @Connor1996
online recovery: fix online recovery timeout mechanism (tikv#6108) (t…
206d412 
…ikv#6112)

close tikv#6107, ref tikv#6108

fix online recovery timeout mechanism

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: Connor1996 <[email protected]>
@Lloyd-Pottiger @ti-chi-bot
[cherry-pick 6327] server: add accelerate-schedule/batch api (tikv#6348)
10b691a 
* server: add accelerate-schedule/batch api

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

* address comments

Signed-off-by: Lloyd-Pottiger <[email protected]>

---------

Signed-off-by: Lloyd-Pottiger <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>
@ti-chi-bot @HuSharp @ti-chi-bot
*: add defer to logs the panic reason and stack (tikv#6123) (tikv#6154)
679c2cc 
ref tikv#6099, ref tikv#6123

add defer logs the panic reason and stack

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: Hu# <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @JmPotato
cluster: fix the merging issue of labels after store reboot (tikv#6468)…
c7db3b3 
… (tikv#6469)

ref tikv#5510, close tikv#6467, ref tikv#6468

tikv#5510 introduced a bug that would cause the store labels to be overwritten wrongly after the store reboot.
This PR fixed this issue.

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: JmPotato <[email protected]>

Co-authored-by: JmPotato <[email protected]>
@ti-chi-bot @nolouch @ti-chi-bot
server: fix the leader cannot election after pd leader lost while etc…
91efbda 
…d leader intact (tikv#6447) (tikv#6460)

close tikv#6403, ref tikv#6447

server: fix the leader cannot election after pd leader lost while etcd leader intact

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>

Co-authored-by: ShuNing <[email protected]>
Co-authored-by: nolouch <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @rleungx @ti-chi-bot
*: use another etcd client for election (tikv#6409) (tikv#6454)
d086bd5 
ref tikv#6403, ref tikv#6409

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @nolouch @ti-chi-bot
replication_mode: fix the state cannot switch to async while existing…
cb52d28 
… learner node (tikv#6452) (tikv#6483)

ref tikv#4399, ref tikv#6452, close tikv/tikv#14704

replication_mode: fix the state cannot switch to async while existing learner node
 - skip the learner node when check the stores state

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>

Co-authored-by: ShuNing <[email protected]>
Co-authored-by: nolouch <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @nolouch
@rleungx @ti-chi-bot
checker: fix unhealth region skip the rule check (tikv#6427) (tikv#6430)
08b8fc8 
close tikv#6426, ref tikv#6427

allow the `schedule=deny` label can do rule constraints check

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>
Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ShuNing <[email protected]>
Co-authored-by: nolouch <[email protected]>
Co-authored-by: Ryan Leung <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@baurine
chore: update tidb-dashboard version to v2023.05.08.1 (tikv#6516)
801601f 
fix pingcap/tidb-dashboard#1515, close tikv#6520

Update tidb-dashboard to the lastest version (v2023.05.08.1)

Signed-off-by: baurine <[email protected]>
@ti-chi-bot @HuSharp
minResolvedTS: Provide an API to fetch the store resolved_ts (tikv#6413…
8e9d0c4 
…) (tikv#6522)

close tikv#6386, ref tikv#6413

Provide an API to fetch the store resolved_ts

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: Hu# <[email protected]>
@ti-chi-bot @HuSharp
config, cluster: add an option to halt the cluster scheduling (tikv#6498
410b176 
) (tikv#6558)

ref tikv#6493, ref tikv#6498

Add an option to halt the cluster scheduling.

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
@baurine
chore: update tidb-dashboard version to v2023.06.07.1 (tikv#6566)
9d320ef 
close tikv#6564

Update tidb-dashboard to the lastest version (v2023.06.07.1)

Signed-off-by: baurine <[email protected]>
@ti-chi-bot @nolouch
checker: fix the too many orphan peers cannot be removed (tikv#6574) (t…
898dde2 
…ikv#6575)

close tikv#6573, ref tikv#6574

rule-checker: fix the too many orphan peers that cannot be removed
- let the health peer can be removed once there exist redundant

Signed-off-by: nolouch <[email protected]>

Co-authored-by: nolouch <[email protected]>
@ti-chi-bot @lhy1024
leadership: avoid potential data race (tikv#6636) (tikv#6702)
cfdcdff 
close tikv#6635, ref tikv#6636

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
@v01dstar @ti-chi-bot
unsafe recovery: Fixing learner store being ignored error (tikv#6683)
15324c5 
ref tikv#6690, ref tikv#6709

Fix learner store/ replica being ignored error in auto detect mode.

Signed-off-by: Yang Zhang <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
baurine and others added 28 commits August 18, 2023 02:24
@baurine
chore: update tidb-dashboard to v2023.08.15.1 (tikv#6967)
4441cd8 
close tikv#6966

1. This PR fixed the statements list export failed issue
2. This PR support config the ngm component timeout param

Signed-off-by: baurine <[email protected]>
@ti-chi-bot @overvenus
pkg/core, cluster: save kv and cache if region flashback (tikv#6911) (t…
cd4eb78 
…ikv#6948)

ref tikv#6912, close tikv/tikv#15258

TiKV Region merge checks region in operator and in local strictly,
and drops commit merge if there is a mismatch between them.

This change is necessary to prevent flashback from blocking
Region merge. For more details, see tikv/tikv#15258.

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Neil Shen <[email protected]>
@HuSharp
region: remove error return logic (tikv#6982)
75bb796 
ref tikv#6920

Signed-off-by: husharp <[email protected]>
@HuSharp
client: support backoff mechanism for memberLoop (tikv#6978)
71e8929 
ref tikv#6556

Signed-off-by: husharp <[email protected]>
@lance6716
client: backport ErrClientTSOStreamClosed (tikv#7021)
947701a 
ref pingcap/tidb#46540

Signed-off-by: lance6716 <[email protected]>
@ti-chi-bot @HuSharp
*: check raftcluster nil (tikv#7054) (tikv#7070)
094ce3c 
close tikv#7053

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
@ti-chi-bot @bufferflies @ti-chi-bot
Scatter: make peer scatter logic same with the leader (tikv#6965) (ti…
11f71ee 
…kv#7027)

close tikv#6962

In past, PD conside peer distribution in the different group influenece by using `TotalCountByStore` , but not include the leader distribution. The max used situation is partition table. 
After this pr, TIDB call scatter api will use same group not different. 

ref:
tikv#3422
pingcap/tidb#46156

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: bufferflies <[email protected]>

Co-authored-by: buffer <[email protected]>
Co-authored-by: bufferflies <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @CabinfeverB
statistics: fix empty region count when resuming (tikv#7009) (tikv#7055)
8c5859e 
close tikv#7008

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: Cabinfever_B <[email protected]>

Co-authored-by: Yongbo Jiang <[email protected]>
Co-authored-by: Cabinfever_B <[email protected]>
@baurine @ti-chi-bot
chore(dashboard): update version to v2023.09.11.1 (tikv#7065)
e8f95ac 
close tikv#7066

1. security: improve tidb-dashboard login security, encrypt the login password to avoid transport plain text (required for 6.5 version)
2. debug-api: support pagination for ddl history for debug-api
3. keyvisual: use scanRegions instead of fetch all regions (required for 6.5 version)
4. refine execution plan, now it can be showed as table style

Signed-off-by: baurine <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@baurine
chore: update tidb-dashboard to v2023.09.21.1 (tikv#7136)
9b04f25 
close tikv#7137

Signed-off-by: baurine <[email protected]>
@ti-chi-bot @HuSharp
security: disable plugin in default and persist file in specified dir (…
5b491e2 
…tikv#7087) (tikv#7141)

close tikv#7094

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: Hu# <[email protected]>
@CabinfeverB
Revert "statistics: fix empty region count when resuming (tikv#7009) (t…
b0207a6 
…ikv#70… (tikv#7150)

ref tikv#7148

Signed-off-by: Cabinfever_B <[email protected]>
@ti-chi-bot @disksing
dr-autosync: add metrics (tikv#7110) (tikv#7153)
ed629e9 
close tikv#6975

Signed-off-by: disksing <[email protected]>

Co-authored-by: disksing <[email protected]>
@ti-chi-bot @CabinfeverB @ti-chi-bot
scatter: fix incorrect judgment condition (tikv#7111) (tikv#7118)
d499de1 
close tikv#7109

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: Cabinfever_B <[email protected]>

Co-authored-by: Yongbo Jiang <[email protected]>
Co-authored-by: Cabinfever_B <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @HuSharp @ti-chi-bot
scheduler: fix scheduler save config (tikv#7108) (tikv#7162)
77d6f5b 
close tikv#6897

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @disksing
replication_mode: use placement to determin canSync and hasMajority (t…
ba09621 
…ikv#7202) (tikv#7209)

close tikv#7201

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: disksing <[email protected]>

Co-authored-by: disksing <[email protected]>
@ti-chi-bot @disksing
replication mode: fix wrong available store list (tikv#7222) (tikv#7224)
e0ab17c 
close tikv#7221

Signed-off-by: disksing <[email protected]>

Co-authored-by: disksing <[email protected]>
@ti-chi-bot @rleungx @nolouch
core: batch get region size (tikv#7252) (tikv#7329)
87c31a9 
close tikv#7248

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: nolouch <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
Co-authored-by: nolouch <[email protected]>
@ti-chi-bot @rleungx @ti-chi-bot
*: fix sync isolation level to default placement rule (tikv#7122) (ti…
7d0708f 
…kv#7126)

close tikv#7121

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@lhy1024 @ti-chi-bot
checker: reduces the probability of deleting normal peers when the st…
5a95241 
…ore becomes unavailable (tikv#7249) (tikv#7337)

close tikv#7249

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @HuSharp @ti-chi-bot
coordinator: use a healthy region count to start coordinator (tikv#7044…
d016fed 
…) (tikv#7363)

close tikv#6988, close tikv#7016

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: Hu# <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @rleungx @HuSharp
schedule: fix panic when switching placement rules (tikv#7415) (tikv#…
dbce3fb 
…7423)

close tikv#7414

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
Co-authored-by: Hu# <[email protected]>
@ti-chi-bot @niubell @rleungx
go.mod: upgrade gin version from v1.8.1 to v1.9.1 (tikv#7451) (tikv#7459
ba7f2fe 
)

close tikv#7438

Upgrade Gin version to v1.9.1 to solve some security issues

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: niubell <[email protected]>
Co-authored-by: Ryan Leung <[email protected]>
@ti-chi-bot @mornyx
dashboard: update version to v20231127.1 (tikv#7458) (tikv#7460)
54bdbc3 
close tikv#7457

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Yexiang Zhang <[email protected]>
@ti-chi-bot @rleungx @ti-chi-bot
schedule: fix location label in placement rule not work well with rul…
b0b1e38 
…e checker (tikv#6660) (tikv#6956)

close tikv#6637

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot @bufferflies
check: remove orphan peer only when the peers is greater than the rul…
542fde9 
…e count (tikv#7581) (tikv#7591)

close tikv#7584

The healthy orphan peer should be the last one to be removed only if there are extra peers to keep the high availablility.

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: bufferflies <[email protected]>

Co-authored-by: tongjian <[email protected]>
Co-authored-by: bufferflies <[email protected]>
@ti-chi-bot @CabinfeverB
scheduler: add aduit log for scheduler config API and add resp msg fo…
1d3d5d0 
…r evict-leader (tikv#7674) (tikv#7686)

close tikv#7672

Signed-off-by: ti-chi-bot <[email protected]>
Signed-off-by: Cabinfever_B <[email protected]>

Co-authored-by: Yongbo Jiang <[email protected]>
Co-authored-by: Cabinfever_B <[email protected]>
@ti-chi-bot @rleungx
api: fix panic when region doesn't have a leader (tikv#7629) (tikv#7651)
4506d63 
close tikv#7630

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ryan Leung <[email protected]>
@codecov-commenter
Copy link

Codecov Report

Attention: 195 lines in your changes are missing coverage. Please review.

Comparison is base (b3eeaae) 75.52% compared to head (4506d63) 75.50%.
Report is 1179 commits behind head on master.

Files Patch % Lines
client/client.go 40.74% 39 Missing and 9 partials ⚠️
server/api/region.go 71.31% 26 Missing and 11 partials ⚠️
client/keyspace_client.go 62.22% 13 Missing and 4 partials ⚠️
server/api/admin.go 64.58% 11 Missing and 6 partials ⚠️
pkg/btree/btree_generic.go 86.66% 11 Missing and 3 partials ⚠️
server/api/min_resolved_ts.go 7.14% 13 Missing ⚠️
pkg/testutil/operator_check.go 86.76% 9 Missing ⚠️
pkg/jsonutil/jsonutil.go 73.33% 2 Missing and 2 partials ⚠️
pkg/syncutil/lock_group.go 82.60% 2 Missing and 2 partials ⚠️
server/api/middleware.go 77.77% 2 Missing and 2 partials ⚠️
... and 13 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master      #36      +/-   ##
==========================================
- Coverage   75.52%   75.50%   -0.02%     
==========================================
  Files         307      332      +25     
  Lines       30233    33775    +3542     
==========================================
+ Hits        22834    25503    +2669     
- Misses       5404     6068     +664     
- Partials     1995     2204     +209
Flag Coverage Δ
unittests 75.50% <77.05%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@lhy1024 lhy1024 closed this Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.