| title | summary | aliases | |
|---|---|---|---|
Backup & Restore FAQ |
Learn about Frequently Asked Questions (FAQ) and the solutions of BR. |
|
This document lists the frequently asked questions (FAQs) and the solutions about Backup & Restore (BR).
What should I do if the error message could not read local://...:download sst failed is returned during data restoration?
When you restore data, each node must have access to all backup files (SST files). By default, if local storage is used, you cannot restore data because the backup files are scattered among different nodes. Therefore, you have to copy the backup file of each TiKV node to the other TiKV nodes.
It is recommended to mount an NFS disk as a backup disk during backup. For details, see Back up a single table to a network disk.
When you use the oltp_read_only scenario of sysbench to back up to a disk (make sure the backup disk and the service disk are different) at full rate, the cluster QPS is decreased by 15%-25%. The impact on the cluster depends on the table schema.
To reduce the impact on the cluster, you can use the --ratelimit parameter to limit the backup rate.
Before v5.1.0, BR filtered out data from the system schema mysql during the backup. Since v5.1.0, BR backs up all data by default, including the system schemas mysql.*. But the technical implementation of restoring the system tables in mysql.* is not complete yet, so the tables in the system schema mysql are not restored by default. For more details, refer to the Back up and restore table data in the mysql system schema (experimental feature).
What should I do to handle the Permission denied or No such file or directory error, even if I have tried to run BR using root in vain?
You need to confirm whether TiKV has access to the backup directory. To back up data, confirm whether TiKV has the write permission. To restore data, confirm whether it has the read permission.
During the backup operation, if the storage medium is the local disk or a network file system (NFS), make sure that the user to start BR and the user to start TiKV are consistent (if BR and TiKV are on different machines, the users' UIDs must be consistent). Otherwise, the Permission denied issue might occur.
Running BR with the root access might fail due to the disk permission, because the backup files (SST files) are saved by TiKV.
Note:
You might encounter the same problem during data restoration. When the SST files are read for the first time, the read permission is verified. The execution duration of DDL suggests that there might be a long interval between checking the permission and running BR. You might receive the error message
Permission deniedafter waiting for a long time.Therefore, it is recommended to check the permission before data restore according to the following steps:
-
Run the Linux-native command for process query:
{{< copyable "shell-regular" >}}
ps aux | grep tikv-serverThe output of the above command:
tidb_ouo 9235 10.9 3.8 2019248 622776 ? Ssl 08:28 1:12 bin/tikv-server --addr 0.0.0.0:20162 --advertise-addr 172.16.6.118:20162 --status-addr 0.0.0.0:20188 --advertise-status-addr 172.16.6.118:20188 --pd 172.16.6.118:2379 --data-dir /home/user1/tidb-data/tikv-20162 --config conf/tikv.toml --log-file /home/user1/tidb-deploy/tikv-20162/log/tikv.log tidb_ouo 9236 9.8 3.8 2048940 631136 ? Ssl 08:28 1:05 bin/tikv-server --addr 0.0.0.0:20161 --advertise-addr 172.16.6.118:20161 --status-addr 0.0.0.0:20189 --advertise-status-addr 172.16.6.118:20189 --pd 172.16.6.118:2379 --data-dir /home/user1/tidb-data/tikv-20161 --config conf/tikv.toml --log-file /home/user1/tidb-deploy/tikv-20161/log/tikv.log
Or you can run the following command:
{{< copyable "shell-regular" >}}
ps aux | grep tikv-server | awk '{print $1}'
The output of the above command:
tidb_ouo tidb_ouo
-
Query the startup information of the cluster using the TiUP command:
{{< copyable "shell-regular" >}}
tiup cluster list
The output of the above command:
[root@Copy-of-VM-EE-CentOS76-v1 br]# tiup cluster list Starting component `cluster`: /root/.tiup/components/cluster/v1.5.2/tiup-cluster list Name User Version Path PrivateKey ---- ---- ------- ---- ---------- tidb_cluster tidb_ouo v5.0.2 /root/.tiup/storage/cluster/clusters/tidb_cluster /root/.tiup/storage/cluster/clusters/tidb_cluster/ssh/id_rsa
-
Check the permission for the backup directory. For example,
backupis for backup data storage:{{< copyable "shell-regular" >}}
ls -al backup
The output of the above command:
[root@Copy-of-VM-EE-CentOS76-v1 user1]# ls -al backup total 0 drwxr-xr-x 2 root root 6 Jun 28 17:48 . drwxr-xr-x 11 root root 310 Jul 4 10:35 ..
From the above output, you can find that the
tikv-serverinstance is started by the usertidb_ouo. But the usertidb_ouodoes not have the write permission forbackup, the backup fails.
Almost all of these problems are system call errors that occur when TiKV writes data to the disk. For example, if you encounter error messages such as Io(Os {code: 13, kind: PermissionDenied...}) or Io(Os {code: 2, kind: NotFound...}), you can first check the mounting method and the file system of the backup directory, and try to back up data to another folder or another hard disk.
For example, you might encounter the Code: 22(invalid argument) error when backing up data to the network disk built by samba.
This error might occur when the capacity of the cluster to restore (using BR) is insufficient. You can further confirm the cause by checking the monitoring metrics of this cluster or the TiKV log.
To handle this issue, you can try to scale out the cluster resources, reduce the concurrency during restore, and enable the RATE_LIMIT option.
When you use local storage, backupmeta is generated on the node where BR is running, and backup files are generated on the Leader nodes of each Region.
During data backup, backup files are generated on the Leader nodes of each Region. The size of the backup is equal to the data size, with no redundant replicas. Therefore, the total data size is approximately the total number of TiKV data divided by the number of replicas.
However, if you want to restore data from local storage, the number of replicas is equal to that of the TiKV nodes, because each TiKV must have access to all backup files.
-
The data restored using BR cannot be replicated to the downstream. This is because BR directly imports SST files but the downstream cluster currently cannot obtain these files from the upstream.
-
Before v4.0.3, DDL jobs generated during the BR restore might cause unexpected DDL executions in TiCDC/Drainer. Therefore, if you need to perform restore on the upstream cluster of TiCDC/Drainer, add all tables restored using BR to the TiCDC/Drainer block list.
You can use filter.rules to configure the block list for TiCDC and use syncer.ignore-table to configure the block list for Drainer.
Does BR back up the SHARD_ROW_ID_BITS and PRE_SPLIT_REGIONS information of a table? Does the restored table have multiple Regions?
Yes. BR backs up the SHARD_ROW_ID_BITS and PRE_SPLIT_REGIONS information of a table. The data of the restored table is also split into multiple Regions.
Why is the region is unavailable error reported for a SQL query after I use BR to restore the backup data?
If the cluster backed up using BR has TiFlash, TableInfo stores the TiFlash information when BR restores the backup data. If the cluster to be restored does not have TiFlash, the region is unavailable error is reported.
No. BR does not support in-place full recovery of some historical backup.
To get the commitTs field of the last BR backup, run the kubectl -n ${namespace} get bk ${name} command using kubectl. You can use the content of this field as --lastbackupts.
BR backupTS defaults to the latest timestamp obtained from PD before the backup starts. You can use pd-ctl tso timestamp to parse the timestamp to obtain an accurate value, or use backupTS >> 18 to quickly obtain an estimated value.
After BR restores the backup data, do I need to execute the ANALYZE statement on the table to update the statistics of TiDB on the tables and indexes?
BR does not back up statistics (except in v4.0.9). Therefore, after restoring the backup data, you need to manually execute ANALYZE TABLE or wait for TiDB to automatically execute ANALYZE.
In v4.0.9, BR backs up statistics by default, which consumes too much memory. To ensure that the backup process goes well, the backup for statistics is disabled by default starting from v4.0.10.
If you do not execute ANALYZE on the table, TiDB will fail to select the optimized execution plan due to inaccurate statistics. If query performance is not a key concern, you can ignore ANALYZE.
It is strongly not recommended to use multiple BR processes at the same time to restore the data of a single cluster for the following reasons:
- When BR restores data, it modifies some global configurations of PD. Therefore, if you use multiple BR processes for data restore at the same time, these configurations might be mistakenly overwritten and cause abnormal cluster status.
- BR consumes a lot of cluster resources to restore data, so in fact, running BR processes in parallel improves the restore speed only to a limited extent.
- There has been no test for running multiple BR processes in parallel for data restore, so it is not guaranteed to succeed.