Indexer-worker sync problems

[Nebojitel]

Hello, i've pulled ton-indexer from master branch (including submodules). Started from 39900000 mc 26.08.24
Yesterday sync process was about 1500 blocks per hour:

Last: 39947201 / 39984233 11:30
Last: 39950901 / 39984610 13:44
Last: 39953817 / 39985136 16:00
Last: 39956578 / 39985577 18:05
Last: 39958574 / 39986240 20:00

But todays logs looks like that:

[ 3][t 3][2024-08-28 12:14:59.856775356][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975188 / 39990408       Blk/s: 0.553718 ETA: 7h 38m 6s  Q[1M, 1b, 3t, 1m]
[ 3][t 3][2024-08-28 12:15:24.676134869][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975192 / 39990408       Blk/s: 0.898346 ETA: 4h 42m 17s Q[1M, 4b, 491t, 889m]
[ 3][t 7][2024-08-28 12:15:24.941996104][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=4, txs=491, msgs=889, traces=0]
[ 2][t 3][2024-08-28 12:15:32.316475494][InterfaceDetectors.hpp:977][!nft_collection_detector]  Failed to verify belonging to collection for 0:2D4DAA78C3F4645781358A6B122A7190F11C6582C5E422D37E545BC35F3728E7: [Error : 504 : NFT Item doesn't belong to the referred collection]
[ 3][t 3][2024-08-28 12:15:32.805175010][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=1, txs=3, msgs=1, traces=0]
[ 3][t 5][2024-08-28 12:17:11.256822236][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975192 / 39990408       Blk/s: 0.654894 ETA: 6h 27m 14s Q[1M, 4b, 491t, 889m]
[ 3][t 5][2024-08-28 12:17:11.257444222][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=62, txs=14851, msgs=26827, traces=0]
[ 3][t 5][2024-08-28 12:17:11.257488636][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=3, b=136, txs=19551, msgs=35588, traces=0]
[ 3][t 6][2024-08-28 12:20:05.394812372][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975197 / 39990408       Blk/s: 0.889405 ETA: 4h 45m 2s  Q[1M, 1b, 3t, 1m]
[ 3][t 6][2024-08-28 12:20:31.199465103][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975199 / 39990478       Blk/s: 0.900418 ETA: 4h 42m 48s Q[0M, 0b, 0t, 0m]
[ 3][t 6][2024-08-28 12:20:54.662009230][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975199 / 39990480       Blk/s: 0.656405 ETA: 6h 27m 59s Q[6M, 110b, 20428t, 35012m]

[ 3][t 7][2024-08-28 14:50:57.129742574][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=1, txs=3, msgs=1, traces=0]
[ 3][t 4][2024-08-28 14:51:33.426276737][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975386 / 39991346       Blk/s: 0.822336 ETA: 5h 23m 28s Q[1M, 1b, 3t, 1m]
[ 3][t 4][2024-08-28 14:51:49.418129042][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=2, b=33, txs=4651, msgs=7483, traces=0]
[ 3][t 5][2024-08-28 14:52:01.808764487][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975386 / 39991350       Blk/s: 0.756092 ETA: 5h 51m 53s Q[2M, 33b, 4651t, 7483m]
[ 3][t 5][2024-08-28 14:52:02.691147538][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=4, b=84, txs=6859, msgs=10790, traces=0]
[ 3][t 2][2024-08-28 14:52:19.419413840][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=41, txs=3885, msgs=6168, traces=0]
[ 3][t 1][2024-08-28 14:52:52.266030492][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975386 / 39991351       Blk/s: 0.780483 ETA: 5h 40m 55s Q[2M, 33b, 4651t, 7483m]
[ 3][t 1][2024-08-28 14:53:04.744814161][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975388 / 39991359       Blk/s: 0.657875 ETA: 6h 44m 36s Q[0M, 0b, 0t, 0m]
[ 3][t 1][2024-08-28 14:53:09.546681128][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=3, b=41, txs=4869, msgs=7716, traces=0]
[ 3][t 6][2024-08-28 14:53:50.972652673][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975388 / 39991359       Blk/s: 0.532879 ETA: 8h 19m 31s Q[3M, 41b, 4869t, 7716m]
[ 3][t 6][2024-08-28 14:54:04.008805241][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975388 / 39991367       Blk/s: 0.679591 ETA: 6h 31m 52s Q[3M, 54b, 4902t, 8021m]
[ 3][t 3][2024-08-28 14:54:14.983695882][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=3, b=54, txs=4902, msgs=8021, traces=0]
[ 3][t 7][2024-08-28 14:54:15.026482165][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975388 / 39991367       Blk/s: 0.611632 ETA: 7h 15m 25s Q[3M, 54b, 4902t, 8021m]
[ 3][t 6][2024-08-28 14:54:22.194458483][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=101, txs=9116, msgs=15149, traces=0]
[ 3][t 6][2024-08-28 14:54:22.194551239][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=2, b=51, txs=4680, msgs=7689, traces=0]
[ 3][t 7][2024-08-28 14:54:26.934946646][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975393 / 39991373       Blk/s: 0.607880 ETA: 7h 18m 8s  Q[3M, 152b, 13796t, 22838m]
[ 3][t 7][2024-08-28 14:54:30.321311795][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=68, txs=8110, msgs=12887, traces=0]
[ 3][t 7][2024-08-28 14:54:30.321337764][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=1, txs=3, msgs=1, traces=0]
[ 3][t 4][2024-08-28 14:55:07.917581700][IndexScheduler.cpp:174][!indexscheduler]       Last: 39975393 / 39991375       Blk/s: 0.492382 ETA: 9h 0m 58s  Q[1M, 68b, 8110t, 12887m]
[ 3][t 4][2024-08-28 14:55:08.053709555][InsertManagerBase.cpp:81][!insertmanager]      Inserting batch[mb=1, b=38, txs=7697, msgs=12033, traces=0]

Seems like it will be unlimited sync process.

env file looks like that:

# TON Indexer config
POSTGRES_HOST=**.**.**.**
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD_FILE=private/postgres_password
POSTGRES_DBNAME=ton_index
POSTGRES_PUBLISH_PORT=5432

TON_INDEXER_API_ROOT_PATH=
TON_INDEXER_API_PORT=8081
TON_INDEXER_API_TITLE=TON Indexer
TON_INDEXER_WORKERS=4

TON_INDEXER_TON_HTTP_API_ENDPOINT=http://**.**.**.**/
TON_WORKER_DBROOT=/var/ton-work/db/
TON_WORKER_FROM=39900000
TON_WORKER_MAX_PARALLEL_TASKS=1024
TON_WORKER_INSERT_BATCH_SIZE=512
TON_WORKER_INSERT_PARALLEL_ACTORS=3

Any suggestions?

[ssnopfanker]

you need to run high IO performance RDS instance to solve sync slow.

[penghian]

@Nebojitel

TON_WORKER_MAX_PARALLEL_TASKS=1024
TON_WORKER_INSERT_BATCH_SIZE=512
TON_WORKER_INSERT_PARALLEL_ACTORS=3

these are not the current default variables right?
how do you add these to your node