[bridge-indexer] revamp task #19245
[bridge-indexer] revamp task #19245
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
longbowlu
requested review from
patrickkuo,
Bridgerz,
dariorussi and
0xaslan
| @@ -63,14 +64,13 @@ impl EthSubscriptionDatasource { | |||
| impl Datasource<RawEthData> for EthSubscriptionDatasource { | |||
| async fn start_data_retrieval( | |||
| &self, | |||
| starting_checkpoint: u64, | |||
| target_checkpoint: u64, | |||
| task: Task, | |||
| @@ -23,6 +23,9 @@ use tokio::task::JoinHandle; | |||
|
|
|||
| use crate::metrics::BridgeIndexerMetrics; | |||
|
|
|||
| const BACKFILL_TASK_INGESTION_READER_BATCH_SIZE: usize = 300; | |||
There was a problem hiding this comment.
I assume this is the value that defines how much is going to be read in one shot?
Can you add a comment as to what this mean?
Also is this something that should be defined in the config file besides and env variable?
There was a problem hiding this comment.
this is the fetching concurrency in ingestion framework. https://github.com/MystenLabs/sui/blob/main/crates/sui-data-ingestion-core/src/reader.rs#L207
it guarantees at most N checkpoints can be fetched at the same time. A recent change downgraded the default value to 10, which significantly bottlenecked the backfill speed. In backfill tasks we should use a larger number, in live sync 10 is fine.
will add in the comment
|
@longbowlu one final question, is this going to affect the deepbook indexer or is this all private implementation to the indexer "framework"? |
longbowlu
force-pushed
the
integrate-progress-saving-policy
branch
from
411ec7b
to
11f47cc
Compare
it will affect deepbook because deepbook is using this framework, right? If you are asking about if DB indexer need to change anything, i think the only thing to add is |
longbowlu
force-pushed
the
revamp-task
branch
from
279bbb1
to
dd0ff09
Compare
## Description This PR reworks `Tasks`: 1. get rid of trait `Tasks` and create struct `Tasks` instead. 2. add `is_live_task` field to `Task` 3. pass `Task` to several functions instead of its parameters. 4. for ingestion framework, use a custom batch read size for backfill tasks (this significantly improves the data download speed) ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
Description
This PR reworks
Tasks:Tasksand create structTasksinstead.is_live_taskfield toTaskTaskto several functions instead of its parameters.Test plan
How did you test the new or updated feature?
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.