Skip to content

Commit

Permalink
[Indexer] Support 2-day retention for objects_history table (#19318)
Browse files Browse the repository at this point in the history
## Description 

Add support to override epoch retention for individual tables. This can
be passed from the command line.
It only supports partitioned table today. For non-partitioned tables it
will take some wiring to make it work and not urgent. We need this today
mainly to prune the objects_history table more aggressively.

## Test plan 

CI

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
  • Loading branch information
lxfind authored Sep 25, 2024
1 parent 857de33 commit fd5a052
Showing 1 changed file with 13 additions and 3 deletions.
16 changes: 13 additions & 3 deletions crates/sui-indexer/src/handlers/pruner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ use crate::store::pg_partition_manager::PgPartitionManager;
use crate::store::PgIndexerStore;
use crate::{metrics::IndexerMetrics, store::IndexerStore, types::IndexerResult};

/// The primary purpose of objects_history is to serve consistency query.
/// A short retention is sufficient.
const OBJECTS_HISTORY_EPOCHS_TO_KEEP: u64 = 2;

pub struct Pruner {
pub store: PgIndexerStore,
pub partition_manager: PgPartitionManager,
Expand Down Expand Up @@ -58,16 +62,21 @@ impl Pruner {
})
.collect();

let prune_to_epoch = last_seen_max_epoch.saturating_sub(self.epochs_to_keep - 1);

for (table_name, (min_partition, max_partition)) in &table_partitions {
if last_seen_max_epoch != *max_partition {
error!(
"Epochs are out of sync for table {}: max_epoch={}, max_partition={}",
table_name, last_seen_max_epoch, max_partition
);
}
for epoch in *min_partition..prune_to_epoch {

let epochs_to_keep = if table_name == "objects_history" {
OBJECTS_HISTORY_EPOCHS_TO_KEEP
} else {
self.epochs_to_keep
};
for epoch in *min_partition..last_seen_max_epoch.saturating_sub(epochs_to_keep - 1)
{
if cancel.is_cancelled() {
info!("Pruner task cancelled.");
return Ok(());
Expand All @@ -82,6 +91,7 @@ impl Pruner {
}
}

let prune_to_epoch = last_seen_max_epoch.saturating_sub(self.epochs_to_keep - 1);
let prune_start_epoch = next_prune_epoch.unwrap_or(min_epoch);
for epoch in prune_start_epoch..prune_to_epoch {
if cancel.is_cancelled() {
Expand Down

0 comments on commit fd5a052

Please sign in to comment.