Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

indexer-alt: index chain identifier #20148

Merged
merged 7 commits into from
Nov 4, 2024
Merged

indexer-alt: index chain identifier #20148

merged 7 commits into from
Nov 4, 2024

Commits on Nov 4, 2024

  1. indexer-alt: sum_displays pipeline 

    ## Description

Pipeline for indexing Display VersionUpdated events, so readers can
render Display for Objects.

## Test plan

Run the indexer on the first ~1.6M checkpoints (the first display events
happen around checkpoint 1.5M), and then inspect the database:

```
sui$ cargo run -p sui-indexer-alt --release --                                   \
  --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
  indexer --remote-store-url https://checkpoints.mainnet.sui.io                  \
  --consistent-range 3600
```

```
sui_indexer_alt=# SELECT object_type, display_id FROM sum_displays LIMIT 10;
                                                                                                              object_type                                                                                                               |                             display_id
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------
 \x72f9c76421170b5a797432ba9e1b3b2e2b7cf6faa26eb955396c773af2479e1e0967616d655f383139320847616d653831393200                                                                                                                             | \xb06e4607642f6c08e196f8bd84b653c784864a6badca2066e63db42e846fdd4c
 \x72ad8f46d4fc3bbc24df15f6c62a98044c2649a9a31895a957254b2cf16e0cb905646d656e7305446d656e7300                                                                                                                                           | \x831cd05a3de12ae3dcea4382e9f56eadded8efd83c742036acd789fc41f1a03f
 \x72ad8f46d4fc3bbc24df15f6c62a98044c2649a9a31895a957254b2cf16e0cb905646d656e7309446d656e734d65746100                                                                                                                                   | \x3e8acb139222c9e335980f8b427270e92d276152140545bfde46c700472a2e11
 \xee496a0cc04d06a345982ba6697c90c619020de9e274408c7819f787ff66e1a1087375696672656e73075375694672656e0107ee496a0cc04d06a345982ba6697c90c619020de9e274408c7819f787ff66e1a10463617079044361707900                                         | \x3d53effd472191435e7735ff9ed0b3083ef10dee4793db51df92038cea48b139
 \x5b45da03d42b064f5e051741b6fed3b29eb817c7923b83b92f37a1d2abf4fbab036e6674034e667400                                                                                                                                                   | \x23453afb61ba4e0c9f040839c0d620c2cab774491656c086e10871aae8f8b3e9
 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b0873686f77636173650853686f776361736500                                                                                                                               | \xa306db0c6d3f5cbb8169143c22da5b61caf3576f1b3f3e77537aef8d4ca4b515
 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b0473756961054d6564616c00                                                                                                                                             | \x70e764f567b976d8e654064c14a7fbd94f0994a229fb0a7e119d80b4f1e4b760
 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b04737569610d506572736f6e616c4d6564616c00                                                                                                                             | \xb438c4cbda18b0890de8ca2a1813be70fc3b64fedf171ea7c658d43605a11ece
 \xbb1531504c9c3235d3cd637ed9573cbe18461255b4175a1cb1e1b07b8aa8e11b08746573745f6e66740b53756961546573744e465400                                                                                                                         | \xa37afca51a40bc90e30991631515f2ab8b9ecef90e51fbf4af12aa0ea5413250
 \xf1e7b4093872df432b2adb50533a1fa6f9e4b20856217657242fe7d9d4ac42e80b6574686f735f73717561640a4574686f7353717561640107f1e7b4093872df432b2adb50533a1fa6f9e4b20856217657242fe7d9d4ac42e80b6574686f735f73717561640b4574686f7353717561643100 | \x7c330d61f99af7983e8714b4e63437d93813316ac54e8f83103b2ab2a021997e
(10 rows)
```
    @amnn
    amnn committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    f209497 View commit details
    Browse the repository at this point in the history

  2. refactor(indexer-alt): expose ingestion client 

    ## Description

Make it possible to access the ingestion client on the ingestion service
and indexer. This will be used to bootstrap the indexer which may need
to wait for the genesis checkpoint to initialise certain tables and
state before indexing proper starts.

## Test plan

CI
    @amnn
    amnn committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    7b51098 View commit details
    Browse the repository at this point in the history

  3. refactor(indexer-alt): IngestionClient::wait_for 

    ## Description

Factor out the logic used by the ingestion broadcaster to retry fetching
a checkpoint if it is not found. This change is behaviour preserving but
the new `wait_for` function will be re-used in some bootstrapping logic
that waits for the existence of the genesis checkpoint.

## Test plan

CI
    @amnn
    amnn committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    fc27b50 View commit details
    Browse the repository at this point in the history

  4. refactor(indexer-alt): expose indexer's connection pool 

    ## Description

...to be used in a bootstrapping script.
    @amnn
    amnn committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    2ff33fb View commit details
    Browse the repository at this point in the history

  5. refactor(indexer-alt): ev indices explicit NOT NULL on key columns 

    ## Description

Add explicit `NOT NULL`s to the key columns inside the event filter
tables. This is not a behavioural change as these columns are already
`NOT NULL` because they are part of the primary key. As per an earlier
PR comment, relying on the implicit `NOT NULL` behaviour was confusing
when reading migrations (and other migration scripts use the explicit
pattern).

## Test plan

CI
    @amnn
    amnn committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    101f1ca View commit details
    Browse the repository at this point in the history

  6. refactor(indexer-alt): graceful shutdown returns values 

    ## Description

Allow `graceful_shutdown` to accept tasks that return something other
than `()`.

## Test plan

CI
    @amnn
    amnn committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    cbd76fc View commit details
    Browse the repository at this point in the history

  7. indexer-alt: index chain identifier 

    ## Description

Index the chain identifier and initial protocol version in a
`kv_genesis` table. This takes the place of chain identifier indexing
and will also be used by indexers for protocol configs and feature
flags, which need to know the initial protocol version to index and the
chain identifier.

## Test plan

Run the indexer twice, and note from the tracing messages that the first
time, the indexer is bootstrapped, and the second time it reads from the
bootstrapped table.
    @amnn
    amnn committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    719db00 View commit details
    Browse the repository at this point in the history