Handle meta events

[roman-khimov]

Is your feature request related to a problem? Please describe.

A part of #2878.

Describe the solution you'd like

Add meta event handling:

• create a new meta DB (single instance per node), reuse https://pkg.go.dev/github.com/nspcc-dev/[email protected]/pkg/core/storage for it
• forget about buckets, use prefixes for everything
• listen for events, catch and process them
• for each block open a new MemCachedStore, process in-block data immediately
• in-block data allows to build an index of OIDs and big/lock indexes
• these prefixes are added into https://pkg.go.dev/github.com/nspcc-dev/[email protected]/pkg/core/mpt when we're done with block processing (keep only latest state, we don't need historic)
• a full header index (see Implement searchv2 #3058) can be managed in a separate routine and not be a part of MPT
• mem cache flushing can be per-block currently

Describe alternatives you've considered

There can be some, but we need this data to be processed fast and we'll need MPT for subsequent synchronization.

[roman-khimov]

@AnnaShaleva and @AliceInHunterland can tell a bit more about the way storage is handled in NeoGo. What we're doing here is very similar to storeBlock except that we work with events and some data needs to be obtained additionally (but it's not a part of the state proper at the same time).

[carpawell]

@roman-khimov

1. So we have a new DB now for our indexes (not sure why it is a reuse of neo-go's one but ok, dont mind) and also have a MemCachedStore memory instance for every block, why? That is exactly "cached" version, not just a memory instance per block, right? Over what is it created, the original disk DB? What does the cache solve?
2. "listen for events", "for each block". What is this new DB subscribed for? I thought that it should receive only our new events from meta-on-chain TXs but it also needs to process every block?
3. "these prefixes are added into https://pkg.go.dev/github.com/nspcc-dev/[email protected]/pkg/core/mpt". What exact "these prefixes"? Why we need mpt here?
4. "full header index ... not be a part of MPT" what is the criterion of what mpt and what is not?

[roman-khimov]

1. It can be done without MemCachedStore using per-block Bolt transactions, but reusing NeoGo storage subsystem allows to simplify some interfaces and make DBs changeable. In future we can also optimize flushing from per-block to something smarter. Memory itself is irrelevant here.
2. Listen for new events, you need to watch for objects, that's the only thing you care about (epochs are interesting too, but maybe not immediately).
3. MPT is to identify and synchronize state. Prefixes are those created above, not the full index.
4. Reproducibility. I can fetch headers and recreate index using data stored in MPT, so adding everything into MPT is a resource waste.

[carpawell]

@roman-khimov, @cthulhu-rider, considering #3085 and #3080 we now have two meta storages: first one is a fully new DB that relies only on the information from FSchain and currently stores only this info in MPT (and persisting it to the new DB instance); the second one(s) is the old metabase per every shard that takes info from PUT/REPLICATE RPC directly without any object flow changes.
AFAIU, our target goal is to have some combination of these PRs. We need to have meta-from-chain indexes in MPT but we also need to have an index(es) of what is stored on disk.
We need to discuss:

1. If we need some third kind of DB that stores meta information but neither in MPT, nor in the old metabases. Indexes for objects that are not stored in the current node but that are also not just MTP data from the FSchain. As I understand ("...a full header index can be managed in a separate routine and not be a part of MPT..."), a full header should also be processed after receiving notification from FSchain, not just be a part of a regular shard's PUT routine.
2. What code should be shared to prevent repeating? Should there be support for more than just bolt (neo-go supports more storages and works on interfaces, our current metabase code -- no)?
3. In what order do we release our features?

[roman-khimov]

1. Same DB, different prefixes/handling.
2. Leave Bolt to the old metabase, reuse DB abstraction provided by NeoGo. But you need to translate headers into KVs and this code can be shared.
3. Search, meta.