storage api

[unicomp21]

Where should I be looking, in order to understand the storage api? Documentation? Is there enough here to implement something like the rocksdb used by kafka streams? Or am I in left field?

[emaxerrno]

we already have a k=v storage :) it is just one level of merge, not N-way. - https://github.com/vectorizedio/redpanda/blob/dev/src/v/storage/kvstore.h

the storage api is in the /storage folder

[unicomp21]

trying to get my head straight. given everything is stored in memory, I shouldn't use k=v for large datasets?

for something similar to a graph database, should I be thinking more in terms of storing objects within compacted topics? and the delta encoded updates in non-compacted topics? if this is the case, do we have anything which allows streaming compression of an entire non-compacted partition? or have I just created an argument for running redpanda on brtfs w/ compression enabled?

[emaxerrno]

Right. Is just a way for you to see the storage api.

The only stable api we offer is the Kafka api atm

[dotnwat]

@unicomp21 for bulk topic storage start with src/v/storage/disk_log_impl.h. the key-value store is only used for a small amount of metadata.