Handling large amounts of data

[kragoth]

Hi,

So, after 18 months now of an app in production that uses akita heavily I want to discuss ways of dealing with large amounts of data.

Now my idea of what causes my problem is based on a hunch as I cannot prove that my theory is totally correct but... the evidence seems to suggest it might be. So, feel free to shoot this down. :)

We have built an app that must work offline. To do this we sync all data to the device that the user may need to perform their job for that day. This can be 50-150MB (I think I've seen this up to 300MB of data but... on average probably 100MB) of data (we have spatial data to deal with).
But, what this means is that any change to any store within akita results in the full state needing to be re-written to indexdb.
Now, I think when this write occurs, the thread is blocked. So, the UI freezes until the persist operation is done. Now this is normally pretty quick but, when you have a lot of data it can be 2-5 seconds.

I have just completed a massive refactor of our app in which I have moved all akita operations to a web worker. This has made a huge difference for us as we no longer get UI freezing when updates occur.

While this does relieve the problems from a UI perspecrtive, the web worker also becomes non responsive while the persist operation is occurring. Which means the UI won't receive updates from queries etc until the web worker is finished persisting the state.

So.... are there any options/ideas on how to deal with this or improve the behaviour.

I toyed with the idea what about a database per store.... but, I guess the downside to this is you could lose data consistency if one write succeeds and another fails.

Anyway, I'd be interested in hearing from anyone else that might be dealing with a lot of data.
Or maybe I'm totally off on what I think is causing the problem!

[NetanelBasal]

Hi!

First, I want to say that your issue isn't related to Akita. It will happen with any other solution.

I'd try these options:

1. Save only the deltas in indexdb ( I'm not familiar with your application so I couldn't suggest more than that)
2. Save the state in these events: offline, visiblityChange, or use requestIdleCallback to save the state.

[NetanelBasal]

When you update any store, Akita saves the entire application state to storage ( the stores that you provided ). If you split it, in your example, if you update entity C, it'll save only the entity C state to storage.

[kragoth]

oooh ok!
Well maybe that's all I need to do!

Thanks @NetanelBasal I'll see if I can get that working. Really appreciate the discussion. 😄

[NetanelBasal]

Sure, good luck.

[kragoth]

@NetanelBasal, just thought I'd let you know.
I have implemented the separate branch in storage strategy and it has made a noticible improvement!
Thanks again!

[NetanelBasal]

Great!