feat: Handle Virtual Documents persistance in PouchLink #1486

Ldoppea · 2024-05-31T14:53:02Z

Note

This PR follows #1483

When a PouchLink is configured, the synchronization process would retrieve only documents that exists on the remote database

Those documents would be served as usual when a .query() is made using the PouchLink

Unfortunately, not all .query() correspond to a document that exists on the remote database. With the collection mechanism, some documents are retrieved using an alternative endpoint and then the cozy-stack generate the document on-demand

For example this is the case for io.cozy.settings.flags, io.cozy.settings.context and io.cozy.settings.disk-usage

Because those documents does not actually exist on the remote database, we cannot sync them using the PouchDB sync mechanism and so they won't be available when the device is offline

To fix this, we chose to manually insert those document in the local PouchDB when we detect that a .query() call retrieve them from the cozy-stack

The easiest way we found to detect them is to check for the existence of meta.rev object in the .query() result. If no rev is present, this means that the documents does not come from the remote database but has been dynamically generated by the cozy-stack

Those documents are inserted into existing PouchBD with the cozyLocalOnly attributes, so we know that they should not be replicated to the remote database on next sync

In order to make the persist mechanism possible, we implemented a new persistData() method to the CozyLink interface

The impact is that every Link should now implement the persistData() interface that should persist the data if it is able to do it, or to forward the action otherwise

If no Link process the action, then the final "default" method will do nothing (contrary to the default for request() that would throw an error)

BREAKING CHANGE: CozyClient's links should now implement a persistData() method that should persist the given data if it is able to do it, or to forward the action to the next Link otherwise

Related PRs:

Crash--

J'aime pas trop le terme de persistData() car c'est pas de la persistance générique qu'on met là-dedans vu qu'on vient rajouter du cozyLocalOnly, qu'on dégage le _type etc. C'est pas juste on file de la data et on la persiste.

Mais à part ça, top !

packages/cozy-client/src/CozyLink.js

packages/cozy-client/src/types.js

paultranvan · 2024-08-12T13:21:49Z

packages/cozy-client/src/CozyClient.js

+   * @returns {Promise<void>}
+   */
+  async persistVirtualDocument(document) {
+    if (document && !document.meta?.rev && !document.cozyLocalOnly) {


⚠️ We need to handle documents with _rev and not only meta.rev, which is only for json-api

Fixed in this commit: 62290ed

However I did not do actual tests yet, I expect to do them tomorrow

paultranvan · 2024-08-12T13:23:24Z

packages/cozy-client/src/CozyClient.js

+    if (!Array.isArray(data)) {
+      await this.persistVirtualDocument(data)
+    } else {
+      for (const document of data) {


I realize this forces a full iteration on all docs for all requests, which is kind of a pity, especially since we actually need this in quite few cases

Yes, but I don't think this will cost that much as this is a pure JS loop that check objects properties. Also I don't have better solution, so I propose to keep it that way for now but to be attentive about performances

but to be attentive about performances

I'm pretty sure we won't detect this overhead. I guess there would be some way to improve, but let's keep it simple for now

paultranvan · 2024-08-12T13:25:47Z

packages/cozy-pouch-link/src/CozyPouchLink.js

@@ -384,6 +386,44 @@ class PouchLink extends CozyLink {
      return this.executeQuery(operation)
    }
  }
+
+  async persistData(data, forward = doNothing) {
+    const docWithoutType = sanitized(data)


sanitized sounds very generic for a function that simply removes the _type 😅

Done in #1524

paultranvan · 2024-08-12T13:28:30Z

packages/cozy-pouch-link/src/CozyPouchLink.js

+    const docWithoutType = sanitized(data)
+    docWithoutType.cozyLocalOnly = true
+
+    const oldDoc = await this.getExistingDocument(data._id, data._type)


It's a detail, but in this very specific case, I wonder if we should not check on id and type instead, as it is what is returned in the json-api format (we add _id and _type in the normalization step)

I should do some verifications first before answering that. I'll keep you in touch

paultranvan · 2024-08-12T13:30:38Z

packages/cozy-pouch-link/src/CozyPouchLink.js

+    }
+
+    const db = this.pouches.getPouch(data._type)
+    await db.put(docWithoutType)


What about the attributes object that is returned by the stack for json-api docs? I assume we should handle it?

What do you mean by "handle it"? We should remove it? Or something else?

I mean that a document returned by the stack will look like this with json-api formatting:

{ id: 123, type: 'io.cozy.todo', attributes: { name: 'do stuff' } }

Thanks to the normalization, the attributes is handled by spreading the content at the document root, typically here: https://github.com/cozy/cozy-client/blob/master/packages/cozy-stack-client/src/SettingsCollection.js#L17
But it still exists, so I think we should remove it before saving it to pouchdb

Done in #1525

paultranvan · 2024-08-12T13:39:28Z

packages/cozy-client/src/CozyClient.js

+      document &&
+      !document.meta?.rev &&
+      !document.cozyLocalOnly &&
+      !document.cozyFromPouch


I feel like this whole commit (5762d43) is not necessary anymore if we correctly handle the ._rev condition?

This may be correct, but I should do some verifications first to confirm the theory. I'll keep you in touch

Done in #1524

packages/cozy-client/src/CozyClient.js

paultranvan · 2024-08-12T14:14:50Z

J'aime pas trop le terme de persistData() car c'est pas de la persistance générique qu'on met là-dedans

Est-ce qu'un persistCozyData te satisferait plus, @Crash-- ? Le Cozy sous-entendant qu'on fait des trucs spécifiques à nous

When specifying fields in a query, e.g. `Q('io.cozy.todos').where({done: true}).select(['date'])`, the revision was missing if not explicitly given. This is now problematic because we rely on the revision existence to identify "virtual" documents, i.e. not persisted in CouchDB, that never have any revision. See #1486 for more insights.

This commit is a copy of #1517 applied to CozyPouchLink When specifying fields in a query, e.g. `Q('io.cozy.todos').where({done: true}).select(['date'])`, the revision was missing if not explicitly given. This is now problematic because we rely on the revision existence to identify "virtual" documents, i.e. not persisted in CouchDB, that never have any revision. See #1486 for more insights.

This guard has been added to handle documents with no `meta.rev` but since we added a check on `_rev` (in addition to `meta.rev`) to trigger the persistence, then we don't need this guard anymore This replies to #1486 (comment) Related commit: b797eb3 Related commit: 62290ed

Those doc's attributes are specific to the JSON API and should not be inserted into the Pouch database This implies that we will have a difference on documents regarding if they are served through the cozy-stack or through a local PouchDB, the first one may include those fields in their result, but not the second one So from now we should avoid, as much as possible, to relies on the `attributes` member to prevent bugs on Offline mode. Multiple commits to fix usages of `attributes` in cozy-client will be done after this one Usage of `attributes` and `meta` members on cozy-app will also have to be fixed. This will be a requirement to implement Offline mode on cozy-apps This replies to #1486 (comment)

In previous implementation we did not await for the `persistVirtualDocuments()` result before completing the request This was done because the persistance was not mandatory for the request's result and doing it in parallel would be a good way to optimize timings But in some recent experiments we found a scenario where we would benefit from the `await` in order to some external code until the data is actually persisted Related PR: #1486

Those doc's attributes are specific to the JSON API and should not be inserted into the Pouch database This implies that we will have a difference on documents regarding if they are served through the cozy-stack or through a local PouchDB, the first one may include those fields in their result, but not the second one So from now we should avoid, as much as possible, to relies on the `attributes` member to prevent bugs on Offline mode. Multiple commits to fix usages of `attributes` in cozy-client will be done after this one Usage of `attributes` and `meta` members on cozy-app will also have to be fixed. This will be a requirement to implement Offline mode on cozy-apps This replies to #1486 (comment)

In previous implementation we did not await for the `persistVirtualDocuments()` result before completing the request This was done because the persistance was not mandatory for the request's result and doing it in parallel would be a good way to optimize timings But in some recent experiments we found a scenario where we would benefit from the `await` in order to some external code until the data is actually persisted Related PR: #1486

Those doc's attributes are specific to the JSON API and should not be inserted into the Pouch database This implies that we will have a difference on documents regarding if they are served through the cozy-stack or through a local PouchDB, the first one may include those fields in their result, but not the second one So from now we should avoid, as much as possible, to relies on the `attributes` member to prevent bugs on Offline mode. Multiple commits to fix usages of `attributes` in cozy-client will be done after this one Usage of `attributes` and `meta` members on cozy-app will also have to be fixed. This will be a requirement to implement Offline mode on cozy-apps This replies to #1486 (comment)

In previous implementation we did not await for the `persistVirtualDocuments()` result before completing the request This was done because the persistance was not mandatory for the request's result and doing it in parallel would be a good way to optimize timings But in some recent experiments we found a scenario where we would benefit from the `await` in order to some external code until the data is actually persisted Related PR: #1486

This replies to #1486 (comment)

This commit is a copy of #1517 applied to CozyPouchLink When specifying fields in a query, e.g. `Q('io.cozy.todos').where({done: true}).select(['date'])`, the revision was missing if not explicitly given. This is now problematic because we rely on the revision existence to identify "virtual" documents, i.e. not persisted in CouchDB, that never have any revision. See #1486 for more insights.

This guard has been added to handle documents with no `meta.rev` but since we added a check on `_rev` (in addition to `meta.rev`) to trigger the persistence, then we don't need this guard anymore This replies to #1486 (comment) Related commit: b797eb3 Related commit: 62290ed

Those doc's attributes are specific to the JSON API and should not be inserted into the Pouch database This implies that we will have a difference on documents regarding if they are served through the cozy-stack or through a local PouchDB, the first one may include those fields in their result, but not the second one So from now we should avoid, as much as possible, to relies on the `attributes` member to prevent bugs on Offline mode. Multiple commits to fix usages of `attributes` in cozy-client will be done after this one Usage of `attributes` and `meta` members on cozy-app will also have to be fixed. This will be a requirement to implement Offline mode on cozy-apps This replies to #1486 (comment)

In previous implementation we did not await for the `persistVirtualDocuments()` result before completing the request This was done because the persistance was not mandatory for the request's result and doing it in parallel would be a good way to optimize timings But in some recent experiments we found a scenario where we would benefit from the `await` in order to some external code until the data is actually persisted Related PR: #1486

This replies to #1486 (comment)

This commit is a copy of #1517 applied to CozyPouchLink When specifying fields in a query, e.g. `Q('io.cozy.todos').where({done: true}).select(['date'])`, the revision was missing if not explicitly given. This is now problematic because we rely on the revision existence to identify "virtual" documents, i.e. not persisted in CouchDB, that never have any revision. See #1486 for more insights.

This guard has been added to handle documents with no `meta.rev` but since we added a check on `_rev` (in addition to `meta.rev`) to trigger the persistence, then we don't need this guard anymore This replies to #1486 (comment) Related commit: b797eb3 Related commit: 62290ed

Those doc's attributes are specific to the JSON API and should not be inserted into the Pouch database This implies that we will have a difference on documents regarding if they are served through the cozy-stack or through a local PouchDB, the first one may include those fields in their result, but not the second one So from now we should avoid, as much as possible, to relies on the `attributes` member to prevent bugs on Offline mode. Multiple commits to fix usages of `attributes` in cozy-client will be done after this one Usage of `attributes` and `meta` members on cozy-app will also have to be fixed. This will be a requirement to implement Offline mode on cozy-apps This replies to #1486 (comment)

In previous implementation we did not await for the `persistVirtualDocuments()` result before completing the request This was done because the persistance was not mandatory for the request's result and doing it in parallel would be a good way to optimize timings But in some recent experiments we found a scenario where we would benefit from the `await` in order to some external code until the data is actually persisted Related PR: #1486

This replies to #1486 (comment)

This commit is a copy of #1517 applied to CozyPouchLink When specifying fields in a query, e.g. `Q('io.cozy.todos').where({done: true}).select(['date'])`, the revision was missing if not explicitly given. This is now problematic because we rely on the revision existence to identify "virtual" documents, i.e. not persisted in CouchDB, that never have any revision. See #1486 for more insights.

This guard has been added to handle documents with no `meta.rev` but since we added a check on `_rev` (in addition to `meta.rev`) to trigger the persistence, then we don't need this guard anymore This replies to #1486 (comment) Related commit: b797eb3 Related commit: 62290ed

Those doc's attributes are specific to the JSON API and should not be inserted into the Pouch database This implies that we will have a difference on documents regarding if they are served through the cozy-stack or through a local PouchDB, the first one may include those fields in their result, but not the second one So from now we should avoid, as much as possible, to relies on the `attributes` member to prevent bugs on Offline mode. Multiple commits to fix usages of `attributes` in cozy-client will be done after this one Usage of `attributes` and `meta` members on cozy-app will also have to be fixed. This will be a requirement to implement Offline mode on cozy-apps This replies to #1486 (comment)

In previous implementation we did not await for the `persistVirtualDocuments()` result before completing the request This was done because the persistance was not mandatory for the request's result and doing it in parallel would be a good way to optimize timings But in some recent experiments we found a scenario where we would benefit from the `await` in order to some external code until the data is actually persisted Related PR: #1486

This replies to #1486 (comment)

This commit is a copy of #1517 applied to CozyPouchLink When specifying fields in a query, e.g. `Q('io.cozy.todos').where({done: true}).select(['date'])`, the revision was missing if not explicitly given. This is now problematic because we rely on the revision existence to identify "virtual" documents, i.e. not persisted in CouchDB, that never have any revision. See #1486 for more insights.

This guard has been added to handle documents with no `meta.rev` but since we added a check on `_rev` (in addition to `meta.rev`) to trigger the persistence, then we don't need this guard anymore This replies to #1486 (comment) Related commit: b797eb3 Related commit: 62290ed

Those doc's attributes are specific to the JSON API and should not be inserted into the Pouch database This implies that we will have a difference on documents regarding if they are served through the cozy-stack or through a local PouchDB, the first one may include those fields in their result, but not the second one So from now we should avoid, as much as possible, to relies on the `attributes` member to prevent bugs on Offline mode. Multiple commits to fix usages of `attributes` in cozy-client will be done after this one Usage of `attributes` and `meta` members on cozy-app will also have to be fixed. This will be a requirement to implement Offline mode on cozy-apps This replies to #1486 (comment)

In previous implementation we did not await for the `persistVirtualDocuments()` result before completing the request This was done because the persistance was not mandatory for the request's result and doing it in parallel would be a good way to optimize timings But in some recent experiments we found a scenario where we would benefit from the `await` in order to some external code until the data is actually persisted Related PR: #1486

This was referenced May 31, 2024

feat: Implement offline support through CozyPouchLink cozy/cozy-flagship-app#1209

Open

feat: Forward operation in StackLink.request() when offline #1487

Merged

feat: Make CozyPouchLink compatible with ReactNative #1483

Merged

Ldoppea force-pushed the feat/pouch_react_native branch from bdea993 to 2f00064 Compare July 16, 2024 09:16

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch from 4c4c489 to 3d29bb0 Compare July 16, 2024 09:18

Ldoppea force-pushed the feat/pouch_react_native branch 3 times, most recently from 9a3b23b to dddcdbc Compare July 16, 2024 15:38

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch from 3d29bb0 to b425f41 Compare July 16, 2024 15:38

Ldoppea force-pushed the feat/pouch_react_native branch from dddcdbc to 17d7454 Compare July 17, 2024 06:57

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch from b425f41 to bcb49e6 Compare July 17, 2024 06:57

Ldoppea force-pushed the feat/pouch_react_native branch from 17d7454 to c376d72 Compare July 17, 2024 07:50

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch from bcb49e6 to 02632b9 Compare July 17, 2024 07:50

Ldoppea force-pushed the feat/pouch_react_native branch from c376d72 to 255e57e Compare July 17, 2024 07:57

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch 3 times, most recently from e842a07 to a761b3b Compare July 17, 2024 13:34

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch from a761b3b to b8abde2 Compare July 26, 2024 14:23

Ldoppea mentioned this pull request Jul 26, 2024

Meta PR: Implement offline support through CozyPouchLink #1507

Merged

Ldoppea marked this pull request as ready for review July 26, 2024 14:46

Ldoppea requested review from Merkur39 and paultranvan as code owners July 26, 2024 14:46

Ldoppea force-pushed the feat/pouch_react_native branch from 255e57e to 0037e6f Compare July 26, 2024 15:04

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch from b8abde2 to d5677b5 Compare July 26, 2024 15:04

Crash-- approved these changes Aug 2, 2024

View reviewed changes

paultranvan reviewed Aug 12, 2024

View reviewed changes

Ldoppea force-pushed the feat/handle_virtual_documents_persistance branch from d5677b5 to 95978d3 Compare August 13, 2024 15:31

paultranvan mentioned this pull request Aug 20, 2024

feat: Force _rev for queries with .select fields #1517

Merged

Ldoppea added a commit that referenced this pull request Sep 19, 2024

refactor: Rename persistData into persistCozyData

fa85c83

This replies to #1486 (comment)

Ldoppea added a commit that referenced this pull request Sep 19, 2024

refactor: Rename persistData into persistCozyData

48c0e93

This replies to #1486 (comment)

Ldoppea added a commit that referenced this pull request Sep 24, 2024

refactor: Rename persistData into persistCozyData

d7bd364

This replies to #1486 (comment)

Ldoppea added a commit that referenced this pull request Sep 24, 2024

refactor: Rename persistData into persistCozyData

b410ac9

This replies to #1486 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Handle Virtual Documents persistance in PouchLink #1486

feat: Handle Virtual Documents persistance in PouchLink #1486

Ldoppea commented May 31, 2024 •

edited

Loading

Crash-- left a comment

paultranvan Aug 12, 2024

Ldoppea Aug 13, 2024

paultranvan Aug 12, 2024

Ldoppea Aug 13, 2024

paultranvan Aug 14, 2024

paultranvan Aug 12, 2024

Ldoppea Sep 4, 2024

paultranvan Aug 12, 2024

Ldoppea Aug 13, 2024

paultranvan Aug 12, 2024

Ldoppea Aug 13, 2024

paultranvan Aug 14, 2024

Ldoppea Sep 4, 2024 •

edited

Loading

paultranvan Aug 12, 2024

Ldoppea Aug 13, 2024

Ldoppea Sep 4, 2024

paultranvan commented Aug 12, 2024

feat: Handle Virtual Documents persistance in PouchLink #1486

feat: Handle Virtual Documents persistance in PouchLink #1486

Conversation

Ldoppea commented May 31, 2024 • edited Loading

Related PRs:

Crash-- left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ldoppea Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paultranvan commented Aug 12, 2024

Ldoppea commented May 31, 2024 •

edited

Loading

Ldoppea Sep 4, 2024 •

edited

Loading