Revision field, who is using this? Should I use it?

[linonetwo]

I'm writing a side project https://github.com/tiddly-gittly/TidGi-Mobile for fun, which will sync tiddler files back to the Desktop nodejs wiki's tiddlers/ folder. But to prevent conflict, I need some metadata. (OT and CRDT require too much metadata, so they are not my options.)

I have some discussion with GPT4, it recommends:

ASSISTANT:

If you want to keep it simple and use minimal metadata, comparing the revision or modified date can be a straightforward approach to resolving conflicts. Here's how you can structure the synchronization process using the title as the key:

1. Title as Key: Since the title is unique, you can use it as a primary identifier to check if the tiddler has been updated on both the server and the client.

2. Using Revision Field or Modified Date:

   • When synchronizing, for every tiddler, compare the revision or modified date from the server with that of the client.
   • If the server's revision (or date) is newer, overwrite the client's version with the server's version.
   • If the client's revision (or date) is newer, update the server with the client's version.
   • If both have the same revision but different content, this indicates a conflict. Here you can employ more advanced merging techniques.

3. Conflict Resolution:

   • Auto-merge: If two tiddlers have the same revision but different content, you can try auto-merging them. Several npm packages, such as diff-match-patch, can be used to perform a diff and patch operation. This works best for text-based content.
   • Git-style markers: If auto-merging isn't possible or isn't satisfactory, you can add git-style conflict markers, allowing users to resolve the conflict manually.
   • Creating Duplicate Entries: Another simple approach is to save both versions as separate tiddlers and inform the user, appending something like "(conflict)" to the title of the newer entry.

4. After Resolving Conflicts:

   • Once the conflict is resolved (either automatically or manually), update the revision field or modified date to reflect the change.
   • Sync the updated tiddler back to the server.

The advantage of this approach is its simplicity. However, manual conflict resolution may not be ideal for all users, and there is potential data loss in cases where automatic overwrites occur. Auto-merging reduces the chances of this happening, but it can introduce errors if the merging algorithm misinterprets the data.

Overall, using revisions or modified dates combined with intelligent conflict resolution can be an effective way to handle synchronization without the overhead of more complex systems like OT or CRDTs.

But I still have question that GPT can't answer (Seems It already forget tiddlywiki5's codebase):

1. How is the revision field used in tiddlywiki5's code? What feature is using it?
2. Will it be deleted when saved to files in /tiddlers folder?
3. Is it safe to use it in mobile-desktop sync?
4. Is it a good idea to increase 1 each time a tiddler is modified (each key typeing will make a tiddler modified).
5. Due to the fact that tiddler save happens very frequently, If I only want to increase it in critical circumstances, what would it be?

@Jermolene need help here, I want to implement cloud sync with SoLiD POD (a backend as a service).

[linonetwo]

Answer by me: a human who can read the source code...

1. In Syncer it will "Ignore the incoming tiddler if it's the same as the revision we've already got"
   1. but it should have trying to merge two tiddler with algorithm like diff-match-patch if revision is same (which might mean it is modified on both sides)
   2. I see Jeremy seems already working on diff-match-patch 5 years ago https://groups.google.com/g/tiddlywikidev/c/CBFGR8eLId4/m/V5S0hpXoAgAJ
   3. There is also a changeCount that can convert to revision tiddlerFields.revision = state.wiki.getChangeCount(title); in GET /recipes/default/tiddlers.json?filter=<filter>
2. I can see PUT /recipes/default/tiddlers/:title will "Remove any revision field". How is changeCount updated?
   1. This make this API useless, if a client wants to sync to the server, it might overwrite server tiddler with a higher revision
   2. This is intended, as said in Revision field should not be inerhited #2549
3. Even in the worst scenario, we can add git conflict mark to the .tid file's text. For field conflict, there is no good solution, just use the one with the newer modified field.
4. No, if desktop type abcd, and mobile type 123, then desktop will overwrite mobile, but they should have same revision: 1
   1. traditionally, with draft mechanism, Expected behaviour when a tiddler open in edit mode is modifed from outside the draft? #2108 (comment) , revision only increase when save a draft
   2. but when using WYSIWYG editor, this is not applied, need a new mechanism to update revision stability.
5. Seems we don't have such scenario that can safely increase revision

[linonetwo]

I'm not very sure about revision, so I decided to just use modified field:

HUMAN:

I'd like to use modified field (which is a timestamp) on each tiddler, and the lastSync timestamp of server info (on mobile) to decide how to solve the conflict. Can you design an algorithm to decide who will overwrite who?

ASSISTANT:

C

ertainly! Using the modified field on each tiddler, along with a lastSync timestamp from the server info, is a sensible approach. Let's break down the process:

Assumptions:

1. Every time a tiddler is modified, its modified timestamp is updated.
2. After every successful sync, the lastSync timestamp is updated.
3. If a tiddler is modified on both mobile and desktop without syncing in between, it's considered a conflict.

Algorithm:

For each tiddler to sync:

• if tiddler does not exist on the other side:
  • If tiddler exists on mobile but not on desktop: Push tiddler to desktop.
  • If tiddler exists on desktop but not on mobile: Pull tiddler to mobile.
• else if tiddler's modified on mobile > server's lastSync AND tiddler's modified on desktop > server's lastSync:
  • // This means the tiddler has been edited on both devices since the last sync: Conflict!
  • HandleConflict(tiddler on mobile, tiddler on desktop)
• else if tiddler's modified on mobile > server's lastSync:
  • // The tiddler on mobile is more recent than the last sync, so push it to desktop.
  • Push tiddler to desktop.
• else if tiddler's modified on desktop > server's lastSync:
  • // The tiddler on desktop is more recent than the last sync, so pull it to mobile.
  • Pull tiddler to mobile.
• else:
  • // The tiddler hasn't been modified on either side since the last sync, so no action needed.
  • Do nothing.

End for each

Handling Conflicts:

The HandleConflict function is crucial. Here are a few ways to address conflicts:

1. Timestamp Dominance: Use the tiddler with the most recent modified timestamp as the "winner", and overwrite the other.
2. Prompt User: If the conflict is detected while the user is present (e.g., during a manual sync), prompt them to decide which version to keep.
3. Merge: Try to merge the two tiddlers. This can be challenging depending on the tiddler's content but is ideal for preventing data loss. For text-based tiddlers, you can indeed use a diff algorithm or tool to help with this.
4. Keep Both: Rename one of the tiddlers (e.g., append "(Conflict)" to its title) and keep both. This ensures no data is lost, but the user will need to manually resolve and delete one of the tiddlers later.

Notes:

• Ensure backups are always made before syncing. This provides a safety net in case anything goes wrong.

• It's vital to update the lastSync timestamp only after a successful sync to ensure data consistency.

• For extra safety, consider introducing a "trash" or "archive" system, so instead of outright deleting a tiddler during a sync conflict resolution, you move it to the trash/archive. This way, if a mistake is made, the user can recover the data.

Remember, synchronization is complex, especially when dealing with potential data conflicts. It's crucial to test the algorithm thoroughly with different scenarios to ensure it works as expected.

Alright, let's lay out the server-side and client-side components of this synchronization system:

1. Client Side:

a. Prepare Data:

Collect all tiddlers with a modified timestamp greater than lastSync.

// Assuming you have a method getChangedTiddlers that filters tiddlers based on the modified timestamp
const changedTiddlers = getChangedTiddlers(lastSync);

b. Send POST Request:

Send the tiddlers to the server for synchronization.

// You'd use a function like fetch in modern JS or Axios in many frameworks
fetch(serverUrl + '/sync', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({ tiddlers: changedTiddlers, lastSync: lastSync }),
})
.then(response => response.json())
.then(syncResponse => {
    // Handle server's response
    processServerResponse(syncResponse);
})
.catch(error => {
    console.error("Sync failed:", error);
});

2. Server Side:

import express from 'express';
// TODO: import diff-match-patch library

const app = express();

app.use(express.json()); // for parsing application/json

app.post('/sync', (req, res) => {
    const clientTiddlers = req.body.tiddlers;
    const clientLastSync = req.body.lastSync;

    const serverResponse = {
        updates: [], // Tiddlers that the client should update or add
        deletes: [], // Tiddler titles that the client should delete
    };

    const processedTiddlerTitles = new Set();

    // Fetch the updated and deleted tiddlers from the server database BEFORE making any changes.
    const serverUpdatedTiddlers = getUpdatedTiddlersFromDatabase(clientLastSync);
    const serverDeletedTiddlers = getDeletedTiddlersFromDatabase(clientLastSync);

    clientTiddlers.forEach(clientTiddler => {
        const title = clientTiddler.title;
        processedTiddlerTitles.add(title);

        const serverTiddler = getTiddlerFromDatabase(title);

        if (!serverTiddler) {
            // Tiddler doesn't exist on server, so save client tiddler
            saveToDatabase(clientTiddler);
        } else if (clientTiddler.operation === 'DELETED') {
            // Tiddler was deleted on the client
            deleteFromDatabase(title);
            serverResponse.deletes.push(title);
        } else if (clientTiddler.modified > serverTiddler.modified) {
            // Client tiddler is newer
            saveToDatabase(clientTiddler);
        } else if (serverTiddler.modified > clientLastSync) {
            // Server tiddler is newer and has changed after client's last sync
            serverResponse.updates.push(serverTiddler);
        } else {
            // Conflict: Both client and server have modified the tiddler since the last sync
            // TODO: Implement diff-match-patch here or other conflict resolution
        }
    });

    serverUpdatedTiddlers.forEach(serverTiddler => {
        // Only add if the client hasn't already processed this tiddler
        if (!processedTiddlerTitles.has(serverTiddler.title)) {
            serverResponse.updates.push(serverTiddler);
        }
    });

    serverDeletedTiddlers.forEach(deletedTitle => {
        if (!processedTiddlerTitles.has(deletedTitle)) {
            serverResponse.deletes.push(deletedTitle);
        }
    });

    res.json(serverResponse);
});

// Assuming some utility functions:
// getTiddlerFromDatabase(title): Fetch a tiddler from the server database
// saveToDatabase(tiddler): Save a tiddler to the server database
// deleteFromDatabase(title): Delete a tiddler from the server database by title
// getDeletedTiddlersFromDatabase(sinceTimestamp): Fetch titles of tiddlers deleted after the given timestamp

3. Back to Client Side:

a. Process Server Response:

After receiving the response from the server, the client should process the updates and deletes.

function processServerResponse(syncResponse) {
    syncResponse.updates.forEach(tiddler => {
        // Update or add the tiddler in the client's storage
        saveTiddler(tiddler);
    });

    syncResponse.deletes.forEach(title => {
        // Delete the tiddler from the client's storage
        deleteTiddler(title);
    });

    // Update lastSync timestamp
    lastSync = getCurrentTimestamp();

    console.log("Sync completed successfully!");
}

This setup gives a clear flow of syncing, handling conflicts, and ensures both the client and server have the latest data after synchronization. Remember to handle edge cases, test thoroughly, and always backup data before making changes.

I think GPT4 is writing a Log-Based Replication.

[Jermolene]

Hi @linonetwo the history of the "bag" and "revision" fields is that they were introduced with TiddlyWeb back in 2008. As you've noted, they represent metadata that is needed for the client to be able to synchronise changes with the server.

I now think it was a mistake to put this information in tiddler fields, because it involves making changes to the users own data.

So, in the Xememex sync adaptor we keep that metadata in a pair of separate data dictionary tiddlers, avoiding having to modify the tiddlers themselves.

As part of the work in #7915 I am expecting to revisit the design of both the API and the sync protocol.

[Jermolene]

I should have added that you can see the Xememx syncadaptor at https://manuals.annafreud.org/ambit