add Force-Push

[BlackFoundry]

Will push to git immediately without having to wait 10 minutes for the next synchronisation.

[justvanrossum]

Move issue to https://github.com/BlackFoundryCom/django-robo-cjk ?

[fabiocaccamo]

No need to move the issue, since I can reference issues on other repositories as you suggested me.

[justvanrossum]

Sure, but in general I agree with your original point that issues should be at the repo that is most relevant. (I didn't know the django repo existed until today :) )

[fabiocaccamo]

Actually for Hanzi, it takes ~11 minutes to generate the physical project from the database, so I had to change the git sync interval to 20 minutes to avoid overlapping synchronizations.

Generating the whole project at once from the database data is surely slow, but at the same time we can be 100% sure that the data is aligned with database data and that there is not stale data on the repo.

I could reduce drastically the time by updating (at file system level) only files changed since last push but I want to think in advance to all possible drawbacks.

Btw, I want to re-think e re-build how the physical project is built.
My current idea is to update files asynchronously directly when data change, in this way there will be not project generation bottleneck and commit/push frequency could be increased.

Any suggestion or idea is welcome.

[justvanrossum]

I hardly know the contraints of the system, so this may be naive, but how about:

• incoming change from designer
• database is updated
• save to disk is scheduled/queued to a separate save-to-disk thread or process
• git push is done scheduled at an interval

This way the on-disk filesystem representation of the projects is much less behind, and a force-push should be less problematic.

If you're worried about data sync issues: perhaps have a single nightly job that compares and complains if something is off? That would also allow you to debug/fix any sync issues that remain.

[justvanrossum]

With "save to disk is scheduled/queued" I mean only for the data that was changed.

[fabiocaccamo]

Yes, that's exactly what I mean, then once a day during the night we could run an "hard sync" that generates the whole project from the database (it is what we actually do), in this way we go to delete possible old files.

[justvanrossum]

1. I wouldn't do a hard sync unless you also analyse what differences the "clean" export has with the actual export, and fix those differences, so that eventually we'll get the confidence that the hard sync isn't needed at all.
2. Why aren't file deletions handled like normal changes? There shouldn't be a cleanup necessary for those.

[fabiocaccamo]

1. That's true, but for comparing the differences we need to iterate over all records and files anyway.
2. File deletions could be handled, but file updates can't because django doesn't fire pre_update / post_update signal when performing UPDATE queries (for performance reasons I think), but since we don't do bulk updates (we update one record at time) I could try to check if it is possible to patch the signals sistem to support a signal on update.

[justvanrossum]

1. In other words: do a once-per-day integrity check of db vs. file system and report differences.
2. Not sure what most of those things mean – are you saying there isn't any django-robo-cjk code in between the user request and the db transaction?

[fabiocaccamo]

1. exactly
2. the database operations are managed by the django ORM, that provides a signals system for many database operations, excluding the UPDATE queries, btw I will think to a workaround.