v3: fix the migration mess

docs/howto/django/migrations.md

@@ -4,14 +4,18 @@ Procrastinate comes with its own migrations so don't forget to run
 `./manage.py migrate`.
 
 Procrastinate provides 2 kinds of migrations:
-- The Django equivalent of the `procrastinate` normal migrations, which are
-  used to create all of the PostgreSQL DDL objects used by Procrastinate.
-- Specific noop migrations used for Django to understand the Procrastinate
-  Models (see {doc}`models`).
+
+-   The Django equivalent of the `procrastinate` normal migrations, which are
+    used to create all of the PostgreSQL DDL objects used by Procrastinate.
+-   Specific noop migrations used for Django to understand the Procrastinate
+    Models (see {doc}`models`).
 
 Procrastinate's Django migrations are always kept
 in sync with your current version of Procrastinate, it's always a good idea
 to check the release notes and read the migrations when upgrading so that you
 know what will be happening to the database.
 
-See {doc}`../production/migrations` for more information on migrations.
+See {doc}`../production/migrations` for more information on migrations, especially
+around `pre` and `post` migrations: if you deploy while the code is running, you'll
+want to ensure you run the `pre-` migrations before you deploy the code and the
+`post-` migrations after.

docs/howto/production/migrations.md

@@ -1,5 +1,10 @@
 # Migrate the Procrastinate schema
 
+:::{warning}
+v3 introduces a new way to handle migrations. Hopefully, easier both for users
+and maintainers. Read about pre- and post-migrations below.
+:::
+
 When the Procrastinate database schema evolves in new Procrastinate releases, new
 migrations are released alongside. Look at the
 [Release notes](https://github.com/procrastinate-org/procrastinate/releases)
@@ -31,17 +36,22 @@ on PyPI. A simple way to list all the migrations is to use the command:
 $ procrastinate schema --migrations-path
 /home/me/my_venv/lib/python3.x/lib/site-packages/procrastinate/sql/migrations
 ```
+
 It's your responsibility to keep track of which migrations have been applied yet
 or not. Thankfully, the names of procrastinate migrations should help you: they
 follow a specific pattern:
 
 ```
-xx.yy.zz_ab_very_short_description_of_the_migration.sql
+{xx.yy.zz}_{ab}_{pre|post}_very_short_description_of_the_migration.sql
 ```
 
-- `xx.yy.zz` is the version of Procrastinate the migration script can be applied to.
-- `ab` is the migration script's serial number, `01` being the first number in the
-  series.
+-   `xx.yy.zz` is the version of Procrastinate the migration script can be applied to.
+-   `ab` is the migration script's serial number, `01` being the first number in the
+    series.
+-   `pre` / `post`: indicates wether the migration should be applied before
+    upgrading the code (`pre`) or after upgrading the code (`post`) in the context
+    of a blue-green deployment. On old migrations, if `pre` or `post` is not
+    specified, it's a `post` migration.
 
 :::{note}
 There is a [debate](https://github.com/procrastinate-org/procrastinate/issues/1040)
@@ -50,51 +60,46 @@ directions for how to use classic ones (apart from Django), please feel free to
 and/or contribute code or documentation if you have an opinion on this.
 :::
 
-Let's say you are currently using Procrastinate 1.9.0, and you want to update to
-Procrastinate 1.15.0. In that case, before upgrading the Procrastinate Python package
-(from 1.9.0 to 1.15.0), you will need to apply all the migration scripts whose versions
-are greater than or equal to 1.9.0, and lower than 1.15.0 (1.9.0 ≤ version \< 1.15.0).
-And you will apply them in version order, and, for a version, in serial number order.
-For example, you will apply the following migration scripts, in that order:
-
-1. `01.09.00_01_xxxxx.sql`
-2. `01.10.00_01_xxxxx.sql`
-3. `01.11.00_01_xxxxx.sql`
-4. `01.11.00_02_xxxxx.sql`
-5. `01.12.00_01_xxxxx.sql`
-6. `01.14.00_01_xxxxx.sql`
-7. `01.14.00_02_xxxxx.sql`
+## How to apply migrations
 
-If you want to upgrade from one Procrastinate major version to another, say from
-Procrastinate 1.6.0 to 3.2.0, there are two options, depending on whether you can
-interrupt the service to do the migration or not.
-
-## The easier way, with service interruption
+##1 The easier way, with service interruption
 
 1. Shut down the services that use Procrastinate: both the services that defer tasks and
    the workers.
-2. Apply all the migration scripts (1.6.0 ≤ version \< 3.2.0).
-3. Upgrade your code to the new Procrastinate version (3.2.0).
+2. Apply all the migration scripts (`pre` & `post`), e.g. with:
+
+```console
+$ MIGRATION_TO_APPLY="02.00.00_01_pre_some_migration.sql"
+$ cat $(procrastinate schema --migrations-path)/${MIGRATION_TO_APPLY} | psql
+$ MIGRATION_TO_APPLY="02.00.00_01_post_some_migration.sql"
+$ cat $(procrastinate schema --migrations-path)/${MIGRATION_TO_APPLY} | psql
+$ ...
+```
+
+3. Upgrade your code to the new Procrastinate version.
 4. Start all the services.
 
 This, as you've noticed, only works if you're able to stop the services.
 
 ## The safer way, without service interruption
 
-:::{note}
-This only applies starting at Procrastinate 0.17.0. For previous versions,
-you will have to interrupt the service or write custom migrations.
-:::
+If you need to ensure service continuity, you'll need to make intermediate upgrades.
+Basically, you'll need to stop at every version that provides migrations.
+
+```console
+$ MIGRATION_TO_APPLY="02.01.00_01_pre_some_migration.sql"
+$ cat $(procrastinate schema --migrations-path)/${MIGRATION_TO_APPLY} | psql
+
+$ yoursystem/deploy procrastinate 2.1.0
 
-If you care about service continuity, you'll need to make intermediate upgrades. For
-example, to upgrade from Procrastinate 1.6.0 to 3.2.0, here are the steps you will need
-to follow:
+$ MIGRATION_TO_APPLY="02.01.00_01_post_some_migration.sql"
+$ cat $(procrastinate schema --migrations-path)/${MIGRATION_TO_APPLY} | psql
 
-1. Apply all the migration scripts between 1.6.0 and 2.0.0 (1.6.0 ≤ version \< 2.0.0).
-2. Live-upgrade the Procrastinate version used in your services, from 1.6.0 to 2.0.0.
-3. Apply all the migration scripts between 2.0.0 and 3.0.0 (2.0.0 ≤ version \< 3.0.0).
-4. Live-upgrade the Procrastinate version used in your services, from 2.0.0 to 3.0.0.
-5. Apply all the migration scripts between 3.0.0 and 3.2.0 (3.0.0 ≤ version \< 3.2.0).
-6. Live-upgrade the Procrastinate version used in your services, from 3.0.0 and 3.2.0.
+$ MIGRATION_TO_APPLY="02.02.00_01_pre_some_migration.sql"
+$ cat $(procrastinate schema --migrations-path)/${MIGRATION_TO_APPLY} | psql
 
-Following this process you can go from 1.6.0 to 3.2.0 with no service discontinuity.
+$ yoursystem/deploy procrastinate 2.2.0
+
+$ MIGRATION_TO_APPLY="02.02.00_01_post_some_migration.sql"
+$ cat $(procrastinate schema --migrations-path)/${MIGRATION_TO_APPLY} | psql
+```

procrastinate/contrib/django/migrations/0031_pre_cancel_notification.py

@@ -0,0 +1,20 @@
+from __future__ import annotations
+
+from django.db import migrations, models
+
+from .. import migrations_utils
+
+
+class Migration(migrations.Migration):
+    operations = [
+        migrations_utils.RunProcrastinateSQL(
+            name="03.00.00_01_pre_cancel_notification.sql"
+        ),
+        migrations.AddField(
+            "procrastinatejob",
+            "abort_requested",
+            models.BooleanField(),
+        ),
+    ]
+    name = "0031_pre_cancel_notification"
+    dependencies = [("procrastinate", "0030_alter_procrastinateevent_options")]

procrastinate/contrib/django/migrations/0031_add_abort_on_procrastinate_jobs.py → procrastinate/contrib/django/migrations/0032_post_cancel_notification.py

@@ -8,7 +8,7 @@
 class Migration(migrations.Migration):
     operations = [
         migrations_utils.RunProcrastinateSQL(
-            name="02.09.02_01_add_abort_on_procrastinate_jobs.sql"
+            name="03.00.00_50_post_cancel_notification.sql"
         ),
         migrations.AlterField(
             "procrastinatejob",
@@ -25,11 +25,6 @@ class Migration(migrations.Migration):
                 max_length=32,
             ),
         ),
-        migrations.AddField(
-            "procrastinatejob",
-            "abort_requested",
-            models.BooleanField(),
-        ),
     ]
-    name = "0031_add_abort_on_procrastinate_jobs"
-    dependencies = [("procrastinate", "0030_alter_procrastinateevent_options")]
+    name = "0032_post_cancel_notification"
+    dependencies = [("procrastinate", "0031_pre_cancel_notification")]