From 176f8489e8210eec6aa42f0c9c4d2d38d41069f0 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 30 Nov 2021 23:33:59 -0600 Subject: [PATCH 1/6] Backward extremities do not indicate whether we have fetched the prev_events Spawning from https://github.com/matrix-org/synapse/pull/9445#discussion_r758958181 --- docs/development/room-dag-concepts.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md index 5eed72bec662..d87524e7f606 100644 --- a/docs/development/room-dag-concepts.md +++ b/docs/development/room-dag-concepts.md @@ -38,16 +38,14 @@ Most-recent-in-time events in the DAG which are not referenced by any other even The forward extremities of a room are used as the `prev_events` when the next event is sent. -## Backwards extremity +## Backward extremity The current marker of where we have backfilled up to and will generally be the oldest-in-time events we know of in the DAG. -This is an event where we haven't fetched all of the `prev_events` for. - -Once we have fetched all of its `prev_events`, it's unmarked as a backwards -extremity (although we may have formed new backwards extremities from the prev -events during the backfilling process). +When we persist a non-outlier event, we clear it as a backward extremity and set +all of its `prev_events` as the new backward extremities if they aren't already +persisted in the `events` table. ## Outliers @@ -56,8 +54,7 @@ We mark an event as an `outlier` when we haven't figured out the state for the room at that point in the DAG yet. We won't *necessarily* have the `prev_events` of an `outlier` in the database, -but it's entirely possible that we *might*. The status of whether we have all of -the `prev_events` is marked as a [backwards extremity](#backwards-extremity). +but it's entirely possible that we *might*. For example, when we fetch the event auth chain or state for a given event, we mark all of those claimed auth events as outliers because we haven't done the From 701e2e5b9b44009c72359f39ee30dea3bb2748da Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 1 Dec 2021 00:00:32 -0600 Subject: [PATCH 2/6] Document inconsistency in outlier, events table, and backward extremity --- docs/development/room-dag-concepts.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md index d87524e7f606..52c68723a6e6 100644 --- a/docs/development/room-dag-concepts.md +++ b/docs/development/room-dag-concepts.md @@ -47,6 +47,14 @@ When we persist a non-outlier event, we clear it as a backward extremity and set all of its `prev_events` as the new backward extremities if they aren't already persisted in the `events` table. +If an event is marked as a backward extremity by being a `prev_event` of some +other event, then persisted as an `outlier`, it can be a backward extremity while +still being in the `events` table. + +If an event is first persited as an `outlier`, then comes across the backward +extremity update by being a `prev_event` of some other event, it won't become a +backward extremity. + ## Outliers From 46d697c2e9aac2472b0d861d14693f1db2def60e Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 1 Dec 2021 00:05:36 -0600 Subject: [PATCH 3/6] Remove the fluff which I think is actually not true --- docs/development/room-dag-concepts.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md index 52c68723a6e6..d87524e7f606 100644 --- a/docs/development/room-dag-concepts.md +++ b/docs/development/room-dag-concepts.md @@ -47,14 +47,6 @@ When we persist a non-outlier event, we clear it as a backward extremity and set all of its `prev_events` as the new backward extremities if they aren't already persisted in the `events` table. -If an event is marked as a backward extremity by being a `prev_event` of some -other event, then persisted as an `outlier`, it can be a backward extremity while -still being in the `events` table. - -If an event is first persited as an `outlier`, then comes across the backward -extremity update by being a `prev_event` of some other event, it won't become a -backward extremity. - ## Outliers From 93145e3fd8f71353ae99c6df7fc1d9d0c38c5562 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 1 Dec 2021 00:20:11 -0600 Subject: [PATCH 4/6] Add changelog --- changelog.d/11469.doc | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/11469.doc diff --git a/changelog.d/11469.doc b/changelog.d/11469.doc new file mode 100644 index 000000000000..51602aa96895 --- /dev/null +++ b/changelog.d/11469.doc @@ -0,0 +1 @@ +Update section about backward extremities in the room DAG concepts doc to correct the misconception about backward extremities indicating whether we have fetched an events' `prev_events`. From 7870e8301d133140ee97e7c75d119815a8b325a6 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Thu, 2 Dec 2021 23:56:16 -0600 Subject: [PATCH 5/6] Add note for backward extremities being a backfill starting point See https://github.com/matrix-org/synapse/pull/11469#discussion_r760078299 --- docs/development/room-dag-concepts.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md index d87524e7f606..ec909023c4ec 100644 --- a/docs/development/room-dag-concepts.md +++ b/docs/development/room-dag-concepts.md @@ -41,7 +41,8 @@ The forward extremities of a room are used as the `prev_events` when the next ev ## Backward extremity The current marker of where we have backfilled up to and will generally be the -oldest-in-time events we know of in the DAG. +oldest-in-time events we know of in the DAG. This gives a starting point when +backfilling history. When we persist a non-outlier event, we clear it as a backward extremity and set all of its `prev_events` as the new backward extremities if they aren't already From e8dc4b7cd49901fecc8817cb8c49d2de98c356af Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 3 Dec 2021 16:56:12 -0600 Subject: [PATCH 6/6] Update docs/development/room-dag-concepts.md Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- docs/development/room-dag-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md index ec909023c4ec..cbc7cf29491c 100644 --- a/docs/development/room-dag-concepts.md +++ b/docs/development/room-dag-concepts.md @@ -41,7 +41,7 @@ The forward extremities of a room are used as the `prev_events` when the next ev ## Backward extremity The current marker of where we have backfilled up to and will generally be the -oldest-in-time events we know of in the DAG. This gives a starting point when +`prev_events` of the oldest-in-time events we have in the DAG. This gives a starting point when backfilling history. When we persist a non-outlier event, we clear it as a backward extremity and set