@resumable: Towards general purpose incremental delivery

[mbleigh]

Defer and stream offer fantastic primitives for incremental delivery on a single chunked request, and I'm looking forward to using them. I also think the new wire protocol is well-crafted to enable pretty broad use case scenarios, which is what I wanted to bring up here.

I'm interested in GraphQL having better capabilities for common patterns like pagination without needing to rely on bespoke interfaces and per-application implementation at the schema level. I believe the incremental delivery protocol offers an interesting avenue to such things. To solve for these, GraphQL could support generic "resumable" operations.

I'm writing this up to see if there's broader interest in exploring this idea further -- if this seems promising, I can put together a more formal proposal and/or prototype a simple client/server implementation to demonstrate the concept.

New directive: @resumable

Let's imagine a schema like the following:

scalar ResumeToken
directive @resumable(label: String) on FIELD_DEFINITION | FIELD

type Post {
  id: ID!
  text: String!
  order: Int!
  categories: [String]
}

type Query {
  posts(limit: Int, category: String, resume: ResumeToken): [Post] @resumable
}

With a corresponding query document that looks something like:

query($category: String) {
  posts(limit: 20, category: $category) @resumable(label: "posts") {
    id, text, categories
  }
}

The @resumable directive can be placed on any field definition so long as the field takes a resume: ResumeToken argument. It can also be placed on any query field so long as the corresponding field is marked as resumable.

When both client and server have opted into resumable operations, the server will return additional metadata in the response following a similar pattern to the pending response field on @defer / @stream responses:

// request wire protocol
{
  "variables": {"category": "Food"},
  "query": "..." // see query document above
}

// response wire protocol
{
  "data": {
    "posts": [
      {"id": "1", "text": "...", "categories": ["Food"]},
      // ... 19 more posts
    ]
  },
  "resumable": {
    "pending": [{
      "id": "0",
      "path": ["posts"],
      "token": "..." // opaque token returned by the server
    }]
  }
}

This response indicates that there are additional results for the posts field that can be fetched by resuming the operation. If the server evaluates the query and does not find additional results, it returns a "complete" entry in resumable instead of pending.

The token value is an opaque cursor that the server will use to determine how to resume the operation on a subsequent request. In the example above, the token might be the ID of the last post that was returned.

The client can then make a subsequent request to the server to resume the operation:

// request wire protocol
{
  // the client must pass the same variables and query as
  // the original request
  "variables": {"category": "Food"},
  "query": "...", // see query document above

  // pass one or more entries from the response's resumable array
  "resume": [
    {"id": "0", "path": ["posts"], "token": "<cursor>"}
  ]
}

The server will receive this request and execute resolvers for the equivalent query of:

query {
  posts(limit: 20, category: "Food", resume: "<cursor>") @resumable(label: "posts") {
    id, text, categories
  }
}

However, the results of the resolver will be delivered in the incremental field of the response, mapped to the same path as the resumable field:

// response wire protocol
{
  // there is no "data" field in the response
  "incremental": [
    // the wire protocol from defer/stream is reused here
    {"id": 0, "label": "posts", "items": [/* up to 20 more posts */]}
  ],
  "resumable": {
    // assuming the server has reached the end of the result set,
    // it returns a "complete" entry in resumable instead of pending
    "complete": [
      {"id": "0", "path": ["posts"]}
    ]
  }
}

The client can use the result of this followup request to append new items to the end of the previous result set, following the same mechanics as incremental delivery from @defer / @stream.

Not Just for Pagination

While the example above is probably the most common use case, the specific behavior of how to turn a resume cursor into incremental updates is left to the server implementation.

Let's imagine a very different scenario: a breaking news alerts system.

type Alert {
  id: ID!
  message: String!
  createdAt: DateTime!
}

type Query {
  latestAlerts(resume: ResumeToken): [Alert] @resumable
}

# query doc
query {
  latestAlerts @resumable {
    id, message, createdAt
  }
}

In this case, "resuming" the query doesn't mean fetching an additional page of existing results, but rather checking to see if there are any new alerts that have been created since the last time the client queried the server.

For this use case, the resumable is never "complete", because there is always a chance that new alerts will be created. A subsequent response might look like:

{
  "incremental": [
    {"id": 0, "label": "latestAlerts", "prependItems": [
      {"id": "1", "message": "Earthquake in San Francisco", "createdAt": "2024-02-14T12:00:00Z"},
      {"id": "2", "message": "Severe weather in New York", "createdAt": "2024-02-14T12:01:00Z"}
    ]}
  ],
  "resumable": {
    "pending": [{"id": "0", "path": ["latestAlerts"], "token": "<cursor>"}]
  }
}

You'll note that the incremental response includes a prependItems field. This is an addition to the @defer / @stream protocol to account for the fact that incremental data may modify existing data in more ways than just appending to the end of the list.

Further Exploration Required

There's significantly more thought required to turn this into a full specification; however, I was surprised at how well this concept seemed to fit with the existing @defer / @stream primitives. While there's a fair amount of "client smarts" needed to do this properly, it seems pretty tractable since the actual updating of the data will be compatible with @defer/@stream.

I'd love to hear early reactions from this group to know if this is something worth my time to pursue further.

Thanks all!

[mbleigh]

/cc @benjie @robrichard curious if you have any initial thoughts since I chatted with both of you a bit about this in broad terms at GraphQLConf.

[benjie]

Hi Michael, thanks for sharing these detailed thoughts!

It looks to me like you're taking the cursor approach from cursor pagination and then extending that to any collection (i.e. simple lists) by sending the cursor (aka "resume token") out-of-band rather than as part of the data. Further; you're making it so that when a collection is "resumed" we're only sending results from that specific collection, and not repeating the entire operation - which aids efficiency (though you wouldn't normally re-issue the same query in cursor pagination) - and you're doing so using the incremental delivery format, which helps minimize effort for adoption.

Some questions:

• You re-issue the query and variables along with the resume token, but do not detail what would happen if the query or variables were to be changed. Should the token include a hash of them?
• Would this allow pagination inside nested lists, via passing the path (including indexes) as part of resume? How would this work, and are there edge cases we need to deal with here?
• What guidance do we need to give people when managing their resume tokens - how can they protect themselves from malicious actors feeding custom or differently sourced tokens?
• Since fields with @resumable can also occur in mutation and subscription operations, how would these operations handle such?
• Incremental delivery has been explicitly designed to handle ordered collections and return their new items in order; with your latestAlerts example this will continue to add items to the list - is that okay?
• In the latestAlerts example, since the resumable is incomplete, what would stop a client continually polling the server as fast as it can to get the next alert? Should there be a recommended back-off in the case of an empty (but incomplete) response from the server?
• Are the resume tokens stateful? I imagine they could be done in a stateless way, like cursors, but care would need to be taken over how they are formed to avoid tampering.
• You mentioned prependItems but this may not be the only case - perhaps they need to insert at a specific index. Either way, if this is something that this system would need, you should get involved with incremental delivery before it's too late - we made a decision a couple years ago to not include the index in the paths for incremental delivery (and make it "append only"), if you have solid needs to change this they should be discussed urgently.

My main question though relates to the guiding principle "Favor no change": can't this all be achieved via cursor pagination without requiring any changes to the specification? Perhaps we should focus on standardizing cursor pagination across the ecosystem rather than adding another mechanism?

[mbleigh]

Thanks for the replies! Some responses inline.

• You re-issue the query and variables along with the resume token, but do not detail what would happen if the query or variables were to be changed. Should the token include a hash of them?

Potentially, although I think it would probably be fine to leave this as an exercise for the server implementor and not make it a concern of the spec. If an implementor is concerned about request tampering, they could include a hash in the token, but I'm concerned about the bloating effect of bytes on wire it would be to do so (since a single query may have many resumable parts).

If the query or variables are changed, then the opaque token would be evaluated against the different query/variables. In many cases that would lead to an error, in some cases it might succeed and produce strange behavior, but in either case I'm not seeing an attack surface here. Are you concerned about security or about accidental misconfiguration of clients?

• Would this allow pagination inside nested lists, via passing the path (including indexes) as part of resume? How would this work, and are there edge cases we need to deal with here?

Yes, and I'm guessing there's more thinking that needs to be done here. The main edge cases I can think of center around handling path/index management for resumables-inside-resumables. I think the way to address this is to make the paths "local" to the specific request. So rather than dealing with a global index of all previous returns, instead a path can always be fully resolved knowing only the data in that specific request.

• What guidance do we need to give people when managing their resume tokens - how can they protect themselves from malicious actors feeding custom or differently sourced tokens?

I'd probably set the guidance as "treat the contents of resume tokens, even if 'decoded' from an opaque string, as equivalent to untrusted variables". By and large I think this is harmless -- if my resume token encodes "apple" and I pass "cucumber" I'm essentially just "skipping" a bunch of records. I'm curious if you can think of any specific threat vectors around this, though.

• Since fields with @resumable can also occur in mutation and subscription operations, how would these operations handle such?

I think similar to @defer and @stream I would restrict resumable to be "query-only" for the time being. I can't really think of a reasonable use case for "resumable mutations" or "resumable subscriptions" off the top of my head.

• Incremental delivery has been explicitly designed to handle ordered collections and return their new items in order; with your latestAlerts example this will continue to add items to the list - is that okay?

Yes, so the use case here is essentially "pull to refresh" type mechanics, or "poll to refresh" as another way of thinking. Resumable results should always be continuously ordered with or patching values of existing results, but should be able to either append (like defer/stream) or prepend (as is described as a new capability in the latestAlerts example) to that ordered list.

• In the latestAlerts example, since the resumable is incomplete, what would stop a client continually polling the server as fast as it can to get the next alert? Should there be a recommended back-off in the case of an empty (but incomplete) response from the server?

I like the idea of some kind of client hinting for how and when to refetch, that could be neat. Nothing would stop the client from spamming the server, but resumables (in my mind) are different from @defer / @stream in that they should not be considered eager. It should never be the default behavior of a client to greedily "consume" resumables, because the point here is to provide hooks for opt-in loading of additional data driven by some kind of client-side event.

Servers should likely have rate-limiting in place to try to handle abusive enumeration attacks, but the same can be said for any kind of cursor-based pagination.

• Are the resume tokens stateful? I imagine they could be done in a stateless way, like cursors, but care would need to be taken over how they are formed to avoid tampering.

As mentioned above, I'm not sure tampering is a problem. It's pretty common practice in my experience to e.g. base64-encode the relevant "sort keys" for a query as the pagination token and believe what the client sends even if it's been tampered with. In theory a client should be able to infinitely paginate a given query if there is a cursor returned, so tampering basically has a worst-case of "skipping" results which seems fine.

• You mentioned prependItems but this may not be the only case - perhaps they need to insert at a specific index. Either way, if this is something that this system would need, you should get involved with incremental delivery before it's too late - we made a decision a couple years ago to not include the index in the paths for incremental delivery (and make it "append only"), if you have solid needs to change this they should be discussed urgently.

Yes, I'd like to get involved here before it's too late. While it's not urgently necessary for defer and stream, what you've already designed is almost a robust "diff patching" protocol for GraphQL data, and I'd love to at least keep the optionality open to explore additional incremental updates in the future.

What's the best way to get involved?

My main question though relates to the guiding principle "Favor no change": can't this all be achieved via cursor pagination without requiring any changes to the specification? Perhaps we should focus on standardizing cursor pagination across the ecosystem rather than adding another mechanism?

Fundamentally I don't think it's possible to solve these problems without a wire-protocol-level intervention either through extensions or through a spec change. Resumables allow for efficient updates to targeted parts of the response. Without resumables, you essentially need to dynamically compose queries together from subqueries (possible to some extent using fragments but really messy), develop an "in-data" protocol for signaling how and where to patch the previous results, then execute the "next page" as a separate query that you manually patch into the previous result out-of-band.

Resumables also solve for a whole class of problems that are commonly griped about in terms of "GraphQL needs abstract types" because you no longer need to implement a common pagination interface on top of each of your application's data types; pagination is handled out-of-band of the response shape.

Basically I think the "incremental" wire protocol is a really well-designed and powerful idea and I don't want to see it boxed up for a narrow use case when it could have much wider-ranging implications for building sophisticated data-fetching.

[benjie]

Are you concerned about security or about accidental misconfiguration of clients?

Both, but primarily security because people tend to be cavalier with this stuff. For example, consider issuing this query:

{
  viewer {
    orgAsUser { # Org type
      members @resumable { id name } # Member type; no email field
    }
  }
}

Imagine that the developers, having been given no guidance or warnings, think it's reasonable for the resume token to consist of a tuple of the Org id and the last member ID returned.

Now the user issues:

{
  viewer {
    orgAsAdmin { # OrgAsAdmin type
      allMembers @resumable { id name email } # MemberAsAdmin type
    }
  }
}

The server may read and validate the resume token, and proceed to list all members of that organization, potentially including hidden/archived/old/etc members that wouldn't normally be visible, and also revealing their email addresses. Normally this would be prevented via the orgAsAdmin resolver returning null, but I don't think you've specified if all of the parent resolvers need to be called again or not so this attack assumes there's some optimized implementation that skips straight to the resumed field (I figured the resume() resolver would accept (resumeToken, args, context, resolveInfo) but maybe it also accepts a parentObject from the parent resolver?)

This is just one example of a potential security flaw that could occur from under-specifying how a token like this might work.

Yes, so the use case here is essentially "pull to refresh" type mechanics, or "poll to refresh" as another way of thinking. Resumable results should always be continuously ordered with or patching values of existing results, but should be able to either append (like defer/stream) or prepend (as is described as a new capability in the latestAlerts example) to that ordered list.

What if an alert were to be deleted, or updated? Is this need better served via subscriptions?

It should never be the default behavior of a client to greedily "consume" resumables, because the point here is to provide hooks for opt-in loading of additional data driven by some kind of client-side event.

This is an important part of the intent for this feature and should be highlighted.

what you've already designed is almost a robust "diff patching" protocol for GraphQL data, and I'd love to at least keep the optionality open to explore additional incremental updates in the future.

This is something that we discussed but ultimately decided against, including the indexes caused too many problems (e.g. splicing arrays is expensive, including the index for every list entry consumed more data on the wire, what do we do if an index is repeated, what do we do for gaps, etc; I don't remember all the reasons off-hand) but I'm sure we'd be happy to have the conversation again with the added context of this suggestion. Please consider adding it as a topic to an upcoming incremental delivery working group. Fair warning: there will likely be some pushback since the current format has been stable since ~June 2023 and is implemented in a number of places already, but we do like to get these things right (and permanent) so if you feel strongly you should definitely represent that ASAP!