Peloton is timing out

[trexfeathers]

Have done some rudimentary investigation with the help of @stephenworsley and @ESadek-MO. By experimenting with PaginatedMutation._PAGINATION:

.github/peloton/update_project.py

Lines 846 to 854 in 6cd87da

	class PaginatedMutation(abc.ABC):
	"""
	Base class for sgqlc mutations, including pagination to avoid timeouts.
	"""
	# _PAGINATION defines the number of mutation fields to include in one
	# mutation. If more are needed, the fields will be broken into
	# multiple pages.
	_PAGINATION = 20

... we have so far confirmed:

• This is a timeout problem - failures consistently occur after 10sec, which we believe is GitHub's chosen timeout¹; dropping _PAGINATION to 1 gets each query under 10secs and then consistently succeeds.
• It is specific to the updateProjectV2ItemFieldValue mutation - this is the last stage of the script; all previous mutation steps (e.g. adding, deleting, field-clearing) consistently succeed with the mutation pagination still set to 20.
• It is probably an issue with overhead - dropping _PAGINATION to 10 produced queries running in around 10secs, dropping to 1 only got down to 6secs. Given GitHub's the apparent 10sec timeout¹, a 5-6sec overhead doesn't leave much wiggle room.
• There are no recent reports of problems like this.

Using a _PAGINATION of 1 is not a good solution - updates will take prohibitively long, especially when recovering from an outage like this (more updates to do). A value of 10, coupled with some sort of repeat pattern using try-except might be the way to go. We agreed it would be worth reporting a MWE - including timings - to GitHub.

For now, the GitHub Action remains disabled. I have managed to update the Peloton by running locally with a _PAGINATION of 1. Anyone can run locally using an appropriate personal access token:

.github/peloton/update_project.py

Lines 1202 to 1209 in 6cd87da

	parser.add_argument(
	"--bearer_token",
	help=(
	"GitHub personal access token with these scopes: project, repo, "
	"read:org, read:discussion ."
	),
	type=str
	)

Footnotes

1. The documentation makes reference to this, although is a little unclear, and it is certainly consistent with what we've seen. ↩ ↩²