Add async start hook to ExtensionApp API

[Zsailer]

Fixes #1203 and #1329

[Zsailer]

@fcollonval @davidbrochart

[Zsailer]

cc @vidartf and @afshin, since we discussed this a bit on the server call today.

I've added documentation and made it possible to write one of these hooks for "classic" server extensions, i.e. extensions that don't inherit from the ExtensionApp.

The function must be named _start_jupyter_server_extension and should be found in the same location as the _load_jupyter_server_extension.

[Zsailer]

Downstream test failures appear to be unrelated.

[fcollonval]

Thanks @Zsailer

I have some questions about this:

• why not using start_extension to get an easier to understand API start/stop_extension ?
• if not renaming to start_extension, we should not prefix the class method with _ (aka ExtensionApp._start_jupyter_server_extension) as the convention implies that those have visibility protected and that does not fit the need for them to be public (to be accessed by the extension manager).
• I understand why you would add support for the classic extension. But in that case, I feel that we should also allow the stop_extension hook as it will surely be needed to clean started resources.

[Zsailer]

Hey @fcollonval 👋

Thanks for looking at this. First, let me acknowledge the obvious... naming is hard.

• why not using start_extension to get an easier to understand API start/stop_extension ?

I originally used this naming pattern, but in the last Jupyter Server meeting, folks asked if we could make this API available for "simple" server extension, i.e. one's that don't inherit from ExtensionApp. The pattern set for classic extension is to use _<verb>_jupyter_server_extension.

I'm fine with have start_extension in the Extension App, though, and just pointing a _start_jupyter_server_extension method at it.

• if not renaming to start_extension, we should not prefix the class method with _ (aka ExtensionApp._start_jupyter_server_extension) as the convention implies that those have visibility protected and that does not fit the need for them to be public (to be accessed by the extension manager).

Hm, I understand what you mean, but this is a slightly odd/edge case. The _start_jupyter_server_extension function lives in an extension, and the extension has two types of consumers that represent the "public" in this case: 1) the extension manager (as you mentioned) and 2) people extending the extension. It's critically important that we make this protected from group (2), i.e. people who are likely to break things if they override this method.

The extension manager is a special case. Yes, technically it's breaking the "public visibility within a class" convention, but it's a special type of "public" here. It must change if the _start_jupyter_server_extension signature changes; this method and the extension manager must evolve together and thus, their relationship needs to be protected.

We followed this same logic when renaming the _load_jupyter_server_extension and _link_jupyter_server_extension APIs.

• I understand why you would add support for the classic extension. But in that case, I feel that we should also allow the stop_extension hook as it will surely be needed to clean started resources.

Sure. I don't think it's strictly necessary for this PR to be reviewed/merged. I'm happy to add it here, but there are many other cases where you need an async start action without leaving lingering resources around that need cleanup.

[bollwyvl]

This is enabled in the current API, and is used by jupyter-lsp, using the function-based mechanism:

def load_jupyter_server_extension(nbapp):
    if hasattr(nbapp, "io_loop"):
        io_loop = nbapp.io_loop
    else:
        # handle jupyter_server 1.x
        io_loop = ioloop.IOLoop.current()

    io_loop.call_later(0, initialize, nbapp, virtual_documents_uri)

async def initialize(nbapp, virtual_documents_uri):
     await #... yadda yadda

The shortcoming is this spews a lot of output (especially with --debug) and obscures the URL.

We added this because people complained about how long a server took to start up in a JupyterHub setting.

Blocking the URL print would feel perceptually worse, but in the end would feel better.

[fcollonval]

Thanks @Zsailer for your detailed answer.

The arguments with the naming make sense. Indeed naming is always a high source of debates 😅

Regarding adding stop_extension in classic extension, don't hesitate to open an issue to have that point tackled in a follow-up PR.

[krassowski]

Hello, this would be super useful to speed-up startup of jupyter-ai and simplify code in jupyterlab-lsp.

What is blocking this PR? @Zsailer if you rebase would the CI pass now?

[Zsailer]

@krassowski I rebased, but some tests are still failing. The test related to the examples is a simple fix (just add a mock license to the package). The failing downstream test in JupyterLab isn't as clear to me.

Otherwise, nothing blocking here once we get the CI green.

[krassowski]

Thanks! I can take a look here tomorrow or Friday and open a PR against your branch if I get it working :)

[Zsailer]

Great, thank you @krassowski! If I get some time today/tomorrow, I'll try to fix too. I'd love to see this get merged too :)

[krassowski]

The failing downstream test in JupyterLab isn't as clear to me.

I've looked into this one - it is because the "critical" logs explaining how to open jupyter were moved from def start_app to async def _post_start. I mean this block:

jupyter_server/jupyter_server/serverapp.py

Lines 3091 to 3129 in e544fa1

	if self.identity_provider.token and self.identity_provider.token_generated:
	# log full URL with generated token, so there's a copy/pasteable link
	# with auth info.
	if self.sock:
	self.log.critical(
	"\n".join(
	[
	"\n",
	"Jupyter Server is listening on %s" % self.display_url,
	"",
	(
	"UNIX sockets are not browser-connectable, but you can tunnel to "
	f"the instance via e.g.`ssh -L 8888:{self.sock} -N user@this_host` and then "
	f"open e.g. {self.connection_url} in a browser."
	),
	]
	)
	)
	else:
	if self.no_browser_open_file:
	message = [
	"\n",
	_i18n("To access the server, copy and paste one of these URLs:"),
	" %s" % self.display_url,
	]
	else:
	message = [
	"\n",
	_i18n(
	"To access the server, open this file in a browser:",
	),
	" %s" % urljoin("file:", pathname2url(self.browser_open_file)),
	_i18n(
	"Or copy and paste one of these URLs:",
	),
	" %s" % self.display_url,
	]
	self.log.critical("\n".join(message))

This is because JupyterLab downstream checks if any errors were logged between application startup and completion of the test task.

We could:

• (a) add an extra filter in LogErrorHandler in JupyterLab's browser_check to ignore these critical's based on some substring match but these will eventually cause problems down the line if these strings are adjusted.
• (b) change these logs from critical to info to reflect what they really are (is this breaking?)
• (c) move this block back to sync start_app and document that this message banner will be shown before the async extensions are registered. I would actually prefer it to behave this way (unless the extension is a core app like Notebook or JupyterLab, then I would want it to only print once this is registered).

[krassowski]

@Zsailer unable to join the call today but wanted to check if you have thoughts on the a-c options above?

[Zsailer]

I think (c) is the right way to go here. Let's do this in a separate PR that we merge first and rebase here. If you have time before I get to it and want to work on such a PR, feel free to open it and assign me to review. Otherwise, I can open a PR later this week.

Thanks @krassowski!