Add metadata service to Jupyter Server

[lresende]

Proposal

The Metadata Service provides a generic and flexible schema-driven data store for metadata that can be easily integrated with other backend and/or frontend components.

Metadata Service overview

The core design of the metadata service is primarily driven by schema. That is, all metadata instances must adhere to a schema. Metadata schemas are organized logically and physically into schema spaces, where a given schema space contains a set of schemas, and each instance of a schema space’s schemas is co-located relative to the configured storage mechanism.

Externally, both in its persisted and serialized forms, a metadata instance is represented as a JSON structure. Within the metadata service, however, this JSON structure is represented as a Metadata class instance. The Metadata class can be extended via subclassing to enable custom behaviors that are triggered following load operations and/or before and after create, update, and delete operations on a per schema level. That is, it is the respective schema that identifies the name of the class to use as the internal representation.

As previously mentioned, the metadata service has a pluggable storage mechanism. By default, metadata instances are stored within the file system. However, alternative storage mechanisms can be configured for, say, NoSQL DB or traditional, table-oriented, databases.

Schemaspaces and their schemas can be dynamically introduced via entrypoints. When the system starts, it will identify applicable entrypoints and load their corresponding objects to seed the list of configured schemaspaces and schemas.

Design proposal

We are proposing a similar design as the one used by terminals

• The component will be implemented as a separate repository
• The component will be integrated and enabled by default to jupyter server

Additional references

More details about the existing Metadata service implementation

• Metadata Service: https://elyra.readthedocs.io/en/stable/developer_guide/metadata.html
• Elyra metadata service implementation: https://github.com/elyra-ai/elyra/tree/main/elyra/metadata
• Metadata command line examples: https://elyra.readthedocs.io/en/stable/user_guide/command-line-interface.html
• Maybe some overlap with this?: https://github.com/jupyter/schema

[lresende]

cc @kevin-bates for awareness.

[lresende]

@jupyter-server/jupyter-server-council Any concerns, comments or questions on this proposal?

[Zsailer]

I love this. I'm in favor of at least moving the metadata service and server extension into the Jupyter Server org. This is a well-tested, robust extension that would likely require little maintenance from the server team and offers a ton of benefit.

[Zsailer]

I also think enabling it by default would benefit users/consumers of the server.

[Zsailer]

I'd propose that we open an explicit vote on moving the metadata service into the jupyter-server org in early January.