Skip to content

Commit

Permalink
Draft summit report
Browse files Browse the repository at this point in the history
  • Loading branch information
@theory
theory committed Jun 14, 2024
1 parent c7e6ddb commit b7951b6
Showing 1 changed file with 131 additions and 37 deletions.
168 changes: 131 additions & 37 deletions content/post/postgres/extension-ecosystem-summit-2024/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ image:
metaOnly: false
---

The (first annual?) PostgreSQL [Extension Ecosystem Summit] took place at
PGConf.dev in Vancouver on May 28, 2024 and it was great! Around 35 extension
developers, users, and fans gathered for an [open-space technology]
(OST)-style unconference. I opened with a brief presentation ([slides]) to
introduce the Summit Theme:
The PostgreSQL [Extension Ecosystem Summit] took place at PGConf.dev in
Vancouver on May 28, 2024 and it was great! Around 35 extension developers,
users, and fans gathered for an [open-space technology] (OST)-style
unconference. I opened with a brief presentation ([slides]) to introduce the
Summit Theme:

> * Extension issues, designs and features
> * Development, packaging, installation, discovery, docs, etc.
Expand All @@ -38,44 +38,44 @@ Before the Summit my co-organizers and I had put up large sticky notes with
potential topics, and after reviewing the four principles and one law of
[OST], we collectively looked them over and various people offered to lead
discussions. Others volunteered to take notes and later published them on the
[community wiki]. Here's our report:
[community wiki]. Here's our report.

Extension Metadata
------------------

Tembo's Samay Sharma of Tembo took point on this discussion, while David
Wagoner of EDB took [notes][metadata-notes]. This wide-ranging discussion
among the five participants covered taxonomies, versioning, system
dependencies, packaging & discoverability, development & compatibility, and
more.
Samay Sharma of Tembo took point on this discussion, while David Wagoner of
EDB took [notes][metadata-notes]. The wide-ranging discussion among the five
participants covered taxonomies, versioning, system dependencies, packaging &
discoverability, development & compatibility, and more.

The discoverability topic seemed particularly engaging, with participants
brainstorming features such as user comments & ratings, usage insights, and
test reporting. The settled on the idea of two types of metadata:
The discoverability topic particularly engaged the participants, as they
brainstormed features such as user comments & ratings, usage insights, and
test reporting. They settled on the idea of two types of metadata:
developer-provided and user ratings. I'm gratified how closely this hews to
the [metadata sketch]'s [packaging][] author and [registry][] (third party)
metadata.
the [metadata sketch]'s proposed [packaging] (author) and [registry] (third
party) metadata.

Binary Distribution Format
--------------------------

I led this session, while Andreas "ads" Scherbaum took [notes][binary-notes].
I proposed to my four colleagues an idea I'd been mulling for a couple months
for an extension binary distribution format inspired by [Python wheel]. It
simply includes pre-compiled files in subdirectories named for each directory
configuration option from `pg_config`. The other half of the idea, inspired by
an [Álvaro Hernández blog post], is to distribute these packages via [OCI] ---
in other words, just like Docker images. The participants agreed it was an
simply includes pre-compiled files in subdirectories named for each
`pg_config` directory config. The other half of the idea, inspired by an
[Álvaro Hernández blog post], is to distribute these packages via [OCI] --- in
other words, just like Docker images. The participants agreed it was an
interesting idea to investigate.

Most of the rest of the time was spent reviewing and trying to understand the
We spent much of the rest of the time reviewing and trying to understand the
inherent difficulty of upgrading binary extensions: there's a period between
when an extension package is upgraded (from Yum, Apt, etc.) and `ALTER
EXTENSION UPDATE` updates it in the database. If the new binary doesn't work
with old versions, they will break (and potentially crash Postgres!) until
they update. This can be difficult in, say, a data analytics environment where
many users use the extension in multiple databases and may not have the
bandwidth to all update their installs and any code that depends on them.
with old versions, it will break (and potentially crash Postgres!) until they
update. This can be difficult in, say, a data analytics environment with uses
of the extension in multiple databases and functions, and users may not have
the bandwidth to `ALTER EXTENSION UPDATE` any code that depends on the
extension.

This issue is best solved by defensive coding of the C library to keep it
working for new and old versions of an extension, but this complicates
Expand All @@ -89,34 +89,114 @@ version standard.
ABI/API discussion
------------------

This discussion focused in issues of API and [ABI] compatibility in the
Postgres core. Today virtually the entire code base is open for use by
extension developers --- anything in header files. Some [recent research]
revealed a few potentially-incompatible changes in minor releases of Postgres,
and this group brainstormed improvements for this situation. Ideas included
Yurii Rashkovskii took point on this session while David Christensen took
[notes][api-notes]. Around 25 attendees participated. The discussion focused
in issues of API and [ABI] compatibility in the Postgres core. Today virtually
the entire code base is open for use by extension developers --- anything in
header files. Some [recent research] revealed a few potentially-incompatible
changes in minor releases of Postgres, leading some to conclude that
extensions must be compiled and distributed separately for every minor
release. The group brainstormed improvements for this situation. Ideas
included:

* Spelunking the source to document and categorize APIs for extensions
* Documenting Color-coded safety classifications for APIs: green, yellow, or
* Documenting color-coded safety classifications for APIs: green, yellow, or
red
* Providing hook registration to access advanced functionality, rather than
the simple functional hooks Postgres offers today
* And of course more and better-designed hooks for behavior that's difficult
to access from extensions today
* Designing and providing a better way to register and call hooks
(observability, administration, isolation, etc.), rather than the simple
functions Postgres offers today
* Developing a test farm to regularly build and tests extensions, especially
ahead of a core release
* And of course creating *more* hooks, such as custom relation type
handling, per-database background workers, a generic node visitor pattern,
and better dependency handling

Including/Excluding Extensions in Core
--------------------------------------


Keith Fiske led the discussion and took [notes][corext-notes] for this
session, along with 10-15 or so attendees. It joined two topics: When should
an extension be brought into core and when should a contrib extension be
removed from core. The central point was the adoption of new features in core
that replace the functionality of and therefore reduce the need for some
extensions.

Replacing an extension with core functionality simplifies things for users.
However, the existence of an extension might prevent core from ever adding its
features. Extensions can undergo faster, independent development cycles
without burdening the committers with more code to maintain. This independence
encourages more people to develop extensions, and potentially compels core to
better support extensions overall (e.g., through better APIs/ABIs).

Contrib extensions currently serve, in part, to ensure that the extension
infrastructure itself is regularly tested. Replacing them with core features
would reduce the test coverage, although one participant proposed a patch to
add such tests to core itself, rather than as part of contrib extensions.

The participants collaborated on a list of contrib extensions to consider
merging into core:

* amcheck
* pageinspect
* pg_buffercache
* pg_freespacemap
* pg_visibility
* pg_walinspect
* pgstattuple

They also suggested moving extension metadata (SQL scripts and control files)
from disk to catalogs and adding support for installing and using multiple
versions of an extension at one time (complicated by shared libraries),
perhaps by the adoption of more explicit extension namespacing.

Potential core changes for extensions, namespaces, etc.
-------------------------------------------------------

Yurii Rashkovskii and David Christensen teamed up on this session, as well
([notes][core-notes]). 15-20 attendees brainstormed core changes to improve
extension development and management. These included:

* File organization/layout, such as putting all the files for an extension
in a single directory and moving some files to the system catalog.
* Provide a registry of "safe" extensions that can be installed without a
superuser.
* Adding a GUC to configure a second directory for extensions, to enable
immutable Postgres images (e.g., Docker, Postgres.app). The attendees
consider this a short term fix, but still useful. (Related: I started a
[pgsql-hackers thread] in April for a patch to to just this).
* The ability to support multiple versions of an extension at once, via
namespacing, came up in this session, as well.
* Participants also expressed a desire to support duplicate names through
deeper namespacing. Fundamentally, the problem of namespace collision
redounds to issues un-relocatable extensions.

Until Next Time
---------------

I found it interesting how many topics cropped up multiple times in separate
sessions. By my reading most cited topics were:

* The need to install and use multiple versions of an extension
* A desire for deeper namespacing, in part to allow for multiple versions of
an extension
* A pretty strong desire for an [ABI] compatibility policy and clearer
understanding of extension-friendly APIs

I expect to put some time into these topics; indeed, I've already started a
[Hackers thread proposing an ABI policy].

I greatly enjoyed the discussions and attention given to a variety of
extension-related topics at the Summit. So much enthusiasm and intelligence in
one places just makes my day!

I'm thinking maybe we should plan to do it again next year. What do you think?
Join the [#extensions] channel on the [Postgres Slack] with your ideas!

[Extension Ecosystem Summit]: https://www.pgevents.ca/events/pgconfdev2024/schedule/session/191/
"PGConf.dev: Extensions Ecosystem Summit: Enabling comprehensive indexing, discovery, and binary distribution"
[PGConf.dev]: https://2024.pgconf.dev "PostgreSQL Development Conference 2024"
[open-space technology]: https://en.wikipedia.org/wiki/Open_space_technology
"Wikipedia: "Open space technology"
"Wikipedia: Open space technology"
[slides]: {{% link "/shared/extension-ecosystem-summit/extension-summit-2024.pdf" %}}
"🏔 PostgreSQL Extension Summit: An Unconference"
[community wiki]: https://wiki.postgresql.org/wiki/PGConf.dev_2024_Extension_Summit
Expand All @@ -139,4 +219,18 @@ Potential core changes for extensions, namespaces, etc.
"Open Container Initiative Distribution Specification"
[ABI]: https://en.wikipedia.org/wiki/Application_binary_interface
"Wikipedia: Application binary interface"
[api-notes]: https://wiki.postgresql.org/wiki/PGConf.dev_2024_Extension_Summit#ABI.2FAPI_discussion
"PostgreSQL Wiki: PGConf.dev 2024 Extension Summit --- ABI/API discussion"
[recent research]: {{% ref "/post/postgres/mini-summit-five" %}} "Mini Summit Five"
[corext-notes]: https://wiki.postgresql.org/wiki/PGConf.dev_2024_Extension_Summit#Including.2FExcluding_Extensions_in_Core
"PostgreSQL Wiki: PGConf.dev 2024 Extension Summit --- Including/Excluding Extensions in Core"
[core-notes]: https://wiki.postgresql.org/wiki/PGConf.dev_2024_Extension_Summit#Potential_core_changes_for_extensions.2C_namespaces.2C_etc
"PostgreSQL Wiki: PGConf.dev 2024 Extension Summit --- Potential core changes for extensions, namespaces, etc"
[pgsql-hackers thread]: https://www.postgresql.org/message-id/flat/E7C7BFFB-8857-48D4-A71F-88B359FADCFD%40justatheory.com
"pgsql-hackers: RFC: Additional Directory for Extensions"
[Hackers thread proposing an ABI policy]: https://www.postgresql.org/message-id/flat/5DA9F9D2-B8B2-43DE-BD4D-53A4160F6E8D%40justatheory.com
"pgsql-hackers: Proposal: Document ABI Compatibility"
[#extensions]: https://postgresteam.slack.com/archives/C056ZA93H1A
"Postgres Slack/#extensions: Extensions and extension-related accessories"
[Postgres Slack]: https://pgtreats.info/slack-invite
"Join the Postgres Slack"

0 comments on commit b7951b6

Please sign in to comment.