Skip to content

Commit

Permalink
Revisions
Browse files Browse the repository at this point in the history
  • Loading branch information
@theory
theory committed Nov 1, 2024
1 parent 066bcca commit 8a88d04
Showing 1 changed file with 90 additions and 94 deletions.
184 changes: 90 additions & 94 deletions content/post/postgres/rfc-extension-packaging-lookup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ more unified, public proposal.
## The Problems

A number of challenges face extension users in various configurations, thanks
to the status quo of extension file organization in the Postgres core. The
common thread among them is the need to add extensions without changing the
Postgres installation itself.
to extension file organization in the Postgres core. The common thread among
them is the need to add extensions without changing the Postgres installation
itself.

### Packager Testing

Expand All @@ -44,9 +44,9 @@ directly from the package build directory.
Alas, the patch isn't ideal, because it simply specifies a prefix and appends
the full `pg_config` directory paths to it. For example, if `--sharedir`
outputs `/opt/share` and `extension_destdir` GUC is set to `/tmp/build/myext`,
the patch will search in `/tmp/build/myext/opt/share`. This approach well for
the packaging use case, which explicitly uses a prefix pattern, but would be a
bit weird for other use cases.
the patch will search in `/tmp/build/myext/opt/share`. This approach works for
the packaging use case, which explicitly uses full paths with a prefix, but
would be a bit weird for other use cases.

### Docker Immutability

Expand All @@ -64,9 +64,9 @@ deployment configuration complexity.
### Postgres.app Immutability

The macOS [Postgres.app] supports extensions. But installing one into
[SHAREDIR/extensions] changes the contents of the Postgres.app bundle,
`SHAREDIR/extensions` changes the contents of the Postgres.app bundle,
breaking the app's Apple-required signature validation. The OS will no longer
be able to validate that the app is legit, so will refuse to start it.
be able to validate that the app is legit and refuse to start it.

## Solution

Expand All @@ -83,13 +83,13 @@ single directory. These include:

Subdirectories roughly correspond to the `pg_config --*dir` options:

* `sql`: SQL files
* `bin`: Executables
* `doc`: Documentation files
* `html`: HTML documentation files
* `lib`: Dynamically loadable modules
* `locale`: Locale support files
* `man`: Manual pages
* `sql`: SQL files
* `share`: Other architecture-independent support files

This layout reduces the cognitive overhead for understanding what files belong
Expand All @@ -98,7 +98,7 @@ Everything is in the `widget` directory.

### Configuration Parameters

Add three new `pg_config` values (names are preliminary):
Add three new `pg_config` values:

```
--extdir-core show location of core extensions
Expand All @@ -112,7 +112,7 @@ described [above](#extension-directories). Its contents would look something
like this:

``` console
❯ ls -1 "$(pg_config --extdir)"
❯ ls -1 "$(pg_config --extdir-core)"
auto_explain
bloom
intagg
Expand All @@ -124,10 +124,7 @@ xml2

Any OS vendor or packaging systems would install non-core extensions into
`--extdir-vendor`, while end-user extensions would be installed into
`--extdir-site`. [PGXS] will be updated to install into `--extdir-site` by
default, with an option to specify the vendor directory (used by packagers) or
any other directory. External projects that install extensions without using
PGXS (like [pgrx]) should do the same.
`--extdir-site`.

Like all other `pg_config` options, these values can be customized at compile
time. By default, the each should point to different directories, so that
Expand All @@ -140,10 +137,9 @@ PG_INSTALL_ROOT/extensions/(core|site|vendor)

### Extension Path

Add an extension lookup path akin to [`dynamic_library_path`]. For the
purposes of this RFC, let's call it `extension_path`. It lists all the
directories that Postgres should search for extensions and their files. The
default value for this GUC would be:
Add an extension lookup path GUC akin to [`dynamic_library_path`], called
`extension_path`. It lists all the directories that Postgres should search for
extensions and their files. The default value for this GUC would be:

``` ini
extension_path = '$extdir_site,$extdir_vendor,$extdir_core'
Expand All @@ -152,7 +148,7 @@ extension_path = '$extdir_site,$extdir_vendor,$extdir_core'
The special values `$extdir_site`, `$extdir_vendor`, and `$extdir_core`
correspond to `pg_config` `--extdir-site`, `--extdir-vendor`, and
`--extdir-core` options, respectively, and function exactly as `$libdir` does
for the `dynamic_library_path` GUC.
for the `dynamic_library_path` GUC, substituting the appropriate values.

### Lookup Execution

Expand All @@ -164,10 +160,10 @@ look for the extension control file in a directory named for the extension:
$dir/$extension/$extension.control
```

The first one it finds should thereafter be considered the canonical location
for the extension. For example, if the control file for the `pair` extension
was found at `/opt/pg17/ext/pair/pair.control`, then Postgres must load files
only from the appropriate subdirectories, e.g.:
The first one it finds should be considered the canonical location for the
extension. For example, if the control file for the `pair` extension was found
at `/opt/pg17/ext/pair/pair.control`, then Postgres must load files only from
the appropriate subdirectories, e.g.:

* SQL files from `/opt/pg17/ext/pair/sql`
* Library files from `/opt/pg17/ext/pair/lib`
Expand All @@ -180,12 +176,14 @@ into which an extension will be installed, and will default to
`--extdir-site`. It can be set to the literal values `$extdir_site`,
`$extdir_vendor`, or `$extdir_core`, or to any path.

The installation behavior will be changed for the following variables:
The `$EXTENSION` variable will be changed to allow only one extension name. If
it's set, the installation behavior will be changed for the following
variables:

* `EXTENSION`: Creates `$EXTDIR/$EXTENSION`, installs
`$EXTDIR/$EXTENSION/$EXTENSION.control`
* `MODULES` and `MODULE_big`: Installed into `$EXTDIR/$EXTENSION/lib`
* `MODULEDIR` Removed
* `MODULEDIR`: Removed
* `DATA`: Installed into `$EXTDIR/$EXTENSION/sql` if end in `.sql`,
otherwise into `$EXTDIR/$EXTENSION/share`
* `SQL`: New variable, like `DATA` but just for SQL files
Expand All @@ -196,13 +194,23 @@ The installation behavior will be changed for the following variables:
`$EXTDIR/$EXTENSION/bin`

Another new variable, `LINKBINS`, would default to true and symlink
`$EXTDIR/$EXTENSION/bin` files to `pg_config --bindir`. Installers can be set
`$EXTDIR/$EXTENSION/bin` files in `pg_config --bindir`. Installers can be set
to false to skip the symlinking, e.g., for immutable Postgres installs.

> [!NOTE]
> External projects that install extensions without using PGXS, like [pgrx],
> should be updated to either follow the same pattern or to delegate
> installation to [PGXS].
### MODULE_PATHNAME

Update the installer to replace `MODULE_PATHNAME` in SQL scripts with the new
install path for shared modules, `$EXTDIR/$EXTENSION/lib`.

### Control File

The `directory` and `module_pathname` control file variables would be removed
and ignored.
The `directory` and `module_pathname` control file variables would be
deprecated and ignored.

## Use Cases

Expand All @@ -211,9 +219,8 @@ the [use cases that have driven it](#the-problems).

### Packager Testing

Rather than [patching][destdir] Postgres to look up `pg_config` directories
under a prefix, a packager who wants to run tests without modifying a
PostgreSQL install would follow these steps:
A packager who wants to run tests without modifying a PostgreSQL install would
follow these steps:

* Set the `extension_path` GUC to search the site extension directory under
the packaging install. Something like
Expand All @@ -228,14 +235,14 @@ The Postgres installation will not have been modified, only the

### Postgres.app

To allow extension installation without invalidating the Postgresl.app bundle
To allow extension installation without invalidating the Postgres.app bundle's
signature, the app would be compiled to have `--extdir-site` point to a
well-known directory outside the app bundle.

Any extensions installed by the user would be placed in that directory,
avoiding any change to the Postgres.app bundle. Postgres would know to find
extensions in that location thanks to the inclusion of `$extdir_site` in the
`extension_path` GUC.
without changing the contents of the Postgres.app bundle. Postgres.app would
know to find extensions in that location thanks to the inclusion of
`$extdir_site` in the `extension_path` GUC.

### Docker/Kubernetes

Expand Down Expand Up @@ -264,28 +271,27 @@ citext
├── lib
│ ├── citext.dylib
│ └── bitcode
│ ├── citext
│ │ └── citext.bc
│ └── citext.index.bc
│ ├── citext
│ │ └── citext.bc
│ └── citext.index.bc
└── sql
├── citext--1.0--1.1.sql
├── citext--1.1--1.2.sql
├── citext--1.2--1.3.sql
├── citext--1.3--1.4.sql
├── citext--1.4--1.5.sql
├── citext--1.4.sql
└── citext--1.5--1.6.sql
├── citext--1.0--1.1.sql
├── citext--1.1--1.2.sql
├── citext--1.2--1.3.sql
├── citext--1.3--1.4.sql
├── citext--1.4--1.5.sql
├── citext--1.4.sql
└── citext--1.5--1.6.sql
```

Third-party extensions would live in one or more other directories on the file
system, generally in the `--extdir-site` directory for user-installed
extensions, and `--extdir-vendor` for OS-packaged extensions. But they may be
installed anywhere, as long as the `extension_path` GUC points to them and
they're accessible to/owned by the Postgres system account.
Third-party extensions would live in other directories on the file system,
generally in the `--extdir-site` directory for user-installed extensions, and
`--extdir-vendor` for OS-packaged extensions. But they may be installed
anywhere, as long as the `extension_path` GUC points to them and they're
accessible to/owned by the Postgres system account.

Say that `--extdir-site` is set to `/opt/pgxn`. Within that directory, we
might have a directory for a pure SQL extension in a directory named “pair”
that looks like this:
Within the install directory, we might have a subdirectory for a pure SQL
extension in a directory named “pair” that looks like this:

``` tree
pair
Expand All @@ -297,8 +303,8 @@ pair
│ │ └── pair.html
│ └── pair.md
└── sql
├── pair--1.0--1.1.sql
└── pair--1.1.sql
├── pair--1.0--1.1.sql
└── pair--1.1.sql
```

A binary application like [pg_top] would live in the `pg_top` directory,
Expand All @@ -311,11 +317,11 @@ pg_top
├── LICENSE
├── README.rst
├── bin
| └── pg_top
│ └── pg_top
└── doc
└── man
└── man3
└── pg_top.3
└── man
└── man3
└── pg_top.3
```

And a C extension like [semver] would live in the semver directory and be
Expand All @@ -330,27 +336,27 @@ semver
│ └── semver.md
├── lib
│ ├── semver.dylib
│ ├── bitcode
│ └── semver
│ │ └── semver.bc
│ └── semver.index.bc
│ └── bitcode
│ ├── semver
│ │ └── semver.bc
│ └── semver.index.bc
└── sql
├── semver--1.0--1.1.sql
└── semver--1.1.sql
├── semver--1.0--1.1.sql
└── semver--1.1.sql
```

## Phase Two: Preloading

The [solution](#solution) proposed above does not allow extension modules to
compatibly be loaded via [shared library preloading], because extension
modules will be installed in extension directories and no longer in the
[`dynamic_library_path`]. Users can use the full path; instead of
[`dynamic_library_path`]. Users can use the full path; for example, instead of

``` ini
shared_preload_libraries = 'pg_partman_bgw'
```

One could use the full path:
One would use:

```ini
shared_preload_libraries = '/opt/postgres/extensions/pg_partman_bgw/lib/pg_partman_bgw'
Expand Down Expand Up @@ -380,8 +386,8 @@ module preloading:
* `local_preload_extensions`

Each takes a list of extensions for which to preload shared modules. In
addition, another new GUC, `local_plugins`, would be an administrator-only GUC
that contains a list of extensions allowed in `local_preload_extensions`. This
addition, another new GUC, `local_plugins`, would contains a list of
administrator-approved extensions allowed in `local_preload_extensions`. This
would complement [`local_preload_libraries`]'s use of a `plugins` directory.

Then modify the preloading code to also preload these files. For each
Expand Down Expand Up @@ -419,41 +425,29 @@ auto_explain
├── auto_explain.control
└── lib
├── auto_explain.dylib
├── bitcode
└── auto_explain
│ └── auto_explain.bc
└── auto_explain.index.bc
└── bitcode
├── auto_explain
│ └── auto_explain.bc
└── auto_explain.index.bc
```

Note the `auto_explain.control` file. We would need to add a variable to
indicate that the extension includes no SQL files, so `CREATE EXTENSION` and
related commands wouldn't try to find them.

With these changes, extensions could become the primary, recommended interface
for extensing PostgreSQL. Perhaps the `LOAD` command could be deprecated, and
for extending PostgreSQL. Perhaps the `LOAD` command could be deprecated, and
the `*_preload_libraries` GUCs along with it.

## Future Changes

* `LOAD`-only extension control files

* Some extensions, such as [pg_hint_plan], include both a `CREATE
EXTENSION` extension and a `LOAD` module, and both have the same name.

## Open Issues

* How will `dynamic_library_path` and `*_preload_libraries` find DSOs in
`extension_path`? Possibilities. Perhaps include the `$extdir_*` variables
in `dynamic_library_path` and use `$extension/$lib/$extension` as the
preload value?

## Compatibility Issues

* The `directory` control file variable and the `MODULEDIR` PGXS variable
would be ignored.
* `dynamic_library_path` would no longer be used to find extension
libraries, though perhaps Postgres would fall back on the old search
pattern if it didn't find an extension in the extension paths.
* The The `directory` and `module_pathname` control file variables and the
`MODULEDIR` PGXS variable would be deprecated and ignored.
* `*_preload_libraries` would no longer be used to find extension modules.
Administrators would have to move extensions listed in those GUCs to the
new `*_preload_extensions` variables.
* `LOAD` would no longer be able to find shared modules included with
extensions.

## Out of Scope

Expand Down Expand Up @@ -490,3 +484,5 @@ This RFC does not include or attempt to address the following issues:
"PostgreSQL Docs: Shared Library Preloading"
[`local_preload_libraries`]: https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCAL-PRELOAD-LIBRARIES
"PostgreSQL Docs: local_preload_libraries"
[`LOAD`]: https://www.postgresql.org/docs/17/sql-load.html
"PostgreSQL Docs: LOAD"

0 comments on commit 8a88d04

Please sign in to comment.