From 8a88d0411cd4f539938cf0d0ea248fa1ff2682b7 Mon Sep 17 00:00:00 2001 From: "David E. Wheeler" Date: Fri, 1 Nov 2024 18:45:26 -0400 Subject: [PATCH] Revisions --- .../rfc-extension-packaging-lookup.md | 184 +++++++++--------- 1 file changed, 90 insertions(+), 94 deletions(-) diff --git a/content/post/postgres/rfc-extension-packaging-lookup.md b/content/post/postgres/rfc-extension-packaging-lookup.md index b949141d..d15fa26c 100644 --- a/content/post/postgres/rfc-extension-packaging-lookup.md +++ b/content/post/postgres/rfc-extension-packaging-lookup.md @@ -22,9 +22,9 @@ more unified, public proposal. ## The Problems A number of challenges face extension users in various configurations, thanks -to the status quo of extension file organization in the Postgres core. The -common thread among them is the need to add extensions without changing the -Postgres installation itself. +to extension file organization in the Postgres core. The common thread among +them is the need to add extensions without changing the Postgres installation +itself. ### Packager Testing @@ -44,9 +44,9 @@ directly from the package build directory. Alas, the patch isn't ideal, because it simply specifies a prefix and appends the full `pg_config` directory paths to it. For example, if `--sharedir` outputs `/opt/share` and `extension_destdir` GUC is set to `/tmp/build/myext`, -the patch will search in `/tmp/build/myext/opt/share`. This approach well for -the packaging use case, which explicitly uses a prefix pattern, but would be a -bit weird for other use cases. +the patch will search in `/tmp/build/myext/opt/share`. This approach works for +the packaging use case, which explicitly uses full paths with a prefix, but +would be a bit weird for other use cases. ### Docker Immutability @@ -64,9 +64,9 @@ deployment configuration complexity. ### Postgres.app Immutability The macOS [Postgres.app] supports extensions. But installing one into -[SHAREDIR/extensions] changes the contents of the Postgres.app bundle, +`SHAREDIR/extensions` changes the contents of the Postgres.app bundle, breaking the app's Apple-required signature validation. The OS will no longer -be able to validate that the app is legit, so will refuse to start it. +be able to validate that the app is legit and refuse to start it. ## Solution @@ -83,13 +83,13 @@ single directory. These include: Subdirectories roughly correspond to the `pg_config --*dir` options: -* `sql`: SQL files * `bin`: Executables * `doc`: Documentation files * `html`: HTML documentation files * `lib`: Dynamically loadable modules * `locale`: Locale support files * `man`: Manual pages +* `sql`: SQL files * `share`: Other architecture-independent support files This layout reduces the cognitive overhead for understanding what files belong @@ -98,7 +98,7 @@ Everything is in the `widget` directory. ### Configuration Parameters -Add three new `pg_config` values (names are preliminary): +Add three new `pg_config` values: ``` --extdir-core show location of core extensions @@ -112,7 +112,7 @@ described [above](#extension-directories). Its contents would look something like this: ``` console -❯ ls -1 "$(pg_config --extdir)" +❯ ls -1 "$(pg_config --extdir-core)" auto_explain bloom intagg @@ -124,10 +124,7 @@ xml2 Any OS vendor or packaging systems would install non-core extensions into `--extdir-vendor`, while end-user extensions would be installed into -`--extdir-site`. [PGXS] will be updated to install into `--extdir-site` by -default, with an option to specify the vendor directory (used by packagers) or -any other directory. External projects that install extensions without using -PGXS (like [pgrx]) should do the same. +`--extdir-site`. Like all other `pg_config` options, these values can be customized at compile time. By default, the each should point to different directories, so that @@ -140,10 +137,9 @@ PG_INSTALL_ROOT/extensions/(core|site|vendor) ### Extension Path -Add an extension lookup path akin to [`dynamic_library_path`]. For the -purposes of this RFC, let's call it `extension_path`. It lists all the -directories that Postgres should search for extensions and their files. The -default value for this GUC would be: +Add an extension lookup path GUC akin to [`dynamic_library_path`], called +`extension_path`. It lists all the directories that Postgres should search for +extensions and their files. The default value for this GUC would be: ``` ini extension_path = '$extdir_site,$extdir_vendor,$extdir_core' @@ -152,7 +148,7 @@ extension_path = '$extdir_site,$extdir_vendor,$extdir_core' The special values `$extdir_site`, `$extdir_vendor`, and `$extdir_core` correspond to `pg_config` `--extdir-site`, `--extdir-vendor`, and `--extdir-core` options, respectively, and function exactly as `$libdir` does -for the `dynamic_library_path` GUC. +for the `dynamic_library_path` GUC, substituting the appropriate values. ### Lookup Execution @@ -164,10 +160,10 @@ look for the extension control file in a directory named for the extension: $dir/$extension/$extension.control ``` -The first one it finds should thereafter be considered the canonical location -for the extension. For example, if the control file for the `pair` extension -was found at `/opt/pg17/ext/pair/pair.control`, then Postgres must load files -only from the appropriate subdirectories, e.g.: +The first one it finds should be considered the canonical location for the +extension. For example, if the control file for the `pair` extension was found +at `/opt/pg17/ext/pair/pair.control`, then Postgres must load files only from +the appropriate subdirectories, e.g.: * SQL files from `/opt/pg17/ext/pair/sql` * Library files from `/opt/pg17/ext/pair/lib` @@ -180,12 +176,14 @@ into which an extension will be installed, and will default to `--extdir-site`. It can be set to the literal values `$extdir_site`, `$extdir_vendor`, or `$extdir_core`, or to any path. -The installation behavior will be changed for the following variables: +The `$EXTENSION` variable will be changed to allow only one extension name. If +it's set, the installation behavior will be changed for the following +variables: * `EXTENSION`: Creates `$EXTDIR/$EXTENSION`, installs `$EXTDIR/$EXTENSION/$EXTENSION.control` * `MODULES` and `MODULE_big`: Installed into `$EXTDIR/$EXTENSION/lib` -* `MODULEDIR` Removed +* `MODULEDIR`: Removed * `DATA`: Installed into `$EXTDIR/$EXTENSION/sql` if end in `.sql`, otherwise into `$EXTDIR/$EXTENSION/share` * `SQL`: New variable, like `DATA` but just for SQL files @@ -196,13 +194,23 @@ The installation behavior will be changed for the following variables: `$EXTDIR/$EXTENSION/bin` Another new variable, `LINKBINS`, would default to true and symlink -`$EXTDIR/$EXTENSION/bin` files to `pg_config --bindir`. Installers can be set +`$EXTDIR/$EXTENSION/bin` files in `pg_config --bindir`. Installers can be set to false to skip the symlinking, e.g., for immutable Postgres installs. +> [!NOTE] +> External projects that install extensions without using PGXS, like [pgrx], +> should be updated to either follow the same pattern or to delegate +> installation to [PGXS]. + +### MODULE_PATHNAME + +Update the installer to replace `MODULE_PATHNAME` in SQL scripts with the new +install path for shared modules, `$EXTDIR/$EXTENSION/lib`. + ### Control File -The `directory` and `module_pathname` control file variables would be removed -and ignored. +The `directory` and `module_pathname` control file variables would be +deprecated and ignored. ## Use Cases @@ -211,9 +219,8 @@ the [use cases that have driven it](#the-problems). ### Packager Testing -Rather than [patching][destdir] Postgres to look up `pg_config` directories -under a prefix, a packager who wants to run tests without modifying a -PostgreSQL install would follow these steps: +A packager who wants to run tests without modifying a PostgreSQL install would +follow these steps: * Set the `extension_path` GUC to search the site extension directory under the packaging install. Something like @@ -228,14 +235,14 @@ The Postgres installation will not have been modified, only the ### Postgres.app -To allow extension installation without invalidating the Postgresl.app bundle +To allow extension installation without invalidating the Postgres.app bundle's signature, the app would be compiled to have `--extdir-site` point to a well-known directory outside the app bundle. Any extensions installed by the user would be placed in that directory, -avoiding any change to the Postgres.app bundle. Postgres would know to find -extensions in that location thanks to the inclusion of `$extdir_site` in the -`extension_path` GUC. +without changing the contents of the Postgres.app bundle. Postgres.app would +know to find extensions in that location thanks to the inclusion of +`$extdir_site` in the `extension_path` GUC. ### Docker/Kubernetes @@ -264,28 +271,27 @@ citext ├── lib │ ├── citext.dylib │ └── bitcode -│ ├── citext -│ │ └── citext.bc -│ └── citext.index.bc +│ ├── citext +│ │ └── citext.bc +│ └── citext.index.bc └── sql - ├── citext--1.0--1.1.sql - ├── citext--1.1--1.2.sql - ├── citext--1.2--1.3.sql - ├── citext--1.3--1.4.sql - ├── citext--1.4--1.5.sql - ├── citext--1.4.sql - └── citext--1.5--1.6.sql + ├── citext--1.0--1.1.sql + ├── citext--1.1--1.2.sql + ├── citext--1.2--1.3.sql + ├── citext--1.3--1.4.sql + ├── citext--1.4--1.5.sql + ├── citext--1.4.sql + └── citext--1.5--1.6.sql ``` -Third-party extensions would live in one or more other directories on the file -system, generally in the `--extdir-site` directory for user-installed -extensions, and `--extdir-vendor` for OS-packaged extensions. But they may be -installed anywhere, as long as the `extension_path` GUC points to them and -they're accessible to/owned by the Postgres system account. +Third-party extensions would live in other directories on the file system, +generally in the `--extdir-site` directory for user-installed extensions, and +`--extdir-vendor` for OS-packaged extensions. But they may be installed +anywhere, as long as the `extension_path` GUC points to them and they're +accessible to/owned by the Postgres system account. -Say that `--extdir-site` is set to `/opt/pgxn`. Within that directory, we -might have a directory for a pure SQL extension in a directory named “pair” -that looks like this: +Within the install directory, we might have a subdirectory for a pure SQL +extension in a directory named “pair” that looks like this: ``` tree pair @@ -297,8 +303,8 @@ pair │ │ └── pair.html │ └── pair.md └── sql -├── pair--1.0--1.1.sql -└── pair--1.1.sql + ├── pair--1.0--1.1.sql + └── pair--1.1.sql ``` A binary application like [pg_top] would live in the `pg_top` directory, @@ -311,11 +317,11 @@ pg_top ├── LICENSE ├── README.rst ├── bin -| └── pg_top +│ └── pg_top └── doc - └── man - └── man3 - └── pg_top.3 + └── man + └── man3 + └── pg_top.3 ``` And a C extension like [semver] would live in the semver directory and be @@ -330,13 +336,13 @@ semver │ └── semver.md ├── lib │ ├── semver.dylib -│ ├── bitcode -│ └── semver -│ │ └── semver.bc -│ └── semver.index.bc +│ └── bitcode +│ ├── semver +│ │ └── semver.bc +│ └── semver.index.bc └── sql - ├── semver--1.0--1.1.sql - └── semver--1.1.sql + ├── semver--1.0--1.1.sql + └── semver--1.1.sql ``` ## Phase Two: Preloading @@ -344,13 +350,13 @@ semver The [solution](#solution) proposed above does not allow extension modules to compatibly be loaded via [shared library preloading], because extension modules will be installed in extension directories and no longer in the -[`dynamic_library_path`]. Users can use the full path; instead of +[`dynamic_library_path`]. Users can use the full path; for example, instead of ``` ini shared_preload_libraries = 'pg_partman_bgw' ``` -One could use the full path: +One would use: ```ini shared_preload_libraries = '/opt/postgres/extensions/pg_partman_bgw/lib/pg_partman_bgw' @@ -380,8 +386,8 @@ module preloading: * `local_preload_extensions` Each takes a list of extensions for which to preload shared modules. In -addition, another new GUC, `local_plugins`, would be an administrator-only GUC -that contains a list of extensions allowed in `local_preload_extensions`. This +addition, another new GUC, `local_plugins`, would contains a list of +administrator-approved extensions allowed in `local_preload_extensions`. This would complement [`local_preload_libraries`]'s use of a `plugins` directory. Then modify the preloading code to also preload these files. For each @@ -419,10 +425,10 @@ auto_explain ├── auto_explain.control └── lib ├── auto_explain.dylib - ├── bitcode - └── auto_explain - │ └── auto_explain.bc - └── auto_explain.index.bc + └── bitcode + ├── auto_explain + │ └── auto_explain.bc + └── auto_explain.index.bc ``` Note the `auto_explain.control` file. We would need to add a variable to @@ -430,30 +436,18 @@ indicate that the extension includes no SQL files, so `CREATE EXTENSION` and related commands wouldn't try to find them. With these changes, extensions could become the primary, recommended interface -for extensing PostgreSQL. Perhaps the `LOAD` command could be deprecated, and +for extending PostgreSQL. Perhaps the `LOAD` command could be deprecated, and the `*_preload_libraries` GUCs along with it. -## Future Changes - -* `LOAD`-only extension control files - -* Some extensions, such as [pg_hint_plan], include both a `CREATE - EXTENSION` extension and a `LOAD` module, and both have the same name. - -## Open Issues - -* How will `dynamic_library_path` and `*_preload_libraries` find DSOs in - `extension_path`? Possibilities. Perhaps include the `$extdir_*` variables - in `dynamic_library_path` and use `$extension/$lib/$extension` as the - preload value? - ## Compatibility Issues -* The `directory` control file variable and the `MODULEDIR` PGXS variable - would be ignored. -* `dynamic_library_path` would no longer be used to find extension - libraries, though perhaps Postgres would fall back on the old search - pattern if it didn't find an extension in the extension paths. +* The The `directory` and `module_pathname` control file variables and the + `MODULEDIR` PGXS variable would be deprecated and ignored. +* `*_preload_libraries` would no longer be used to find extension modules. + Administrators would have to move extensions listed in those GUCs to the + new `*_preload_extensions` variables. +* `LOAD` would no longer be able to find shared modules included with + extensions. ## Out of Scope @@ -490,3 +484,5 @@ This RFC does not include or attempt to address the following issues: "PostgreSQL Docs: Shared Library Preloading" [`local_preload_libraries`]: https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCAL-PRELOAD-LIBRARIES "PostgreSQL Docs: local_preload_libraries" + [`LOAD`]: https://www.postgresql.org/docs/17/sql-load.html + "PostgreSQL Docs: LOAD"