Skip to content

Commit

Permalink
[wip] to be squashed over the previous one
Browse files Browse the repository at this point in the history
  • Loading branch information
@AndersonTorres
AndersonTorres committed Nov 5, 2024
1 parent b84ab23 commit b4f3a94
Showing 1 changed file with 16 additions and 14 deletions.
30 changes: 16 additions & 14 deletions doc/hooks/versionCheckHook.section.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,19 +70,21 @@ Given the current limitations of Nixpkgs's tooling and CI, `versionCheckHook` fi
Below we tabulate the differences between `versionCheckHook` and `testers.testVersion`.


| Item | `versionCheckHook` | `testers.testVersion` |
|:---------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
| Customization | Besides the attributes described above, the hook provides no other methods for controlling it. | The tester has access to all feature set of Nix and Nixpkgs, like generating multiple version test derivations from a list of strings, and accessing programs / tools outside the inputs of the derivation. |
| Execution environment | The hook uses `env --ignore-environment --chdir=/` to run the executables in an environment as clean as possible, but there is no way to change its behavior. | The tester executes in an identical environment of a consumer, independent from the building environment. |
| Rebuild after modification | The hook rebuilds the package when the hook is modified, since it is a phase running during the build time of the package. | The tester does not require rebuilding the package, since it is a derivation. |
| Overhead during package building | Negligible. Although running during the build time of the package, the hook is lean and executes few commands. | Zero, since the tester is a derivation that runs after package building. |
| Content-addressed-derivation awareness | Since it runs during the building, the hook does not deal with failures happening after building, like rewritings that happen post installation. | Since it runs after the building, the tester detects failures at this time. |
| Execution by OfBorg CI tool | Yes, since the hook is a phase running during the build time of the package. | OfBorg does run the tester for _directly affected packages only_; transitive dependencies are ignored by default, requiring extra commands like `@ofborg build dependency1.tests dependency2.tests ...`. |
| Execution by `nixpkgs-review` tool | Yes, since the hook is a phase running during the build time of the package. | No, since the tool has no support for executing `passthru.tests`. [^1] |
| Package breakage awareness | Loud and clear as soon as the hook is reached, since it is a phase running during the build time of the package. | Requires specific command to be noticed, e.g. `nix-build -A pkg.tests.version`, since it is a derivation dependent on the package |
| Transitive package breakage | Never ever goes unnoticed by `nixpkgs-review`, since the tool executes the transitive dependencies. | Since human beings are prone to forget the duty of running passthru tests and `nixpkgs-review` has no support for running it, it certainly goes unnoticed. |


[^1]: There is a [pull request proposal](https://github.com/Mic92/nixpkgs-review/pull/397) against [`nixpkgs-review`](https://github.com/Mic92/nixpkgs-review) for an extra flag to run `passthru.tests`.
|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Item | `versionCheckHook` | `testers.testVersion` via `passthru.tests.version` |
|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| Customization | Besides the attributes described above, `versionCheckHook` provides no other methods for controlling it. | `tests.version` has access to the whole feature set of Nixpkgs, like generating multiple test derivations and accessing tools outside the package inputs. |
| Execution environment | `versionCheckHook` uses `env --ignore-environment --chdir=/` to run the executables in an environment as clean as possible, but there is no way to change its behavior. | `tests.version` executes in an environment identical to that of a consumer, independent from the building environment. |
| Rebuild after modification | Modifying `versionCheckHook` triggers the package rebuild, since it is a phase executed during the the package build time. | `tests.version` does not require rebuilding the package, since it is a derivation that runs after package build time. |
| Overhead during package build time | Negligible. Although executed during the the package build time, `versionCheckHook` is lean and executes few commands. | Zero, since `tests.version` is a derivation that runs after package build time. |
| Failure detection time | `versionCheckHook` runs during the package build time, therefore it does not deal with failures happening after that. | `tests.version` detects failures after package build time, like rewritings and relocations. |
| OfBorg CI support | `versionCheckHook` will be executed by OfBorg, since it is a phase executed during the the package build time. | OfBorg supports automatic execution of `passthru.tests` for _directly affected packages_; transitive dependencies are ignored by default. |
| `nixpkgs-review` support | `versionCheckHook` will be executed by `nixpkgs-review`, since it is a phase executed during the the package build time. | `nixpkgs-review` has no support for executing `passthru.tests` at all, whether for the package or its transitive dependencies. [^1] |
| Package breakage awareness | Loud and clear as soon as `versionCheckHook` is reached during the package build time. | Requires specific commands to be noticed, e.g. `nix-build -A pkg.tests.version`. |
| Transitive dependencies awareness | Since `nixpkgs-review` executes the transitive dependencies, the ones containing `versionCheckHook` will never ever be ignored by this tool. | Since `nixpkgs-review` does not run `tests.version` and ofBorg does not execute transitive dependencies, extra measures are needed to find them. |
|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|


[^1]: There is a [pull request proposal](https://github.com/Mic92/nixpkgs-review/pull/397) against [`nixpkgs-review`](https://github.com/Mic92/nixpkgs-review) for adding support to `passthru.tests` execution.

As said before, given the current limitations of Nixpkgs's tooling and CI demonstrated above, `versionCheckHook` fits better in most of typical situations.

0 comments on commit b4f3a94

Please sign in to comment.