RPM v6 package format, first public draft for commenting

[pmatilai]

In the spirit of release early and often, here goes the (admittedly rough) first public draft of the rpm v6 package format that people have seen mentions of here and there. As you'll notice, this will be a relatively conservative update to the format, a facelift if you like, rather than a radical redesign.

Feedback and discussion are welcome and expected. We plan on keeping this thread open for several months to give the relevant parties sufficient time to participate. Please keep in mind that this draft is strictly about the concrete package format of .rpm files, and not a wishlist for rpm v6.0 features.

This initial post and the associated draft version number will be updated as the items get added and the plan grows more detailed.

---

RPM v6 package format (draft v0.2)

Summary

RPM v6 is a face-lift on the existing RPM package v4 format, not a radical
redesign. The primary motivations for v6 are to bring RPM to this millenium
technology-wise: eliminate long obsolete crypto, ensure sufficiently large
sizes everywhere and shed compatibility babbage from the last 20 years.

The following mainly describes differences to the undocumented v4 format,
the v6 format has to, and will be properly and thoroughly documented.

"Dropped" in the below means: no longer generated or accepted in the v6
format itself. As part of v4 packages, they will remain accepted as long
as v4 is relevant (ie, a long long time).

The last rpm 4.x version will be able to read + install v6 packages.
Rpm v6 will be able to read and install v4 packages, but v3 support
will be dropped.

The fundamental file-format remains the same: an rpm package consists of

• lead
• signature header
• main header
• optional payload, optionally compressed

Headers

• tag entries are always sorted
• tag numbers are unique across signature and main header
• duplicate tags are not permitted
• all padding must be zeros
• 64 bit sizes everywhere
  • package-level sizes
  • file sizes
• crypto modernization:
  • MD5 and SHA1 dropped everywhere
  • DSA1 dropped everywhere

Lead

• except for the rpm magic, the lead will be zeros only

Signature header

• header+payload digests and signatures (aka V3 signatures) are dropped
  (v3 signatures are no longer generated by default since 4.16)
  • RPMSIGTAG_MD5
  • RPMSIGTAG_PGP
  • RPMSIGTAG_GPG
  • RPMSIGTAG_PGP5
• size tags are dropped:
  • RPMSIGTAG_SIZE
  • RPMSIGTAG_LONGSIZE
  • RPMSIGTAG_PAYLOADSIZE
  • RPMSIGTAG_LONGARCHIVESIZE (move to main header or drop?)
• signature reservation moved to some other mechanism (TBD):
  • RPMSIGTAG_RESERVEDSPACE
• eliminate tag value overlap between signature and header tags - happens
  when all of the above is done
• drop RPMTAG_SHA1HEADER
• protect ima + fsverity signatures somehow (separate signature?)
• the signature header is self-padded, ie doesn't require special actions from reader

Main header

• file sizes are always 64bit:
  • RPMTAG_LONGFILESIZES
  • RPMTAG_LONGSIZE
  • RPMTAG_LONGARCHIVESIZE
• 32bit size tags are dropped:
  • RPMTAG_SIZE
  • RPMTAG_FILESIZES
  • RPMTAG_ARCHIVESIZE
• always utf-8 encoded
• group tag is made optional
• compression?

Payload

• always in the "new" large file format, never cpio
• reflect this in the PAYLOADFORMAT tag too

Metadata level

• rpmlib() dependencies are reset (ideally they'd be replaced by a better
  mechanism but that's probably out of scope)
• proper multiarch dependencies (instead of ()(64bit) markers)

Open questions

• how to irrevocably set and detect v6 format? must not be "downgradeable" to v4 for more relaxed checks

[DemiMarie]

Could the signature also cover something indicating the package version?

[pmatilai]

That's a good question. Starting with, how exactly is the package version determined? If we assume stricter header layout for v6 then it can't really be in a header tag because we need to know the format before parsing it. A new header magic I guess, but that feels heavy-handed when the header format is for all practical purposes still the same. But then, it may be well worth it simply for the clean break -effect.

[lnussel]

Is there an actual need to have any extra headers in the payload? The payload could be just raw file content. https://github.com/lnussel/rpm2extents does that in a page aligned way but could also be back-to-back and compressed of course.

[pmatilai]

The file order in payload doesn't match the one in the header, at least for hardlinks and other stuff that may not have content at all. Maybe in a perfect world one could expect that order to be strictly defined and always correct, but there are multiple cases in rpm history where one bug or the other has led into order changing or things like %ghost payload in the header where it's not expected. So I for one will sleep a lot better knowing there's a mapping from payload to the header.

[lnussel]

The file order in payload doesn't match the one in the header, at least for hardlinks and other stuff that may not have content at all. Maybe in a perfect world one could expect that order to be strictly defined and always correct, but there are multiple cases in rpm history where one bug or the other has led into order changing or things like %ghost payload in the header where it's not expected. So I for one will sleep a lot better knowing there's a mapping from payload to the header.

The point would be to not allow arbitrary ordering anymore. If if the payload is a raw data area, there is no way to deviate from the header. So instead of allowing to refer back and forth between payload and header we only allow pointing from the header into the payload. The rules are rather simple:

• the order of file data is the same order as those of the file related arrays
• offset and length of the data for each file are calculated based based on the FILESIZES array
• skip non regular files (FILEMODES)
• skip ghost files aka the marker that the payload does not contain the file's data (FILEFLAGS)
• each unique inode only once aka don't duplicate hardlinked data (FILEINODES)

If at the same time we restrict FILEINODES to numbers from 0 to max nfiles-1, then no extra inode search/lookup would be needed either.

[pmatilai]

Yep, of course such a format is possible to design. Just saying that this is not in our plans. The scope of change in the v6 format is intentionally quite restricted to avoid it becoming a forever project.

[smooge]

I found this from the rpm.org website on a completely different thing. Will the discussion be only held here or is it on a mailing list also? [And sorry if my questions is obvious but missed]

[pmatilai]

GH ticket, PR and discussions activity are gated to rpm-maint (eg start of this thread is here, and AFAIK replying via email does work. We've no intention to have a separate discussion on any mailing list, but those not willing to be on GH for any reason should be able to participate nevertheless.

[smooge]

No problem. I just didn't know if I should subscribe to a list like the old rpm list. I can track this ticket.

[DemiMarie]

Would checking that padding is zeroed be a part of this? What about banning dribbles from the signature header?

[DemiMarie]

You’re welcome. What about padding between one tag and the next?

Personally, I think that using a dribble for the padding is a fine, provided that:

1. No other tags are allowed in the dribble.
2. Regions are checked to be consistent.
3. The padding is required to be in the dribble.
4. The padding is ignored by hdrblobImport().

That said, the current solution is simpler and much more obvious from the perspective of package reading.

[pmatilai]

What about padding between one tag and the next?

See above, "you'll have an explicit requirement for all padding to be zeros in v6" (emphasis added)

That said, the current solution is simpler and much more obvious from the perspective of package reading.

In some ways yeah, but the padding is an annoying wrinkle that requires separate functions/special cases to read what is supposed to be just two headers in a row. So a self-padding signature header (whatever the exact means) would be a nice simplification for the reader.

Whether dribbles or not etc, I totally agree there needs to be exactly one accepted way.

[SamuelWAnderson45]

What does "dribble" mean in this context?

[dralley]

Relevant #2462

[pmatilai]

FWIW, dribbles are now (briefly) documented in https://rpm-software-management.github.io/rpm/manual/format_header.html#immutable-regions

[Conan-Kudo]

Could we adjust the RPM format so that things like multi-arch RPMs (i.e. RPMs for macOS containing binaries that are universal binaries) could be correctly indicated? That is, instead of assuming that we have a single "arch" for the whole package, we identify the arches per file and collect the arches for identifying system compatibility?

[pmatilai]

I'm all for properly supporting multi-arch in packages in principle (the multiarch deps are even mentioned in the v6 spec here), but getting rid of "arch" is likely to be, uh, complicated. Even if rpm itself could, the repodata and every single depsolver out there has very deeply wired assumptions about that. Very unlikely to happen in this timeframe.

[DemiMarie]

What about adding per-file architectures, and keeping the legacy "arch" purely for compatibility with old tools?

[pmatilai]

Add more of the stuff that we want to get rid of? 😄

[Conan-Kudo]

Well, RPM has the "fat" arch for multi-arch RPMs already... So detecting arches per file and using "fat" as the rpmarch would work for me.

[Conan-Kudo]

Actually, maybe we should rename fat to multi instead and use that to inform RPM that it needs to process per-file architecture information. A new rpmarch would allow us to handle specifying the behavior for "universal binary" type RPMs, which are currently undefined.

[dralley]

Payload
always in the "new" large file format, never cpio
reflect this in the PAYLOADFORMAT tag too

Where can I read more about this?

[Conan-Kudo]

I said that mtimes for files in the payload are the only thing that actually matters for clamping to SOURCE_DATE_EPOCH. We absolutely don't want SOURCE_DATE_EPOCH affecting package buildtimes by default.

[dralley]

Sorry for the miscommunication. The question I had started out with was

but the point of source_date_epoch is to make packages reproducable right? If you allow the buildtime to be different, then the package is no longer reproducable, and there's no reason to use source_date_epoch

You pointed out that the purpose of use_source_date_epoch_as_buildtime is that various infrastructure needs a meaningful, non-static buildtime to function properly, but that there is still value in having a reproducible payload even if the rest of the package is not bit-identical. In order to get payload reproducibility you need to clamp the mtimes of the files using SOURCE_DATE_EPOCH, because that metadata is baked into the CPIO. So you want to specify SOURCE_DATE_EPOCH but keep the buildtime independent, and that requires having a configuration option - use_source_date_epoch_as_buildtime.

My point is that if the RPMv6 payload format doesn't include that mtime metadata in the archive metadata, then you can have payload reproducibility all the time and don't need to set SOURCE_DATE_EPOCH to get it. Therefore, if you want full reproducibility you could specify SOURCE_DATE_EPOCH, and if you don't want it, then you can skip SOURCE_DATE_EPOCH entirely, and you still get the property of having a reproducible payload regardless. You can collapse a bunch of options into just "do I want a fully reproducible package, or not".

That would also bring what rpmbuild does into closer alignment with the SOURCE_DATE_EPOCH specification, which states that "Build processes MUST use this variable for embedded timestamps in place of the "current" date and time." If it's set, it's supposed to be used. That's not practical now for all of the above reasons, so rpmbuild doesn't follow it strictly. But in theory RPMv6 could eliminate the practical issues and allow this to be followed strictly.

Granted as this behavior has now been in use for a while, it may not be easy to remove or change by default. I'm just saying that in theory, it could be simplified.

[pmatilai]

The "new" payload format doesn't carry the per-file info in the payload like cpio does, but we of course still store the file mtime in the header. Those aren't going anywhere at all, and so none of the source-date-epoch stuff is affected or even related.

[dralley]

but we of course still store the file mtime in the header.

Right, I understand, no disagreement there. If you need a fully reproducible package, you need to set source-date-epoch.

But if you don't need or even want a fully reproducible package (but you still want the payload to be reproducible), you no longer need to set source-date-epoch, whereas currently you do (and also a bunch of flags to disable aspects of it). It's all of those flags for disabling aspects of source-date-epoch which become potentially unnecessary.

[dralley]

@Conan-Kudo Is this making any sense?

[dralley]

except for the rpm magic, the lead will be zeros only

I'm curious if we want to apply this even to the "major" file format version number in the lead? Of all the fields, that one seems like it could eventually have some potential use.

[pmatilai]

See the very first comment in this topic from @DemiMarie - the format version should be signed to prevent changes. I suppose the lead could be signed, but I'm not sure we want to resurrect to repurpose this 25+ years old dead structure here.

[pmatilai]

Coming back to this: to avoid gratuitious compatibility breakage, we can't put either 0 or 6 in there.
The best we can do is put 4 in there because all of rpm 4.x used 3 as the major version for alleged LSB compatibility, so it at least differentiates it a bit. It's not such a terrible lie even because the fundamental header structure of v6 is basically v4...

[dralley]

Why can't 6 go there?

[pmatilai]

Because in rpm < 4.18, we have this silly check in there:

    if (lead->major < 3 || lead->major > 4) {
        *msg = xstrdup(_("unsupported RPM package version"));
        return RPMRC_FAIL;
    }

Putting 6 in the otherwise unused lead would render v6 packages unreadable by a huge range of older rpm versions that have otherwise zero problems accessing it, with precisely zero benefit to rpm. It's just not worth it.

[j-mracek]

What about inconsistency between repo metadata and in RPMDB with package checksums. Metadata contains file checksum (only one type), but RPMDB contains a header checksum (multiple types). We have some users who wants to align available packages with installed one and there is not enough information stored in DNF, RPM, or repo metadata to do it. NEVRA, repoid is not enough.

Repository metadata must contain file checksum for verification as well as the size of file. Adding additional checksum would be possible, but expensive for distribution (compatibility could be also affected).

There is possible solution on DNF side - store both checksum types in DNF5 system state, but that information will be not available for packages installed directly by RPM or by Zypper.

What kind of option do we have from RPM side?

[pmatilai]

An rpm cannot obviously carry a checksum on itself (for matching what's in the repodata). I think it's long overdue the repo metadata is overhauled to address the various defiencies in it, but that's another topic.

[j-mracek]

Is there any planned change related to changelogs?

[pmatilai]

I don't see how rpm could help with repo metadata being non-extensible (despite initial intentions - it's quite ironical that the supposedly extensible xml became such an immovable obstacle).

As for changelogs, as you can see in this draft, no changes are planned.

[dralley]

What's the background on the issue you are referring to here?

[j-mracek]

If I am not mistaken - changelogs are stored in rpm header and they are uncompressed. Is there any reason to have them uncompressed?

[pmatilai]

Well, nothing in the header is compressed. Compressing the entire header would be more interesting (see #2220)

[dralley]

How about no longer setting RPMTAG_GROUP to Unspecified automatically? Just exclude the tag.

rpm/build/parsePreamble.c

Line 1239 in 93ee7d9

headerPutString(pkg->header, RPMTAG_GROUP, "Unspecified");

[pmatilai]

Draft updated to v0.02:

• all padding must be zeros
• duplicate tags disallowe
• signature header is self-padding (details open)
• all size tags from signature header are dropped
• group tag made optional

Also added a section for open questions, such as the actual format dection.

[dralley]

group tag made optional

Come to think of it, is this considered legacy nowadays? IIRC the Fedora packaging guidelines and other distros recommend against it, I believe comps kind of replaced it?

[Conan-Kudo]

Group tag is still used by the Mandrake family (Mageia, OpenMandriva, PLD, ALT, etc.).

[pmatilai]

It may be considered legacy in Fedora but I disagree. It's not going anywhere.

[dralley]

re: "crypto modernization", maybe look at supporting SHA-3 checksums?

[pmatilai]

Indeed, the choice of algorithms will need a second look. SHA-256 seems to be falling out of fashion already, and we'd like v6 to be future proof for at least a few years 😄

[dralley]

I wouldn't say SHA2-256 is falling out of fashion, most crypto-people seem to think there's not much risk of it being broken any time soon. But a bit of future proofing wouldn't hurt.

[pzb]

Have you considered breaking the main header up into chunks to make using the parallel arrays easier? For example, first have a package section that only has tags that are per package, then have a dependencies section, then have a files section?

Separately, should all text strings be specified as containing UTF-8 in Unicode normalization form C (NFC)?

[dralley]

How would that make using the parallel arrays easier?

[dralley]

MD5 and SHA1 dropped everywhere

What about the CLI flags --hdrid and --pkgid, which bake in MD5 and SHA1?

rpm/docs/man/rpm.8.md

Lines 657 to 660 in 21457de

	**\--hdrid ***SHA1*
	: Query package that contains a given header identifier, i.e. the
	*SHA1* digest of the immutable header region.

and

rpm/docs/man/rpm.8.md

Lines 683 to 687 in 21457de

	**\--pkgid ***MD5*
	: Query package that contains a given package identifier, i.e. the
	*MD5* digest of the combined header and payload contents.

[pmatilai]

Oh, those... Thanks for the reminder. We'll indeed need to figure what to do with those because they are even indexes in the rpmdb. This is tracked as #2633 now.

[zregvart]

I wonder if there is room to include software supply chain attestation data in this version? I would like to see SLSA Provenance and SBOM (e.g. SPDX) support.

[pmatilai]

There's no reason to link that to v6 in particular, additional data can be added any time. Head over to #2389 to discuss that.

[pmatilai]

For the curious, I've started exploring what a v6 package might look like and what work it all entails at https://github.com/pmatilai/rpm/tree/v6gen

This is all very early days, and the above branch is unlikely to become any single PR, this is just to get a feel for the work ahead. Bits and pieces from that work are likely to land in the codebase of course, at some point.

[jobol]

I see use case for having unsigned property headers ignored at installation.

Maybe I'm wrong but I think that it is not possible today.

Here is what I have in mind: If i get a RPM from a factory and if I want to add some data of my own (data that use crypto on its side for prooving its authenticity) but I want not modify the original RPM.

Such property could be used for managing RPMs.

It could also be used in plugins for specific action.

[dralley]

rpmlib() dependencies are reset (ideally they'd be replaced by a better mechanism but that's probably out of scope)

Unrelated (or maybe related) question - why are rpmlib dependencies set with less-than or equal-to the version in which the feature was added? Is trying to express that "if the version is <= this version, rpm needs to be patched with this feature to accept the rpm"?

I'm not sure I understand why it wouldn't just check for support of the feature, period (if that is indeed what it's doing). Otherwise I would think it would be trying to express "I need at least this version (greater than) because of this feature"

[pmatilai]

See https://web.archive.org/web/20070621191805/https://www.redhat.com/archives/rpm-list/2001-April/msg00283.html

Whether it makes sense or not is another question.

[dralley]

Yup. The dependency is tracking the use of syntax that will create a package that won't work quite right with versions of rpm before 3.0.3.

...

Not a bug. The dependency is written with <= so that the range is closed, as >= would make the implicit promise "forever". Meanwhile, since the matching capability is Provides: rpmlib(VersionedDependencies) = 3.0.3-1 there's only a single point covered by the overlapping dependency ranges.

It makes some sense, but it's definitely counterintuitive. The first statement isn't entirely true because of the "or equals". The second statement almost doesn't matter, because rpm presents provides with the old versions anyway and only the "equals" portion of the range is used. And if RPM presents them as a list of capabilities, then why not just match on the capabilities rather than dragging version numbers into it?

I guess to make error messages more informative? Assuming that is taken advantage of. If it's treated as a normal dependency I expect the resulting error message would be backwards.

[dralley]

From a discussion with @Conan-Kudo

(me) what is the intended purpose behind RPM automatically adding a config($pkgname) dependency to both the provides and requires dependency lists of a package with a %config declared in the specfile? why self-require like that?

(neal) My understanding is the original intent is to make it trivial to identify packages that have config files marked as %config. The self-requires thing comes as a consequence of the usage of %config triggering that requirement. But honestly, self-requires should be optimized out in general. The "other" RPM did this and it nicely reduced the amount of data produced for repodata.

I'm curious whether self-requires are something that could be dropped?

[pmatilai]

File-based self-dependencies are not going anywhere in general, because rpm uses that info for its own purposes. Besides, they serve as a dependency generation sanity check, and avoid surprises when (not if) packages get split to smaller pieces.

Whether those %config() deps in particular serve any purpose is a whole other question. AIUI the alleged original use-case is to allow installing packages with --noconfigs while supplying alternative configuration via eg site-specific packages. Only it never worked (without resorting to --nodeps and such) because --noconfigs is global.

[dralley]

%readme is documented as being obsolete, perhaps it should be removed with a shim that behaves as though it was marked as %doc instead, with a warning message to update the specfile?

Likewise for any other directives documented as obsolete, if there's a straightforward replacement.

[pmatilai]

%readme as a spec directive may be obsolete, but there's no reason to drop support for the virtual file attribute. On the contrary, if it was properly integrated with %doc it would be nice to 'rpm --show-readme mypackage' instead of having to chase around in /usr/share/doc.

[dralley]

How would that work exactly? Add a %doc(readme) to replace %readme? Files marked as such would then have both doc and readme flags, similar to %config(noreplace) setting both config and noreplace flags?

[pmatilai]

That's a spec syntax issue, it doesn't matter for package format.

[pmatilai]

There hasn't been much direct activity here recently, but doesn't mean no work has been going on. I'm planning to produce an updated version of the draft in the coming weeks, but the main point is going to be:

The overriding priority for V6 is the obsolence of V4 crypto. We need a replacement format now, not in five or ten years time. And to make this happen now, V6 packages will need to be significantly compatible with existing rpm versions to allow existing infrastructure to handle them. This will mean backpedalling a bit on some things - such as zeroing the lead which would achieve absolutely nothing except cause an unnecessary incompatibility.

This isn't any big revelation actually, it's just going back to where it started after getting just a little bit carried away for a while: the package level fundamentals are already implemented in rpm >= 4.14, v6 is really more about defaults and dusting dark corners than anything else.

The time for more forward-looking changes is after we have v6 out and deployed. Then we can start planning for v7 in the next 5-10 years scale. The 20+ years since v4 was much, much too long.

[pmatilai]

There's now what should be much closer to final (but isn't yet, certainly) draft with clarified scope and compatibility info at #2919

Closing this topic as it has served its purpose and is getting too cluttered to follow.
Thanks everybody for the feedback so far!