RPM v6 package format draft, major update

[pmatilai]

Based on the feedback on the initial draft and work in the background, here's a major update to the v6 package format draft. There are still some open questions and details but by now the scope is much clearer, and this should give the reader a fair idea of what to expect in the final spec.

---

RPM v6 package format (draft v0.8.2)

Summary

RPM v6 is a face-lift on the existing RPM package v4 format, not a radical redesign. The primary motivations for v6 are to bring RPM to this millenium technology-wise: eliminate long obsolete crypto, ensure sufficiently large sizes everywhere and shed some compatibility babbage from the last 20 years.

Format

The following mainly describes differences to the v4 format. The fundamental file-format in v6 remains the same: an rpm package consists of

• lead
• signature header
• main header
• payload, optionally compressed

Compatibility

As the v6 format is practically built from extensions to the v4 format, it is largely compatible with later rpm 4.x versions. Roughly, v6 format packages can be

• read and queried by rpm >= 4.6
• unpacked by rpm >= 4.12
• verified by rpm >= 4.14.2
• installability varies by the used feature set (tracked by rpmlib dependencies), but the bare minimum is around 4.12 to 4.14.2

Headers

"Dropped" in the following sections means: no longer generated or accepted in the v6
format itself. As part of v4 packages, they will remain accepted as long
as v4 is relevant (ie, a long long time).

• tag entries are always sorted
• tag numbers are unique across signature and main header
• duplicate tags are not permitted
• all padding must be zeros
• 64 bit sizes everywhere
  • package-level sizes
  • file sizes
• crypto modernization:
  • MD5 and SHA1 dropped everywhere
  • DSA1 dropped everywhere

Lead

• major version is 4 (rpm 4.x used 3)
• os and arch are zeros

Signature header

• header+payload digests and signatures (aka V3 signatures) are dropped
  (v3 signatures are no longer generated by default since 4.16)
  • RPMSIGTAG_MD5
  • RPMSIGTAG_PGP
  • RPMSIGTAG_GPG
  • RPMSIGTAG_PGP5
• all size tags are dropped:
  • RPMSIGTAG_SIZE
  • RPMSIGTAG_LONGSIZE
  • RPMSIGTAG_PAYLOADSIZE
  • RPMSIGTAG_LONGARCHIVESIZE
  • RPMTAG_FILESIGNATURELENGTH
• signature reservation moved to a new tag number 999 (RPMSIGTAG_RESERVED)
• eliminate tag value conflicts between signature and header tags - happens
  when all of the above is done: all signature tags are between 256 and 999
• drop RPMTAG_SHA1HEADER
• open questions:
  • protect ima + fsverity signatures somehow (separate signature?)
  • use RPMSIGTAG_RESERVED also for signature self-padding?

Main header

• file sizes are always 64bit:
  • RPMTAG_LONGFILESIZES
  • RPMTAG_LONGSIZE
  • RPMTAG_LONGARCHIVESIZE
• 32bit size tags are dropped:
  • RPMTAG_SIZE
  • RPMTAG_FILESIZES
  • RPMTAG_ARCHIVESIZE
• new payload size tags (always 64bit):
  • RPMTAG_PAYLOADSIZE
  • RPMTAG_PAYLOADSIZEALT
• new tag to carry package format version: RPMTAG_RPMFORMAT
• always utf-8 encoded
• group tag is made optional

Payload

• always in the "new" large file format, never cpio
• this cannot be reflected in the existing PAYLOADFORMAT as that would be a gratituous compatibility break, introduce a new tag?

Metadata level

• rpmlib() requires tracking pre-4.6.0 specific dependencies are dropped (older rpm versions cannot read v6):
  • PayloadFilesHavePrefix
  • CompressedFileNames
  • PartialHardlinkSets
  • FileDigests
• proper multiarch dependencies (instead of ()(64bit) markers)
• libmagic strings (RPMTAG_FILECLASS) replaced with MIME classification

[Conan-Kudo]

proper multiarch dependencies (instead of ()(64bit) markers)

As a reminder, I took a stab at this in #360 and later #1038. The big change since then is the introduction of subarches for both ARM and x86_64, and I still expect that to happen for RISC-V too. What are we thinking for this now?

[pmatilai]

Figuring out what to do with the multiarch dependencies is one of the next things on my plate.

[pmatilai]

There's now also a draft PR that implements much of the fundamental structural changes as far as package generation goes: #2920

[pmatilai]

Oh, RPMSIGTAG_FILESIGNATURELENGTH needs to be dropped because it's just broken.

[dralley]

this cannot be reflected in PAYLOADFORMAT as that would be a gratituous compatibility break

Ironically dropping the tag entirely would work fine, because of the backwards compatibility backflips that are already in place to deal with v3 packages.

rpm/lib/depends.c

Lines 72 to 78 in 1825dbf

	rpmRC rc = RPMRC_OK;
	const char *payloadfmt = headerGetString(h, RPMTAG_PAYLOADFORMAT);
	/*
	* XXX Ugh, rpm 3.x packages don't have payload format tag. Instead
	* of blindly allowing, should check somehow (HDRID existence or... ?)
	*/
	if (!payloadfmt) return rc;

So... why not? RPMTAG_PAYLOADFORMAT has been retroactively made kind of useless - you need to actually look at the payload in order to figure out what type of payload it is anyway, and you have to go through RPM to do anything with it because it's not a standard format anymore. So then, what is the point of knowing what type of payload is used?

And if you do need to know for some reason, the presence or absence of RPMTAG_LONGFILESIZES is a more reliable indicator in the real world... looking at RPMTAG_PAYLOADFORMAT can do nothing but mislead.

Alternatively - can we at least update this check with a new name (stripped-cpio or something) and get that backported, so that PAYLOADFORMAT could theoretically be fixed in a couple years (rather than decades), the same as was done with the lead checks?

[pmatilai]

Heh, I didn't remember the workaround for 3.x, that's pretty funny. That would've been added by a rather green me, probably in 2007 or so, thinking this checking is a good thing. And then nobody thought to update the name when the new payload format was added years later. It's annoying how much that little slip some ten years ago is hurting just now.

It's something to consider. While v6 always uses the new non-cpio payload, the community has shown interest in other types of payload too, and having a clean mechanism for this would of course be nice.

[pmatilai]

So I guess what we could do here is what old tags do when they go to die: rename to OLD-prefix, reuse the name with a new id and add a header extension to retrieve the old tag if new is not set. And, since v4 merrily ignores the old tag being missing, compatibility is retained. So should be doable, but how useful it is in reality I don't know. A new payload format would have to be tracked by an rpmlib() dependency and not just a random string.

[dralley]

rpmlib() dependencies on v3 compatibility are dropped:

• PayloadFilesHavePrefix
• CompressedFileNames
• PartialHardlinkSets

Are these being specifically singled out for deletion, or can we blanket-delete feature dependencies introduced before 4.6? Because that's a much longer list, e.g.

• ExplicitPackageProvides
• FileCaps (maybe this depends on how you compile rpm)
• FileDigests
• PayloadIsBzip2 (maybe this depends on how you compile rpm)
• PayloadIsLzma (maybe this depends on how you compile rpm)

[pmatilai]

Those three were specifically singled out as only being relevant for v3. Things like caps and payload compressors relate to actual rpm capabilities depending on how it was built, and will need to stay.
But .. considering rpm < 4.6 will not be able to read v6 then that's indeed a better cut-off point than general v4, so FileDigests can go, and probably something else too.

Note that this is only about generating said dependencies into packages. What capabilities librpm provides is a separate matter, eg ExplicitPackageProvides is never generated although it is provided. Ditto for HeaderLoadSortsTags and ConcurrentAccess, maybe something else. But getting rid of those would be just general cleanup.

[pmatilai]

One almost forgotten metadata-level thing added: replace libmagic strings (in RPMTAG_FILECLASS) with MIME classification. I don't know if it makes more sense to reuse the existing tag or reserve a new one just for the purpose though, but probably better to make the change explicit and just use RPMTAG_FILEMIME or such for the new data.

[Conan-Kudo]

I'd probably go with RPMTAG_FILEMIME.

[pmatilai]

I guess it's time to move on with the low-level spec: #2988

[dralley]

How about defaulting to Zstd compression as opposed to Gzip?

(less of a format change and more of a change to the tool defaults)

Actually, speaking of, there are perhaps other defaults that ought to be tweaked. A distro can always override them, I know, but updating the upstream defaults eventually is probably useful.

[pmatilai]

It's not a bad idea. Gzip is getting a bit long in the tooth, and I guess all distros moved to something better 10-15 years ago.

[Conan-Kudo]

I think defaulting to zstd would be a reasonable choice too. Most distros have been doing this for a while now.

[keszybz]

Is HEADERIMMUTABLE still present in the new format, and if yes, can it be dropped?

[pmatilai]

It very much is and a rather fundamental part of it. Taking it away would basically take us back to v3 format.
Why would you want to drop it it?

[keszybz]

When looking into fields that always change when doing builds to test build reproducibility, I was told that HEADERIMMUTABLE is a copy of fields that are not supposed to change in an rpm files, as protection against … something, maybe file system corruption (?) and that it's not really used. But looking at the format description, I now see that this was a misunderstanding. Please disregard my earlier comment.