How can I find details on the binary representation of the RPM DB?

[shanebishop]

I am trying to find an RPM version agnostic way to find details on installed software on any Linux distro that uses RPM packages.

A co-worker found for one version of RPM that the magic byte array/slice []byte{0, 1, 0x43, 0} works for extracting each installed package from the RPM DB using regex (I don't know which version this works for). However, this does not work for other RPM versions.

Is there any documentation for the binary representation of the RPM DB for each RPM version I could look at?

If there is no documentation for this, where can I find the relevant part of the source code for the rpm binaries? (My thought is if there is no documentation, I could try reading the source code to figure out how I can extract the information I need from the RPM DB for each RPM version.)

[pmatilai]

That's the wrong end to be looking at, totally.

'rpm -q' with --queryformat gives you access to every single bit of data in the rpmdb.

The other option is the librpm API, whether through C or Python bindings. The API for database access is largely unchanged in the last 14-15 years.

[shanebishop]

I found that, starting with RPM 4.16, RPM uses an SQLite database (reference).

From querying my local table, I see there are the following tables:

sqlite> .tables
Basenames             Name                  Sigmd5              
Conflictname          Obsoletename          Suggestname         
Dirnames              Packages              Supplementname      
Enhancename           Providename           Transfiletriggername
Filetriggername       Recommendname         Triggername         
Group                 Requirename         
Installtid            Sha1header          

I can look at the schemas of individual tables like so:

sqlite> .schema Name
CREATE TABLE IF NOT EXISTS 'Name' (key 'TEXT' NOT NULL, hnum INTEGER NOT NULL, idx INTEGER NOT NULL, FOREIGN KEY (hnum) REFERENCES 'Packages'(hnum));
CREATE INDEX 'Name_key_idx' ON 'Name'(key ASC);

Is there any documentation for what is stored in each table?

For example, it seems the Name table stores the names of all of the installed RPM packages, and from the schema for this table I can see that it has a foreign hnum key for the Packages table. But when I look at the Packages table, I can't make sense of the data, since it stores blobs:

sqlite> .schema Packages
CREATE TABLE IF NOT EXISTS 'Packages' (hnum INTEGER PRIMARY KEY AUTOINCREMENT,blob BLOB NOT NULL);
sqlite> select * from Packages limit 10;
1|
2|
3|
4|
5|
6|
7|
8|
9|
10|

[pmatilai]

The details of the database format are OFF-LIMITS TO EXTERNAL USERS!

Sqlite is but one of the possible database formats. If you want to access the rpm database, you do so through the librpm API.

[DemiMarie]

The details of the database format are OFF-LIMITS TO EXTERNAL USERS!

Sqlite is but one of the possible database formats. If you want to access the rpm database, you do so through the librpm API.

Just because one should not access the rpmdb without going through librpm does not mean that one might not be curious about the actual rpmdb format. I know I am!

[pmatilai]

Nothing wrong with curiosity, and this being open source, obviously nobody can prevent people from looking.
However I have no incentive to help planned misuse, which is clearly the case here.

The rpmdb format is undocumented because it is a private implementation detail which rpm is free to change at will.

[pmatilai]

And that prevents you from reporting bugs? If so, the security world is even sadder place than I thought.

[DemiMarie]

It does not @pmatilai, but many people are uncomfortable with situations where reporting a bug via the proper channels (public GitHub issue) means publicizing a 0day vulnerability in their own product. They would prefer if security problems in their product caused by upstream bugs were considered security vulnerabilities in the upstream code, so that they could be handled via the private channels used for such things ([email protected] in this case).

[jerome-diver]

Hi @pmatilai , i want to create a general kind of package viewer (not to install anything, but just show) than can embed a lot of other packages managers (not only the one for ditro, but also for snap and flatpak containers, and language specific packages/modules managers). And iḿ doing this in Go.

Query from my own code

I don´t want to do something dirty like to exec some shell command from my own code. Instead, i read files generated by managers and/or databases (for dnf, it is easy).

So there is 2 (or maybe more) options:

1. read the rpmdb.sqlite sqlite3 database (or the Packages.db from openSuse from ndb database)
2. use a golang api that can access/query rpm safely.

Question

1. From the first option: is there a kind of UML database diagram that can show and explain what i can read or should i do it myself ?
2. From the second option: do you know an go API that can query the way you probably like the most (me too) the rpm things ?
3. An other one idea is very welcome (but sure not to exec shell code from source code), do you have some better options around ?

[DemiMarie]

The supported solutions are to call librpm via CGo or to spawn a subprocess. @rhdesmond was in the very specific situation of needing to interact with a potentially malicious RPM database, which required using SQLite directly for security reasons.

[shanebishop]

A Go-specific alternative to linking against librpm and using cgo is to use https://github.com/knqyf263/go-rpmdb. Do this at your own risk - it is possible that at any time, this third-party library could be incompatible with the RPM DB, could have bugs, or could have security vulnerabilities. Example usage can be found here.

[jerome-diver]

@DemiMarie yes, i was just reading about how to parse C code with cGo to do that and was reading intensively the librpm code to do so... i finished to read all the database content without fond any alternative easy way to get some data from there. It looks like metadata of packages are inside a blob field. I was also reading the Python API to go the same way to parse rpmlib with cGo.
@shanebishop that is my bast news of the day ! I love this idea to trust reasonably the community work instead of reinvent the wheel again (nice sentence to justify my laziness too...). I will support the day it does happened an update or maybe to try to help to maintain an existing other project my little code will depend of to paid back for my lazy choice.

Edit ==>

@shanebishop actually, this package from knqyf263 repo seems to have something wrong with rpm Fedora sqlite3 database file. I have to dig on that.

Thank you very much @DemiMarie , @shanebishop and @pmatilai for your quick help with that, i do appreciate it.

[shanebishop]

@jerome-diver, are you referring to knqyf263/go-rpmdb#54, or are you referring to some other SQLite problem with https://github.com/knqyf263/go-rpmdb?

[jerome-diver]

@shanebishop yes, i am refereing to this repo. I can see that the sqlite3 driver is outdated (i think). It should (maybe) use the mattn one .
But anyway, i consider at final step that the long term and good practice is to use a cGo code to parse the rpmlib instead. Even if i read a bit the code and i understand that it seems to reproduct the C code of RPM the Go way... but it is still a third party not recognized thing (it is not bad, it is good with learning things process and with a little test case time to consume to go ahead other way around).
And then i let a bug report on the repo concerned about the driver problem (exactly the one you show... mine).

[pmatilai]

The rpmdb as a database is completely uninteresting. It's just a fairly dumb storage for header blobs, which is where all the actual data is. go-rpmdb seems to be a reimplementation of rpm bits and pieces up to the level where it can read some of the more common bits of data from the header. That's interesting in its own right and nothing wrong with that, but it seems rather incomplete, rpm knows about ~260 tags whereas go-rpmdb lists circa 30 of them. And, like noted above, the format of the rpmdb is a private implementation detail and there's absolutely zero regard to outside users of that data so accessing it without going through librpm can and eventually will break with no notice.

[pmatilai]

Some users cannot use librpm, whether they want to or not, unless you are willing to guarantee that librpm is secure against a malicious rpmdb.

This is an absolutely bizarre point of view.

librpm needs to be secure against tampered with rpmdb as with any data, because if there's a "malicious rpmdb" then librpm is the first thing that will encounter it in every single case! And if you need to look at what you suspect is a tampered with piece of data that may have caused a security incident of some kind, surely you take precautions rather than rewrite everything from scratch in an allegedly safe language? It's not like reading bytes from disk is going to melt down the neighbourhood.

If there are bugs (security issues or otherwise), file tickets! I have preciously little patience for this sort of alleged security hubbabubba.

[pmatilai]

I realized I need to clarify the above outburst somewhat, apologies for that - shouldn't write online when tired and cranky.

I don't know the story and reasons behind go-rpmdb, but I can imagine a whole bunch of reasons why you could not use librpm in some specific situation (none of them directly involve security, but that's not relevant here). And if you need to, you create the tools to scrape the data you need. That's homo sapiens for you, and engineers are a particularly awesome tool-creator subspecies 😅 There's absolutely nothing wrong with that. The author knows the circumstance for which the tool was created and understands the limitations etc.

Where it becomes a problem is when somebody else takes the tool to some other context and builds something on top of it, without that understanding. And maybe somebody builds something on top of that, at which point the original context is completely lost for sure. And maybe that tools grows popular to the point of being irreplaceable. And then when we one day want to improve the rpmdb we find that we can't because doing so would break that important tool using the alternative implementation. So in order to do something to our own database we need to stop and somehow deal with a 3rd party project written in who knows what language, and that burden falls on us, not the creator of that tool or the stack of dependencies beneath it.

Mind, this is not hypotethical in the least. The existing sqlite implementation is rock dumb in part because at least half the energy in that project got spent over dealing with other projects that were making assumptions about the external appearance of rpmdb and/or using alternative implementations and other irreplaceable tooling built on top (look no further than dnf and libsolv). And so I know we'll be revisiting the sqlite schema one day, and I know there will be other rpmdb formats sooner or later. I'd rather concentrate on getting those details right instead of worrying about 3rd party implementations. And that is why I'm so vehemently shooing people away from the rpmdb guts. Want to satisfy your own curiosity - knock yourself out. Want to build software accessing the rpmdb? You really, really need to use librpm.

Getting back to the security aspect for a bit: rpm is no doubt making all manner of assumptions that may have been legit in the nineties but no longer are in the complicated world of containers and whatnot. I'd welcome constructive talk about that stuff, but an arbitrary ultimatum like "willing to guarantee against malicious database" is pointless, nobody can guarantee anything at all.

[DemiMarie]

That was bad wording on my part, sorry about that. What I meant is “Does RPM try to be secure against a malicious rpmdb, at least so long as one doesn’t try to install or remove any packages?”

[jerome-diver]

On my side, i just try to build a little application in Golang that would one day pretend to just show any packages installed (and it is a game to me actuelly and a kind of learning game in the idea that it can be an utility added on the next step maybe). When iḿ writing about widely, i mean, widely like... from any Unix and Linux like distro-OS and from any language's own source code managers, and any kind of containers managers the way that finally it shows all that has been installed and what they are linked with as dependent or dependencies too.

So it is something to be considered as "non destructive action" (because i just go to read things and extract data copied there, never modify these data). I do understand the idea that for some or any reasons, my code can generate a secure leak if it was not well thinking or writing out of librpm consideration/interface to use (but in the same time, it is all open source, everybody can do anything else around). So i will go this way.

So now, for example, in Debian like distro, i can scroll down from my source code inside directories containing packages files that own the requested things by Debian-like deb packages, and scroll down any of these files to find informations searched for the "easy way" with performance from Go memory management too if the algorythm is not too bad. I mean... no one and nothing makes the perf down if not me.

Same for fedora kind of dnf, but inside its own sqlite database because of no blob to obscure or complicate the process to just read something (that is not private also at any places including rpm things).

And same from the librpm API that can "facilitate" as long as i finish to understand how to use it without any examples and how to import the lib with cGo too... (it is C code, very good compiled code, nothing bad there).
But i have to consider then to be able to use librpm in my code, to ask user of my code to install rpm-devel packages to apply.
This is, on my own, the only one down point i can see that makes my use case lower in the idea to offer a 'easy tool to show anything installed" when it is requiring to install things around. It ;eans that i also have to go ahead in front of that to offer some packages builder to manage this for all distro that are asking for that things to be.

[pmatilai]

That was bad wording on my part, sorry about that. What I meant is “Does RPM try to be secure against a malicious rpmdb, at least so long as one doesn’t try to install or remove any packages?”

Ack, ok. Sorry for the delayed response, been a crazy week.

The answer is, uh, "it's complicated":

You need to keep in mind that just reading through the rpmdb, its nothing but a bunch of header blobs, that are then loaded through the same code as the package reading does. You've been there, that is a fairly tightly guarded path, if complicated. To get there you of course need to read it off the database first and that depends on the underlying db backend - I don't know the sqlite track there, except that they have an extremely good test coverage of their code.

An almost amusing part is that rpm >= 4.0 goes to great lengths to digitally verify both digests and signatures on the headers on db every retrieaval before actually loading them. AIUI that was sort of the driving force behind the immutable region business in v4. So one could say, yes, it cares a lot. On the other hand, it does not enforce this in any way because v3 packages didn't have anything verifiable post-install, so it's the same weird behavior as with signed packages traditionally: it verifies by default, if it finds a signature. If it doesn't, unsigned is fine. So you could modify the header in the db to get past that, and here we are over twenty years later and all we have is this ticket: #811 😅 Maybe now that in >= 6.0 we no longer need to care about the v3 packages at all, we can move along with it eventually.

The third part is the indexes, which get exercised in queries, dependency checking and all that. And in that area, I'm sure there would be all sorts of gaps. Rpm needs to be able to trust its own database to a degree. At least the sqlite backend can do some self-health checking but due to the nature of rpmdb, its necessarily quite superficial. I've dreamed of a full db validator for a while, but it's quite a job with little payback, it's just too easy to run --rebuilddb if you think something messed up your indexes (such as just somebody kill -9'ing rpm transaction midflight - this was a common occurrence with BDB, but no longer)