pre-compile Tracee's ebpf probe for different kernels

[itaysk]

Today we expect people to compile the bpf probe on their hosts, either manually using make bpf or automatically by letting Tracee do it at start time.
We should be able to pre-compile the probe for different kernel versions, and have Tracee use that.

I'm creating this discussion to discuss some aspect of this feature before we create issues:

Issue 1 - tracee-ebpf's probe discovery

Today Tracee-ebpf's start sequence is somewhat like:

1. Identify current kernel, and Tracee version, generate expected probe name
2. TRY look for existing probe in TRACEE_BPF_FILE
3. TRY look for existing probe next in CWD
4. TRY look for existing probe in --install-path
5. TRY building new probe into --install-path (according to --build-policy)
6. Error

We should add a new step to TRY downloading the probe.

• Step precedence is between 4 and 5 (so that existing probes will be reused)
• The probe should be downloaded into the --install-path
• There should be an option to skip this step
• There should be an option to change the remote server.

As with other cases we have experienced, this is a good opportunity to unify the UX:

1. --bpf build:{never,always,ifneeded} default: ifneeded (in place of the existing --build-policy flag)
2. --bpf dir:/path/to/dir default: /tmp/tracee (in place of the existing (--install-path flag)
3. --bpf file:/path/to/bpf.o (precedes over existing TRACEE_BPF_FILE env var)
4. --bpf remote:{never,always,ifneeded}
5. --bpf remote-url:https://url default: TBD (see Issue 2)

future enhancement (out of scope)

1. User can host their own probe internally
2. Authenticating HTTPS certs
3. Checksum
4. Support more remote protocols (git,s3,etc)

Mentioning here we should use go-getter in the future which will solve most of those challenges.

Issue 2 - Build system

We should create an automated build system that compiles the eBPF probe for the different supported kernel version.

• Since we can't cross-compile this needs to be actually spinning up custom VMs, therefore I think managed GitHub Actions is not an option. Perhaps self-hosted GitHub actions could be an option.
• we can probably cross-compile by using the correct kernel headers as part of the build. needs POC
• We should be able to update the desired kernel versions matrix independently from Tracee's release lifecycle (for example we decide we need to support a new kernel)
• Preferably able to run docker since it's easier to build the probe this way

Issue 3 - Probes registry

• We need somewhere to host the compiled artifacts
• Artifacts should be grouped according to Tracee version
• Artifacts should be reference-able by name
• Artifacts should be individually downloadable over HTTP without authentication

Our instinct would be to use GitHub releases, however I don't think it's the best option in this case:

1. Large number of files would complicate up the release page where we send users to find Tracee
2. We may want to add new kernels to the matrix, in which case we will need to modify previous releases. not sure if this feels right (release is a point in time snapshot).
3. Presumably programmatically manipulating the release "Assets" section is not as easy as alternative.
4. It would be nice if we could track download usage

I think we should create our own minimalistic solution based on some cloud storage, and simple proxy.

[itaysk]

re the Build system issue, we probably don't need a real kernel (VM) in order to build the probe, we could cross-compile it using just the appropriate headers, as suggested by Amir Jerbi. Needs to be validated but sounds promising

[grantseltzer]

Yea if we could have a releases page for tracee-ebpf (or a tracee container) built for various kernel versions that'd be great. I don't feel that we should distribute the tracee go binary separately from the bpf object. When we ship the complete binary we can also embed the bpf object using the go 1.16 embed package (#434) to further simplify things. Taking it a step further we can compile statically to make it even easier.

[yanivagman]

Embedding the bpf code in the binary is a good idea! The only problem I can think about with that approach is the size of the created binary if we embed many bpf object files in it. Currently, the size of the bpf.o files is ~1.3MB. Say that we want to support 5 distros, and 10 versions per distro (on average), we get ~65MB extra for the binary. A different approach can be putting the bpf objects as part of the container image.

[grantseltzer]

Yea embedding all of them would be a little insane, I pictured only one, and distribute a different binary for each kernel version.

[itaysk]

We have a POC of cross-compiling using the target version kernel headers. One thing we need to consider is which kernel headers to use.

1. use each distribution's package packaged headers.
2. make headers from source

option 2 is easier to implement because it's one code flow, but it potentially is risky, because we don't know how each distro is configured (K configs).
options 1 is probably safer but is harder to maintain.

@yanivagman what do you think?

[yanivagman]

Not sure I understand what is "make headers from source" - do you mean the linux mainline sources?
If so, I don't think that this will work well - in addition to the configuration problem you mentioned, some distros can backport features from newer kernels and change the sources (e.g. redhat).
On the other hand, if you mean downloading the sources from the distros websites instead of using the packages - then in this case I think that the source already includes the .config used, doesn't it?

[itaysk]

yes I meant the first one. thanks for confirming

[itaysk]

in this case, we need a matrix of $tracee.$distro.$kernel right?

[itaysk]

@afdesk has been working on this. I think that soon we could open issues/PRs.
WIP:
falcosecurity/driverkit#100
https://github.com/afdesk/my-driverkit/
https://github.com/afdesk/tracee-building-action
https://github.com/afdesk/tracee-demo-actions
https://github.com/afdesk/tracee/pull/2/files

[itaysk]

in this case, we need a matrix of $tracee.$distro.$kernel

We need to better understand what is the kernel version component, meaning what's the granularity we need to build for.
previously we assumed the full version (v.major.minor-patch) but we might not need patch variations as @rafaeldtinoco suggested. @yanivagman WDYT?

[rafaeldtinoco]

@rafaeldtinoco I think that one pre-compiled CO-RE enabled bpf code should be enough to support all distros that have BTF support turned on. Eventually, this is what CO-RE comes to solve - porting the same binary to different kernels.

Yes, my proposal is:

CO-RE for every kernel that supports BTF. If kernel does not support BTF (because it is old, or it does not have it enabled) we make libbpf to load BTF from elsewhere (an external file) that contains BTF info (that can be generated from DWARF symbols) INSTEAD of having different BPF binaries to each possible environment.

This way we would only be CO-RE, no matter what.

But, like @itaysk said, this is an alternative path right now as the multiple objects approach had already been discussed, is more mature, and is the path chosen right now.

Also see: https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html#dealing-with-kernel-version-and-configuration-differences

Yes, I have seen your network changes doing the LINUX_KERNEL_VERSION technique for the bitwise sk_buf fields that turned into full variables. I'm glad they give us the kernel version.

Unfortunately, I think the build time LINUX_KERNEL_VERSION definition (comes from linux-headers) might be wrong in some distros. Debian/Ubuntu are wrong iirc. I know Ubuntu has 4.15.0 as the LINUX_KERNEL_VERSION in its headers... when it should be 4.15.18 in Bionic, for example, the base for their 4.15 kernels.

In those cases we would have to have build tests I guess (for different supported scenarios).

I discovered that when I introduced the upstream kern_version ebpf attr parameter (I had to specificy 4.15.18 in the eBPF parameter of the object i was loading to 4.15.0 kernels to make it work).

[yanivagman]

Unfortunately, I think the build time LINUX_KERNEL_VERSION definition (comes from linux-headers) might be wrong in some distros. Debian/Ubuntu are wrong iirc. I know Ubuntu has 4.15.0 as the LINUX_KERNEL_VERSION in its headers... when it should be 4.15.18 in Bionic, for example, the base for their 4.15 kernels.

I know about the kernel version mismatch in ubuntu when you do uname and it is not compatible with what should be given to the bpf version variable. However, this should not be a problem as we use LINUX_KERNEL_VERSION from version.h header, and not taking it from uname.
Anyways, I was referring to CO-RE, where we should not use this macro, but the other methods as described in the link I gave above.

CO-RE for every kernel that supports BTF.

ok, so we agree about one CORE bpf object for all kernels with BTF.

If kernel does not support BTF (because it is old, or it does not have it enabled) we make libbpf to load BTF from elsewhere (an external file) that contains BTF info (that can be generated from DWARF symbols) INSTEAD of having different BPF binaries to each possible environment.
This way we would only be CO-RE, no matter what.

Ok, understood. That's a nice idea!

[yanivagman]

Actually, when thinking about your idea, if we have one database that contains BTFs for every kernel we want to support, there is no more dependency in the tracee version itself, and it can be used for every version of tracee, which is awesome. Instead of compiling for every version, we just need to create such BTF database once.
This can actually be a nice idea for a new open source repository, that I'm now asking myself -why no one did it until now? With such db, anyone who want to support ebpf for old kernels with no btf will be able to do this almost instantly

[rafaeldtinoco]

I know about the kernel version mismatch in ubuntu when you do uname and it is not compatible with what should be given to the bpf version variable. However, this should not be a problem as we use LINUX_KERNEL_VERSION from version.h header, and not taking it from uname.

version.h is wrong as well... and there is no current version variable we can get runtime... (at least I could not find as all version like stuff are macros).

[rafaeldtinoco]

Actually, when thinking about your idea, if we have one database that contains BTFs for every kernel we want to support, there is no more dependency in the tracee version itself, and it can be used for every version of tracee, which is awesome. Instead of compiling for every version, we just need to create such BTF database once.

AND there is a good idea from @itaysk as well. Theoretically our BTF does not need to be complete. We could even strip it to the data we're currently using within tracee only (pahole has the recursive option when displaying data, I wonder if you can format from binary to C and then back to binary). But this other step.

For the BTF database, we could have 1 BTF per major distribution kernel release and supported kernel.

Fedora

Version (Code name)	Release	Kernel
27	2017-11-14	4.13
28	2018-05-01	4.16
29	2018-10-30	4.18
30	2019-05-07	5.0

Ubuntu
Focal (4.19)
Focal HWE (5.4)
Groovy (5.4)
Groovy HWE (5.8)
Next Ubuntu LTS (5.14 ?)
Next Ubuntu LTS HWE (5. ?)

etc...