Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Need to build a module for Anthos on-prem kernel (5.4.0-1014-gkeop) #1193

Open
ssurovich opened this issue Jun 6, 2023 · 11 comments
Open

Need to build a module for Anthos on-prem kernel (5.4.0-1014-gkeop) #1193

ssurovich opened this issue Jun 6, 2023 · 11 comments

Comments

@ssurovich
Copy link

The list of supported Kernels dont have anything that works with the appliances Google provides for Anthos on VMware - They always add the gkeop to the kernel, so it will never be found in the supported list.

Can I still use the manually build steps to make a compatible collector for my cluster?
@Stringy
Copy link
Collaborator

Stringy commented Jun 7, 2023

Hi @ssurovich, provided you have access to kernel headers for the nodes in your cluster, the manual build steps can be used to build drivers.

Alternatively, latest versions of collector now contain a built-in BTF enabled driver which may work on your cluster out-of-the-box. This can be enabled by setting the environment variable: COLLECTION_METHOD=core-bpf (collector version 3.15.0 or newer)

@ssurovich
Copy link
Author

Hi @Stringy - Thanks for the quick reply - Ill try setting that variable to see if it works.

@ssurovich
Copy link
Author

ssurovich commented Jun 8, 2023 •
edited
Loading

Ok, Im using a collector release of 4.0.2 and I have the arg set to core-bfp and still no luck. In the collector logs, it verifies that I did pass the collection-method (User configured collection-method=core_bpf)

It looks like its still tryiing to pull a module for the kernel itself, "attempting to download collector-ebpf-5.4.0-1054-gkeop.o"

It will be a challenge to create a module, all of my nodes in an air-gap environment and the nodes are appliances, making any additional software/modules a challenge to add.

Ive created Falco modules in the past, I just needed to pass access to /usr/src to the deployment - the linux-headers are in that directory on all nodes. Can I create a module using the included headers?

@erthalion
Copy link
Contributor

Ok, Im using a collector release of 4.0.2 and I have the arg set to core-bfp and still no luck. In the collector logs, it verifies that I did pass the collection-method (User configured collection-method=core_bpf)

It looks like its still tryiing to pull a module for the kernel itself, "attempting to download collector-ebpf-5.4.0-1054-gkeop.o"

What exactly doesn't work, do you get any error message with core_bpf? The log messages could be a bit inconsistent, so it's better to verify the final result.

@ssurovich
Copy link
Author

Ah sorry, should have added more details.

It errors out when attempting to download the kernel object from https://sensor.stackrox.svc:443/kernel-objects/2.4.0/collector-ebpf-5.4.0-1054-gkeops.o.gz

Right after that line, a bunch of : Unexpected HTTP request failure (HTTP 500)

After 90 failures with the 500 errors:

No suitable kernel object downloaded for collector-ebpf-5.4.0-1054-gkeops.o.gz

In the diagnostics portion, it says the sensor is connected, under kernel driver candidates: CO.RE eBPG probe (available) and collector-ebpf-5.4.0-1054-gkeops.o.gz (unavailable)

Im not sure why it knows I have the core selected and that seems to be passing, but it still tries to get the kernel module that doesnt exist?

@erthalion
Copy link
Contributor

This sounds strange indeed. Could you post the whole log output, ideally with the logLevel:debug in the Collector configuration?

@02fa
Copy link

02fa commented Jul 2, 2023

Many thanks, @Stringy , for the hint COLLECTION_METHOD=core-bpf! It saved my day, because Oracle Linux 9 arm64 'unbreakable enterprise' kernels aren't not included into KERNEL_VERSIONS either.

@ssurovich , collector release 4.0.2 isn't a thing. Try to use 3.15.0 branch. It worked for me:

Collector logs with collection-method=core-bpf 
Collector Version: 3.15.0
OS: Oracle Linux Server 9.2
Kernel Version: 5.15.0-102.110.5.1.el9uek.aarch64
Starting StackRox Collector...
[INFO    2023/07/02 12:06:40] Hostname: 'dex'
[INFO    2023/07/02 12:06:40] User configured collection-method=core_bpf
[INFO    2023/07/02 12:06:40] Afterglow is enabled
[INFO    2023/07/02 12:06:40] Sensor configured at address: sensor.stackrox.svc:443
[INFO    2023/07/02 12:06:40] Attempting to connect to Sensor
[INFO    2023/07/02 12:06:49] Successfully connected to Sensor.
[INFO    2023/07/02 12:06:49] Module version: 2.5.0
[INFO    2023/07/02 12:06:49] Config: collection_method:core_bpf, useChiselCache:1, scrape_interval:30, turn_off_scrape:0, hostname:dex, processesListeningOnPorts:1, logLevel:INFO, set_import_users:0
[INFO    2023/07/02 12:06:49] Attempting to find eBPF probe - Candidate versions: 
[INFO    2023/07/02 12:06:49] CO.RE eBPF probe
[INFO    2023/07/02 12:06:49] collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o
[INFO    2023/07/02 12:06:50] 
[INFO    2023/07/02 12:06:50] This product uses ebpf subcomponents licensed under the GNU
[INFO    2023/07/02 12:06:50] GENERAL PURPOSE LICENSE Version 2 outlined in the /kernel-modules/LICENSE file.
[INFO    2023/07/02 12:06:50] Source code for the ebpf subcomponents is available at
[INFO    2023/07/02 12:06:50] https://github.com/stackrox/falcosecurity-libs/
[INFO    2023/07/02 12:06:50] 
[INFO    2023/07/02 12:06:50] 
[INFO    2023/07/02 12:06:50] == Collector Startup Diagnostics: ==
[INFO    2023/07/02 12:06:50]  Connected to Sensor?       true
[INFO    2023/07/02 12:06:50]  Kernel driver candidates:
[INFO    2023/07/02 12:06:50]    CO.RE eBPF probe (available)
[INFO    2023/07/02 12:06:50]  Driver loaded into kernel: CO.RE eBPF probe
[INFO    2023/07/02 12:06:50] ====================================
[INFO    2023/07/02 12:06:50] 
[INFO    2023/07/02 12:06:50] Network scrape interval set to 30 seconds
[INFO    2023/07/02 12:06:50] Waiting for Sensor to become ready ...
[INFO    2023/07/02 12:06:50] Sensor connectivity is successful
[INFO    2023/07/02 12:06:50] Started network status notifier.
[INFO    2023/07/02 12:06:50] Established network connection info stream.
[INFO    2023/07/02 12:06:50] Trying to establish GRPC stream for signals ...
[INFO    2023/07/02 12:06:50] Successfully established GRPC stream for signals.
[INFO    2023/07/02 12:06:50] Found self-check process event.
[INFO    2023/07/02 12:06:51] Found self-check connection event.
[INFO    2023/07/02 12:07:00] self-check (pid=77) exitted with status: 0

versus

Collector logs with collection-method=ebpf 
Collector Version: 3.15.0
OS: Oracle Linux Server 9.2
Kernel Version: 5.15.0-102.110.5.1.el9uek.aarch64
Starting StackRox Collector...
[INFO    2023/07/01 21:52:16] Hostname: 'dex'
[INFO    2023/07/01 21:52:16] User configured collection-method=ebpf
[INFO    2023/07/01 21:52:16] Afterglow is enabled
[INFO    2023/07/01 21:52:16] Sensor configured at address: sensor.stackrox.svc:443
[INFO    2023/07/01 21:52:16] Attempting to connect to Sensor
[INFO    2023/07/01 21:52:16] Successfully connected to Sensor.
[INFO    2023/07/01 21:52:16] Module version: 2.5.0
[INFO    2023/07/01 21:52:16] Config: collection_method:ebpf, useChiselCache:1, scrape_interval:30, turn_off_scrape:0, hostname:dex, processesListeningOnPorts:1, logLevel:INFO, set_import_users:0
[INFO    2023/07/01 21:52:16] Attempting to find eBPF probe - Candidate versions: 
[INFO    2023/07/01 21:52:16] collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o
[INFO    2023/07/01 21:52:16] Attempting to download collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o
[INFO    2023/07/01 21:52:16] Attempting to download kernel object from https://sensor.stackrox.svc:443/kernel-objects/2.5.0/collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o.gz
[INFO    2023/07/01 21:52:16] HTTP Request failed with error code 404
[WARNING 2023/07/01 21:55:12] Attempted to download collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o.gz 36 time(s)
[WARNING 2023/07/01 21:55:12] Failed to download from collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o.gz
[WARNING 2023/07/01 21:55:12] Unable to download kernel object collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o to /module/collector-ebpf.o.gz
[WARNING 2023/07/01 21:55:12] No suitable kernel object downloaded for collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o
[ERROR   2023/07/01 21:55:12] Failed to initialize collector kernel components.
[INFO    2023/07/01 21:55:12] 
[INFO    2023/07/01 21:55:12] == Collector Startup Diagnostics: ==
[INFO    2023/07/01 21:55:12]  Connected to Sensor?       true
[INFO    2023/07/01 21:55:12]  Kernel driver candidates:
[INFO    2023/07/01 21:55:12]    collector-ebpf-5.15.0-102.110.5.1.el9uek.aarch64.o (unavailable)
[INFO    2023/07/01 21:55:12] ====================================
[INFO    2023/07/01 21:55:12] 
[FATAL   2023/07/01 21:55:12] Failed to initialize collector kernel components.

@ssurovich
Copy link
Author

Sounds like a plan - Willing to try whatever may allow scanning to complete :) Thanks for the heads up - Ill give 3.15.0 a try

@ssurovich
Copy link
Author

This sounds strange indeed. Could you post the whole log output, ideally with the logLevel:debug in the Collector configuration?

It a challenge to show any logs, all of my environments are air-gapped, no easy way to get any log information to post. Wish I could...

@porridge
Copy link

Any updates from your side @ssurovich ?

@msugakov
Copy link
Contributor

msugakov commented Oct 8, 2024

Hi @ssurovich
It's been a while and perhaps this issue lost relevance. Ok if we close it or do you need some assistance with troubleshooting the setup?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@porridge @msugakov @erthalion @Stringy @02fa @ssurovich