What is the "heuristics" for the default number of threads?

[HenrikBengtsson]

The man page says:

-j NUM, --threads=NUM
           This  flag sets the approximate number of threads to use. A
           value of 0 (which is the default) causes ripgrep to  choose
           the thread count using heuristics.

I've been trying to find more information on what the "heuristics" behind this default is, but I failed. Is it documented anywhere?

I found:

ripgrep/crates/core/flags/hiargs.rs

Line 172 in 71d71d2

std::thread::available_parallelism().map_or(1, |n| n.get()).min(12)

which suggests that --threads defaults the number of CPU cores on the host, but at most 12. Is that the case?

Wishes

• Document the default more precisely. It'll help users and sysadms.

• Make it possible to control the default maximum number of threads via an environment variable (e.g. RIPGREP_DEFAULT_MAX_THREADS), so that sysadms can limit it on shared multi-tenant systems.

Background

We discovered a single rg process running wild on our UCSF Wynton HPC system (1,400 users), where it consumed tons of CPU (definitely more than 12) and lots of memory. The system was so sluggish (>500 cores worth of load on a 48-core host) that we struggled to troubleshoot, so we don't have the exact details, but as soon as we terminated this single rg process, everything went back to normal. FWIW, we think this rg process was launched via a remote VS Code session, but not 100% sure).

[BurntSushi]

See #2854 and #2856 as related issues.

Document the default more precisely. It'll help users and sysadms.

I suppose I'm fine with this as long as it's made clear that the specific heuristics may change. PRs are welcome.

Make it possible to control the default maximum number of threads via an environment variable (e.g. RIPGREP_DEFAULT_MAX_THREADS), so that sysadms can limit it on shared multi-tenant systems.

I'm not really inclined to do this. ripgrep is very spartan in terms of what environment variables it supports. And I don't really think it's ripgrep's responsibility to control resource usage beyond what the user invoking rg tells it to do.

[HenrikBengtsson]

I suppose I'm fine with this as long as it's made clear that the specific heuristics may change. PRs are welcome.

I've created #2969. Please note that I'm a rookie when it comes to rust and rust packages, so I struggled to navigate where this is all documented. Based on rg --help and man rg, I ended up using:

grep -F "number of threads" -r --include="*"

to locate where this is documented. I only found one file, so that's the one edited in the above PR.