Recommended Spec #905

domwhewell-sage · 2023-12-14T09:25:33Z

domwhewell-sage
Dec 14, 2023

Hi there,
Is there a specific specification you recommend?
I am currently running it on an m5.xlarge EC2 instance which I believe is 4 Cores x 16GB ram and the server becomes unresponsive after a short while when running bbot on a large domain.

Running it on ginandjuice.shop from portswigger as a test is fine as that is a tiny domain
For reference the domain I am using it on should have 1000+ sub-domains

TheTechromancer · 2023-12-14T13:20:41Z

TheTechromancer
Dec 14, 2023
Maintainer

This is a known issue and should be solvable. Actually I've been running into it a lot this week since we're assessing a very large client. So it's my plan to work on it after finishing with your GitHub PR.

That spec should be more than plenty.

0 replies

TheTechromancer · 2023-12-21T15:46:33Z

TheTechromancer
Dec 21, 2023
Maintainer

@domwhewell-sage after some testing, I've noticed that sometimes it's DNS rate-limiting that causes the slowdown. Basically, BBOT sends too many requests to your system DNS server (e.g. 8.8.8.8), and it blocks you.

A PR has been merged that automatically rotates between your DNS servers.

I'm curious if this helps your issue. Also, are you aware of the CPU usage during those slow times. Is it high?

3 replies

domwhewell-sage Dec 21, 2023
Author

Hey,
I see, It is doing a lot of DNS queries.
[ 356.628201] cloud-init[1268]: �[1;38;5;69m[INFO]�[0m slippery_jacqueline: Modules running (incoming:processing:outgoing) excavate(0:0:3,933), speculate(1:0:1,532), columbus(497:0:622), subdomaincenter(497:0:589), crt(497:0:564), rapiddns(497:0:484), virustotal(497:0:427), hackertarget(497:0:411), github(497:0:242), urlscan(497:0:93), otx(497:0:71), wayback(497:0:60), dnsdumpster(497:0:33), binaryedge(497:0:20), ipneighbor(0:0:487), massdns(395:1:64), azure_tenant(1:0:341), asn(0:0:81), httpx(0:0:17), dnscommonsrv(1:4:0), sslcert(0:0:2), anubisdb(1:0:0), azure_realm(1:0:0), certspotter(1:0:0), digitorus(1:0:0), dnszonetransfer(1:0:0), leakix(1:0:0), myssl(1:0:0), nsec(1:0:0), oauth(1:0:0), riddler(1:0:0), sitedossier(1:0:0), subdomain_hijack(1:0:0), threatminer(1:0:0), csv(1:0:0), human(1:0:0), json(1:0:0), subdomains(1:0:0) [ 356.638215] cloud-init[1268]: �[1;38;5;69m[INFO]�[0m slippery_jacqueline: Events produced so far: DNS_NAME: 846, IP_ADDRESS: 259, DNS_NAME_UNRESOLVED: 210, URL_UNVERIFIED: 142, URL: 48, HTTP_RESPONSE: 47, EMAIL_ADDRESS: 31, CODE_REPOSITORY: 25, ASN: 10, OPEN_TCP_PORT: 2, FINDING: 2 [ 356.650012] cloud-init[1268]: �[1;38;5;69m[INFO]�[0m slippery_jacqueline: 12,444 events in queue (DNS_NAME: 5,071, URL_UNVERIFIED: 3,441, IP_ADDRESS: 1,970, EMAIL_ADDRESS: 984, OPEN_TCP_PORT: 831, CODE_REPOSITORY: 95, ASN: 17, STORAGE_BUCKET: 15, HTTP_RESPONSE: 9, URL: 8, AZURE_TENANT: 1, FINDING: 1, PROTOCOL: 1)

I am running it in an EC2 instance non-interactively so haven't managed to get good data (The only semi-interaction is with the GetSystemLog ec2 command which does not update very often).
When I raised the initial question my EC2 server had been running for 4 days and on the 4th day had 100% CPU utilization preventing me from SSH'ing in.

For now just want to get some stats on how many DNS names it finds so am just running it with -f subdomain-enum after I get back from the christmas break I will try running the dev branch

domwhewell-sage Dec 22, 2023
Author

Interestingly with only the subdomain-enum enabled a scan on that same domain completed in about 2 hours. The subdomains.txt file contains 1509 records.

TheTechromancer Dec 22, 2023
Maintainer

Hmm seems likely one of the other modules is the culprit then

domwhewell-sage · 2024-01-21T10:12:08Z

domwhewell-sage
Jan 21, 2024
Author

Hey sorry I didn't get back on this my EC2 instance is still running out of memory. All I can get from it is this

I'm thinking about running it on my laptop and leaving it on so I should be able to get some decent data using the --debug switch

2 replies

TheTechromancer Jan 21, 2024
Maintainer

Okay yeah this looks like a serious issue. It might take a little work to pinpoint the offending module. Having debug logs would definitely help.

TheTechromancer Jan 22, 2024
Maintainer

I recently ran into a similar memory issue, and the culprit was nuclei. Sadly since nuclei is a separate tool we don't have much control over it. However, you can decrease the module's batch size (default it is 200) with -c modules.nuclei.batch_size=50. This will decrease the number of targets passed to nuclei per run, and probably cut down on its memory usage.

domwhewell-sage · 2024-01-23T08:26:46Z

domwhewell-sage
Jan 23, 2024
Author

I Changed the image to a t3.xlarge, It has the same 4 Cores and 16GB ram but it offers a connection via the serial port which I can use in case SSH disconnects.
It has been running for about 20 hours now, not all SSH connections are honoured but I can retrieve stats via the serial console. And the issue may be a problem between chair and keyboard as I did not configure storage... But heres some stats anyway

Debug memory usage from bbot output

[72844.748545] cloud-init[1268]: [DBUG] MODULE MEMORY USAGE:
[72844.748601] cloud-init[1268]: [DBUG]     - ffuf_shortnames: 883.33KB
[72844.748654] cloud-init[1268]: [DBUG]     - massdns: 517.47KB
[72844.748703] cloud-init[1268]: [DBUG]     - paramminer_cookies: 516.04KB
[72844.748752] cloud-init[1268]: [DBUG]     - paramminer_getparams: 516.04KB
[72844.748804] cloud-init[1268]: [DBUG]     - ffuf: 173.91KB
[72846.244705] cloud-init[1268]: [DBUG]     - bucket_azure: 162.79KB
[72846.244791] cloud-init[1268]: [DBUG]     - affiliates: 132.63KB
[72846.244851] cloud-init[1268]: [DBUG]     - crt: 130.89KB
[72846.244916] cloud-init[1268]: [DBUG]     - asset_inventory: 53.93KB
[72846.244972] cloud-init[1268]: [DBUG]     - filedownload: 49.77KB
[72846.245105] cloud-init[1268]: [DBUG]     - vhost: 44.42KB
[72846.245168] cloud-init[1268]: [DBUG]     - bucket_amazon: 36.79KB
[72846.245220] cloud-init[1268]: [DBUG]     - bucket_google: 36.79KB
[72846.245269] cloud-init[1268]: [DBUG]     - paramminer_headers: 36.04KB
[72846.245318] cloud-init[1268]: [DBUG]     - speculate: 35.61KB
[72846.245364] cloud-init[1268]: [DBUG]     - csv: 35.24KB
[72846.245414] cloud-init[1268]: [DBUG]     - azure_tenant: 35.08KB
[72846.245460] cloud-init[1268]: [DBUG]     - nmap: 34.99KB
[72846.245510] cloud-init[1268]: [DBUG]     - subdomains: 34.82KB
[72846.245556] cloud-init[1268]: [DBUG]     - human: 34.75KB
[72846.245601] cloud-init[1268]: [DBUG]     - json: 34.75KB
[72846.245644] cloud-init[1268]: [DBUG]     - subdomain_hijack: 34.69KB
[72846.245689] cloud-init[1268]: [DBUG]     - azure_realm: 34.62KB
[72846.245737] cloud-init[1268]: [DBUG]     - secretsdb: 17.07KB
[72846.245783] cloud-init[1268]: [DBUG]     - sslcert: 13.43KB
[72846.245826] cloud-init[1268]: [DBUG]     - httpx: 13.25KB
[72846.245871] cloud-init[1268]: [DBUG]     - bucket_digitalocean: 12.79KB
[72846.245915] cloud-init[1268]: [DBUG]     - bucket_firebase: 12.79KB
[72846.245958] cloud-init[1268]: [DBUG]     - social: 10.77KB
[72846.246004] cloud-init[1268]: [DBUG]     - emails: 10.75KB
[72846.246049] cloud-init[1268]: [DBUG]     - dnscommonsrv: 10.38KB
[72846.246097] cloud-init[1268]: [DBUG]     - gowitness: 8.94KB
[72846.246143] cloud-init[1268]: [DBUG]     - generic_ssrf: 7.31KB
[72846.246189] cloud-init[1268]: [DBUG]     - ipneighbor: 6.70KB
[72846.246235] cloud-init[1268]: [DBUG]     - iis_shortnames: 6.66KB
[72846.246279] cloud-init[1268]: [DBUG]     - internetdb: 6.60KB
[72846.246334] cloud-init[1268]: [DBUG]     - asn: 6.13KB
[72846.246378] cloud-init[1268]: [DBUG]     - excavate: 5.43KB
[72846.246424] cloud-init[1268]: [DBUG]     - nuclei: 5.34KB
[72846.246473] cloud-init[1268]: [DBUG]     - oauth: 4.82KB
[72846.246516] cloud-init[1268]: [DBUG]     - wappalyzer: 4.49KB
[72846.246561] cloud-init[1268]: [DBUG]     - dnszonetransfer: 4.47KB
[72846.246606] cloud-init[1268]: [DBUG]     - telerik: 4.47KB
[72846.246651] cloud-init[1268]: [DBUG]     - nsec: 4.38KB
[72846.246696] cloud-init[1268]: [DBUG]     - fingerprintx: 4.35KB
[72846.246742] cloud-init[1268]: [DBUG]     - ajaxpro: 4.35KB
[72846.246787] cloud-init[1268]: [DBUG]     - git: 4.35KB
[72846.246835] cloud-init[1268]: [DBUG]     - hunt: 4.35KB
[72846.246880] cloud-init[1268]: [DBUG]     - robots: 4.35KB
[72846.246925] cloud-init[1268]: [DBUG]     - wafw00f: 4.35KB
[72846.246971] cloud-init[1268]: [DBUG]     - host_header: 4.32KB
[72846.247015] cloud-init[1268]: [DBUG]     - masscan: 3.77KB
[72847.744221] cloud-init[1268]: [DBUG]     - url_manipulation: 3.30KB
[72847.744315] cloud-init[1268]: [DBUG]     - github_org: 3.20KB
[72847.744365] cloud-init[1268]: [DBUG]     - binaryedge: 3.12KB
[72847.744423] cloud-init[1268]: [DBUG]     - github_codesearch: 3.12KB
[72847.744477] cloud-init[1268]: [DBUG]     - virustotal: 3.07KB
[72847.744527] cloud-init[1268]: [DBUG]     - leakix: 3.00KB
[72847.744573] cloud-init[1268]: [DBUG]     - dastardly: 3.00KB
[72847.744622] cloud-init[1268]: [DBUG]     - ntlm: 2.94KB
[72847.744665] cloud-init[1268]: [DBUG]     - bucket_file_enum: 2.94KB
[72847.744712] cloud-init[1268]: [DBUG]     - badsecrets: 2.85KB
[72847.744759] cloud-init[1268]: [DBUG]     - smuggler: 2.85KB
[72847.744884] cloud-init[1268]: [DBUG]     - wayback: 2.76KB
[72847.744945] cloud-init[1268]: [DBUG]     - viewdns: 2.68KB
[72847.744993] cloud-init[1268]: [DBUG]     - urlscan: 2.67KB
[72847.745040] cloud-init[1268]: [DBUG]     - crobat: 2.62KB
[72847.745084] cloud-init[1268]: [DBUG]     - sublist3r: 2.62KB
[72847.745127] cloud-init[1268]: [DBUG]     - anubisdb: 2.62KB
[72847.745170] cloud-init[1268]: [DBUG]     - certspotter: 2.62KB
[72847.745216] cloud-init[1268]: [DBUG]     - columbus: 2.62KB
[72847.745266] cloud-init[1268]: [DBUG]     - digitorus: 2.62KB
[72847.745312] cloud-init[1268]: [DBUG]     - dnsdumpster: 2.62KB
[72847.745359] cloud-init[1268]: [DBUG]     - hackertarget: 2.62KB
[72847.745405] cloud-init[1268]: [DBUG]     - myssl: 2.62KB
[72847.745451] cloud-init[1268]: [DBUG]     - otx: 2.62KB
[72847.745502] cloud-init[1268]: [DBUG]     - pgp: 2.62KB
[72847.745548] cloud-init[1268]: [DBUG]     - postman: 2.62KB
[72847.745596] cloud-init[1268]: [DBUG]     - rapiddns: 2.62KB
[72847.745641] cloud-init[1268]: [DBUG]     - riddler: 2.62KB
[72847.745687] cloud-init[1268]: [DBUG]     - sitedossier: 2.62KB
[72847.745733] cloud-init[1268]: [DBUG]     - subdomaincenter: 2.62KB
[72847.745776] cloud-init[1268]: [DBUG]     - threatminer: 2.62KB
[72847.745825] cloud-init[1268]: [DBUG]     - aggregate: 2.35KB
[72847.745871] cloud-init[1268]: [DBUG]     - bypass403: 2.35KB
[72847.745918] cloud-init[1268]: [DBUG]     - emailformat: 2.35KB
[72847.745969] cloud-init[1268]: [DBUG]     - skymem: 2.35KB

Running top on the machine seems as though ffuf sits between 40%-60% cpu usage

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  22989 root      20   0  719200  19864   6912 S  62.0   0.1 452:48.95 ffuf
   5629 root      20   0 1432944  49496  34432 S   0.3   0.3   3:14.46 containerd
      1 root      20   0  167440  12660   8180 S   0.0   0.1   0:11.31 systemd

Running free shows there's a decent amount of RAM left

               total        used        free      shared  buff/cache   available
Mem:            15Gi       5.4Gi       4.8Gi       0.0Ki       5.2Gi       9.7Gi

But the biggest problem is storage space 👎

Filesystem       Size  Used Avail Use% Mounted on
/dev/root        7.6G  5.9G  1.7G  78% /

I will keep an eye on it but I am anticipating at some point all the available space is going to be used and that is when the memory usage skyrockets and I will see that screenshot from my last message.

0 replies

domwhewell-sage · 2024-01-26T09:25:36Z

domwhewell-sage
Jan 26, 2024
Author

Seems like I can close this now, During investigation the MODULE MEMORY USAGE from the comment above stayed pretty much consistent throughout, fuff did sit around 40%-60% CPU usage and I did occasionally see it spike up to 100% and not honour any SSH connections but after some time it did go back down. I think the recent update helped a lot!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recommended Spec #905

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Recommended Spec #905

domwhewell-sage Dec 14, 2023

Replies: 5 comments · 5 replies

TheTechromancer Dec 14, 2023 Maintainer

TheTechromancer Dec 21, 2023 Maintainer

domwhewell-sage Dec 21, 2023 Author

domwhewell-sage Dec 22, 2023 Author

TheTechromancer Dec 22, 2023 Maintainer

domwhewell-sage Jan 21, 2024 Author

TheTechromancer Jan 21, 2024 Maintainer

TheTechromancer Jan 22, 2024 Maintainer

domwhewell-sage Jan 23, 2024 Author

domwhewell-sage Jan 26, 2024 Author

domwhewell-sage
Dec 14, 2023

Replies: 5 comments 5 replies

TheTechromancer
Dec 14, 2023
Maintainer

TheTechromancer
Dec 21, 2023
Maintainer

domwhewell-sage Dec 21, 2023
Author

domwhewell-sage Dec 22, 2023
Author

TheTechromancer Dec 22, 2023
Maintainer

domwhewell-sage
Jan 21, 2024
Author

TheTechromancer Jan 21, 2024
Maintainer

TheTechromancer Jan 22, 2024
Maintainer

domwhewell-sage
Jan 23, 2024
Author

domwhewell-sage
Jan 26, 2024
Author