Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADD] Bytedance hidden bot #216

Open
bohwaz opened this issue Jan 9, 2025 · 1 comment
Open

[ADD] Bytedance hidden bot #216

bohwaz opened this issue Jan 9, 2025 · 1 comment
Assignees

Comments

@bohwaz
Copy link

bohwaz commented Jan 9, 2025

Is this an Addition / Removal Request?

Addition

Please List the User-Agent string or Referrer to be added/removed

BrowserMatchNoCase "(?:\b)Build/MRA58N(?:\b)" badbot
BrowserMatchNoCase "(?:\b)Build/OPD3.170816.012(?:\b)" badbot
BrowserMatchNoCase "(?:\b)Build/LRX21T(?:\b)" badbot
BrowserMatchNoCase "(?:\b)CPU iPhone OS 11_0 like Mac OS X.*Chrome/(?:\b)" badbot

Please explain why it should be added

These are parts of user-agents that are used by Bytedance, coming from users of their apps (ie. TikTok). Bytedance are trying to hide their traffic, but these user agents are very weird, for example, iOS 11 running Chrome, or old Android devices.

For Additions: Please include a log sample 3-5 lines is adequate

XXXX:443 74.221.151.32 - - [09/Jan/2025:00:01:39 +0100] "GET /xxx/doc/xxxx/www/admin/.htaccess HTTP/1.1" 200 3188 "-" "Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.6530.1545 Mobile Safari/537.36"
XXXX:443 75.229.229.57 - - [09/Jan/2025:00:01:39 +0100] "GET /xxx/draft1/tree?ci=yyyy&name=src%2Ftemplates%2Fconfig%2Fcatxxxx&type=tree HTTP/1.1" 200 6970 "-" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.9180.1767 Mobile Safari/537.36"
XXXX:443 75.180.26.109 - - [09/Jan/2025:00:01:40 +0100] "GET /xxx/draft1/finfo?ci=yyyy&name=doc%2Findex.md HTTP/1.1" 200 21074 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.9415.1534 Mobile Safari/537.36"
XXXX:443 68.12.64.235 - - [09/Jan/2025:00:01:40 +0100] "GET /xxx/draft1/finfo?ci=merge-in%3A59f69df&name=doc%2Fadmin%2Fbxxxs.md HTTP/1.1" 200 18713 "-" "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2451.1463 Mobile Safari/537.36"
@bohwaz
Copy link
Author

bohwaz commented Jan 9, 2025

Other sources on this:
https://www.webmasterworld.com/search_engine_spiders/5088284.htm
https://xenforo.com/community/threads/known-bots.148723/page-4

Webmaster World
News and discussion for the independent webprofessional
XenForo community
The lists can be used to block by user agent at the web server. Those two places are just good sources of user agent info.

Yeah I just... Pictured in my mind a hundred guys copying the .txt files and adding them to their robots.txt file, thinking that'll work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants