From 23a83beb62163aeeca8979eecf61aa5546b4530d Mon Sep 17 00:00:00 2001 From: confused-Techie Date: Sun, 27 Oct 2024 17:13:55 -0700 Subject: [PATCH] Block "FriendlyCrawler" --- static/robots.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/static/robots.txt b/static/robots.txt index 3c2e776..cfbac9c 100644 --- a/static/robots.txt +++ b/static/robots.txt @@ -4,4 +4,6 @@ User-agent: AdsBot-Google User-agent: Mediapartners-Google User-agent: AdsBot-Google-Mobile-Apps Allow: / +User-agent: FriendlyCrawler +Disallow: / Sitemap: https://web.pulsar-edit.dev/sitemap.xml