[BUG] (Cron update should temporary remove whitelist before the curl test)

[judehuang]

Describe the bug
If your server IP is in the whitelist, which is suggested in the installation, then the curl test will fail on CURL_RESPONSE_BAD when running update-apacheblocker.sh

To Reproduce
Enter your server static IP in the whitelist-ips.conf as suggested, NOT 127.0.0.1, make some changes to the globalblacklist.conf, e.g. delete the last commented line, to make sure the diff will allow the rest of the script to run. Then when it hits curl test area, it will fail because it expect to get 403 for CURL_RESPONSE_BAD. Since your local server IP is whitelisted, you actually get 200.

Expected behavior
If there is a difference in globalblacklist.conf, it should update and pass the test.

Screenshots
Bad bot CURL FAIL

Bad bot curl tests have failed on [my-server-name].

Server (please complete the following information):

• OS: Ubuntu
• Apache Version Server version: Apache/2.4.58 (Ubuntu)
  Server built: 2024-10-02T12:40:51
  Server's Module Magic Number: 20120211:129
  Server loaded: APR 1.7.2, APR-UTIL 1.6.3, PCRE 10.42 2022-12-11
  Compiled using: APR 1.7.2, APR-UTIL 1.6.3, PCRE 10.42 2022-12-11
  Architecture: 64-bit
  Server MPM: prefork
  threaded: no
  forked: yes (variable process count)
  Server compiled with....
  -D APR_HAS_SENDFILE
  -D APR_HAS_MMAP
  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
  -D APR_USE_PROC_PTHREAD_SERIALIZE
  -D APR_USE_PTHREAD_SERIALIZE
  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
  -D APR_HAS_OTHER_CHILD
  -D AP_HAVE_RELIABLE_PIPED_LOGS
  -D DYNAMIC_MODULE_LIMIT=256
  -D HTTPD_ROOT="/etc/apache2"
  -D SUEXEC_BIN="/usr/lib/apache2/suexec"
  -D DEFAULT_PIDLOG="/var/run/apache2.pid"
  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
  -D DEFAULT_ERRORLOG="logs/error_log"
  -D AP_TYPES_CONFIG_FILE="mime.types"
  -D SERVER_CONFIG_FILE="apache2.conf"
• Other Environments [None]
• Any applicable error messages [None]

Additional information
This maybe the cause of some people complained about installation. I had my local dev machine IP whitelisted. After I installed per your README, I kept getting 200 when I was testing bad bots and bad referrer in Step 8. Then it passed as soon as I commented out my own IP in the whitelist-ips.conf

Here is my suggested fix in the bash script:
sed '/Include custom.d/whitelist-ips.conf/s/^/#/' ${APACHE_CONF}/globalblacklist.conf > ${APACHE_CONF}/globalblacklist.tmp
mv ${APACHE_CONF}/globalblacklist.tmp ${APACHE_CONF}/globalblacklist.conf
CURL_RESPONSE_BAD=$(curl --location -A "masscan" -Isk -o /dev/null -w %{http_code} ${CURL_TEST_PROTOCOL}://${CURL_TEST_URL_NAME} | tr -dc '[:alnum:]')
CURL_RESPONSE_GOOD=$(curl --location -A "googlebot" -Isk -o /dev/null -w %{http_code} ${CURL_TEST_PROTOCOL}://${CURL_TEST_URL_NAME} | tr -dc '[:alnum:]')
sed '/Include custom.d/whitelist-ips.conf/s/^#//' ${APACHE_CONF}/globalblacklist.conf > ${APACHE_CONF}/globalblacklist.tmp
mv ${APACHE_CONF}/globalblacklist.tmp ${APACHE_CONF}/globalblacklist.conf