A bash script for automatically restarting an unresponsive Apache server only when needed

One of my servers responds to about 100,000 requests per day. Apache would randomly hang a few times a day, taking all of the websites down. I tried every solution I could find online to prevent Apache from hanging, but nothing helped, so I decided to stop wasting time on it and simply develop a good bandage for the problem.

At first I used a script to automatically restart Apache once every two hours. This wasn’t good enough because sometimes Apache would hang multiple times within one hour, other times it would keep running for 12 hours without issue (probably having to do with traffic spikes and waves of spider bots).

Eventually I developed the following script, which restarts Apache only when needed. The script tries to download a page from the website, and if the server fails to respond within 15 seconds, the script assumes Apache is down and restarts it. The script should be run once every minute for eternity.

The script

#!/bin/bash

now="$(date)"

# prevent multiple instances of this script from running
# at the same time. Note that the script's file name is
# auto_restart_unresponsive_apache, update it to match
# whatever file name you use for this script
for pid in $(pidof -x auto_restart_unresponsive_apache); do
    if [ $pid != $$ ]; then
        #Process is already running with PID $pid"
        exit 1
    fi
done

# try to download a webpage from one of the sites on the server.
# we use --head to only download its header, we don't need the page's contents
# we pass a ?t=some_number parameter to the web page, this allows the script
# to bypass web caches like varnish, otherwise Apache could be down but a caching
# mechanism could still return a functional web page for hours

if curl -s --max-time 15 --head http://example.com/?t=`date +%s` | grep "200 OK" > /dev/null
    then
        echo "$now:The HTTP server is up!" > /dev/null
    else
        service apache2 restart
        echo "$now: Restarting apache" >> /var/log/apache_restart_log
fi

The cron job

Below is the cron job I use to run the script once every minute. The output is sent to /dev/null, and any errors are also sent there. Otherwise you could receive emails on the server every time the script runs.

* * * * *  /some_path/auto_restart_unresponsive_apache >/dev/null 2>&1