One of my servers responds to about 100,000 requests per day. Apache would randomly hang a few times a day, taking all of the websites down. I tried every solution I could find online to prevent Apache from hanging, but nothing helped, so I decided to stop wasting time on it and simply develop a good bandage for the problem.
At first I used a script to automatically restart Apache once every two hours. This wasn’t good enough because sometimes Apache would hang multiple times within one hour, other times it would keep running for 12 hours without issue (probably having to do with traffic spikes and waves of spider bots).
Eventually I developed the following script, which restarts Apache only when needed. The script tries to download a page from the website, and if the server fails to respond within 15 seconds, the script assumes Apache is down and restarts it. The script should be run once every minute for eternity.
#!/bin/bash now="$(date)" # prevent multiple instances of this script from running # at the same time. Note that the script's file name is # auto_restart_unresponsive_apache, update it to match # whatever file name you use for this script for pid in $(pidof -x auto_restart_unresponsive_apache); do if [ $pid != $$ ]; then #Process is already running with PID $pid" exit 1 fi done # try to download a webpage from one of the sites on the server. # we use --head to only download its header, we don't need the page's contents # we pass a ?t=some_number parameter to the web page, this allows the script # to bypass web caches like varnish, otherwise Apache could be down but a caching # mechanism could still return a functional web page for hours if curl -s --max-time 15 --head http://example.com/?t=`date +%s` | grep "200 OK" > /dev/null then echo "$now:The HTTP server is up!" > /dev/null else service apache2 restart echo "$now: Restarting apache" >> /var/log/apache_restart_log fi
The cron job
Below is the cron job I use to run the script once every minute. The output is sent to /dev/null, and any errors are also sent there. Otherwise you could receive emails on the server every time the script runs.
* * * * * /some_path/auto_restart_unresponsive_apache >/dev/null 2>&1