Like most of us, I am sure you love to do a lot of tinkering under the hood and trying out DIY solutions. This is one such example of building a website monitor. There are a multitude of free and paid website monitoring systems and they do exceptionally well in informing you when a site is goes down and comes back up again. This article is not meant to be a complete replacement for these, but as a breeding ground for you to expand on building a quick & working website monitor.
We are trying to create a basic, yet fully functional website monitoring system. It is designed to run with a very small footprint and was tested with a 128MB NAT VPS. It relies on external applications – Slack & a Git Server (GitLab in this example).
The Git server, stores the URLs for monitoring and you can add/remove entries via version control. Slack is the destination for notifying you when a website goes down. You could expand this example to use any webhook such as Amazon SNS or use Twilio’s SMS API to send you a message.
Let me outline the process here:
Let’s implement this. Since all projects get a name, I am going to call this
200orBust
In your git server, create a text file with URLs in this format
The format is the full URL (http/https) and a separator \ followed by 200. All sites are initially added as “Up”, i.e. HTTP Status of 200.
Commit the change and push to the git server. Note the raw URL of the text, for e.g.,
https://gitlab.com/userid/reponame/raw/master/urllists.txt
If you already have a slack workspace, use this link (https://api.slack.com/incoming-webhooks) to create a new incoming webhook integration for your workspace. Note the webhook URL. It is of the format – https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXX XXXXXXXXX I recommend that when you create a new webhook, you also create a new channel for monitoring site availability so as to not clutter up your existing channels. I have named mine #site-up-down
Before we begin, we need to do an initial clone of the git repo. Use git clone
Now, we are all set to begin our test. Here is the complete code (I have removed some comments for brevity)
The next for loop reads through the file urllists.txt, parses it (converting the input line into the URL to be checked and the previous HTTP status code).
We do a quick curl to get the HTTP status code. I will quickly explain what this does
We pass on the user agent via the option -A. This is required, as some sites such as amazon.com return a 503 status code when no user agent is provided (this prevents bots from screen scraping)
The -L option is to follow redirects. This helps get the right status code for querying websites that use load balancers to redirect your request.
Now, the output of the curl is stored in $status_code which is checked against the previous status code and if changed, send the appropriate message to slack. We are also setting a flag UPDATED to true so that it triggers a git commit and push to the repo at the end.
Here is how the slack messages appear in your channel
You can have this script run automatically through cron. To set it up, pull up crontab – e and add the path to your job like so
Since the cron does not run from the directory of the shell script, you will see that the very first command in our script is cd /home/user/200orbust/200orbust.sh
The above cron job runs every minute and posts to slack of failures and when sites come back up again.
Quick word of caution, keep track of the size of your syslog that tracks the execution of the cron job. You may have to periodically reset it to avoid the syslog becoming a massive file after a month of execution.
Hope this script has helped you in designing a quick and useful server monitor. Let us know how you expanded this example. Happy Monitoring!
1. Beautifying your Slack Messages – https://api.slack.com/incoming – webhooks#advanced_message_formatting
2. Using Twilio’s SMS API – https://www.twilio.com/docs/sms/send-messages
3. The Post to Slack function – http://blog.pragbits.com/it/2015/02/09/slack- notifications-via-curl/
