As you may have observed, server uptime and application availability is a theme I keep alluding to in my blog posts. This week we are going to install monit – a service monitoring app that watches over your server and when something is wrong, it tries to fix it rather than just letting you react to an alert.
Monit from Tildeslash Ltd. is an open source utility that installs on your server and keeps a constant lookout for performance and takes necessary action based on pre-set conditions. Supporting a wide swathe of platforms (except Windows), you can install Monit on BSD/Linux and even on your Mac.
Monit monitors processes – check if your daemons are running. It looks over files, directories – check if a file has been modified (I am looking at you wp-config.php). Keep track of if a program or script is functioning as expected at regular intervals. Watch how your system is performing in terms of CPU load, memory usage etc. Finally, Monit can check if remote systems are connected – check if SSH is running on a remote server.
Monit also comes bundled with a lightweight HTTP server which provides server status.
Monit’s server status screenshot (from monit’s homepage)
Monit can be easily installed on most distributions, through the default package managers.
apt-get install monit # Debian/Ubuntu
yum install monit # RHEL/Centos/Fedora
You can also install them directly as binaries, by downloading the appropriate version from https://mmonit.com/monit/#download
Untar the binary and copy the following files
cp bin/monit /usr/local/bin/ # Or to a folder in your path
cp conf/monitrc /etc/
Finally, you can also compile them from source from the bitbucket repo – https://bitbucket.org/tildeslash/monit/
Once installed, you can begin using monit to watch over your VPS
All the configuration information for monit is located at the ~/.monitrc (or at /etc/monitrc). Invoke monit at the command line,
monit
Monit starts and will detach itself from the terminal and run as a background process. As a daemon, it runs in sleep and wake cycles endlessly and processes the directives in the configuration file
Monit needs to runs as root to process certain tests such as Disk I/O reporting. Though if you run monit as a root, any processes that get started by monit will start as root. This can be overridden by specifying UID, GID in the start directive
The monitrc file must have permissions no higher than 0700, else monit would throw an error and fail.
Syntax of the configuration file is using monit’s own Domain Specific Language (DSL). Comments begin with a #. Typical lines consists of a services or global options in a free-format, token-oriented syntax. Here is an example of how to set a global option
set daemon 20
This sets the daemon process to run every 20 seconds. As you look at the sample configuration file, you will see that a lot of the syntaxes read like regular English. The above set command can have an additional startup delay. The instruction now looks like this
set daemon 20
with start delay 120
This could also be written as
set daemon 20
start delay 120
The word “with” along with ‘if‘, ‘and’, ‘within’, ‘has’, ‘us(ing|e)’, ‘on(ly)’, ‘then’, ‘for’, ‘of’ are called “noisy” keywords which can be ignored and are used only for making the instructions more readable.
Before we get into the checks, you can embed configurations files using the include directive. This way you can split your configurations accordingly.
include /etc/monit/services
include /etc/monit/files
To add checks, the keyword is check. The fields following the check provides information on what needs to be checked. For checking a process
check process <unique-name> <pidfile <path | matching <regex>>
A check for nginx might look like this
check process nginx with pidfile /run/nginx.pid
Processes can be restarted based on conditions, let us say that if mysql uses more than 512MB memory, restart the mysql service
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
if totalmem > 512.0 MB for 10 cycles then restart
You could check a file. Let’s check wp-config.php
check file wp-config.php with path /var/www/thisexample.com/html/wp-config.php
if failed checksum and
expect the sum 79054025255fb1a26e4bc422aef54eb4 then unmonitor
alert security@thisexample.com
An email alert is sent to the address security@thisexample.com when the wp-config.php file fails the checksum specified. The unmonitor is to stop monitoring this and generating duplicate alerts everytime the daemon process runs.
To check remote servers, you could use check host like this
check host thisexample with address 12.34.56.789
if failed ping then alert
if failed port 443 protocol https
and request /blog with content = "Recent Posts"
then alert
In the above example, we are checking a couple of things. One if the IP 12.34.56.789 is available through a ping. Next, we check if port 443 is listening on protocol https. Finally, it requests the URL /blog and checks if the content “Recent Posts” exists in the returned content.
Before moving on to the next section, here is a quick intro on how ALERT recepients are setup. In our checksum example, we alerted security@thisexample.com, but the previous example has no parameter to the alert command.
In this case, the default alert recipient is used. The default is set as
set alert defaul@thisexample.com
To run monit automatically at startup, you can configure monit as a service. Here is the example with systemd systems
Monit needs to be installed from source. Once installed, add monit service configuration in the file /lib/systemd/system/monit.service
# Monit Service
[Unit]
Description=Pro-active monitoring utility for unix systems
After=network.target
Documentation=man:monit(1) https://mmonit.com/wiki/Monit/HowTo
[Service]
Type=simple
KillMode=process
ExecStart=/usr/local/bin/monit -I
ExecStop=/usr/local/bin/monit quit
ExecReload=/usr/local/bin/monit reload
Restart = on-abnormal
StandardOutput=null
[Install]
WantedBy=multi-user.target
You can enable and start monit like this
systemctl enable monit.service
systemctl start monit.service
Now you are all set with a watchdog that is monitoring your server 24×7.
