The simplest nagios setup to make a phone call on a critical notification – using twilio service

Author:

The aim of this article is to show you the simplest way you can achieve your monitoring Nagios system to make a phone call on a critical, in our example CRITICAL host DOWN (the host is unreachable)! You will not need any server software to setup (including VoiceOverIP server), the only thing you will need is an account in https://www.twilio.com/, which even could be a free/trial account. The idea here is to be simple, cheap and easy to be accomplished.
To summarize it up:

to make a phone call to a real phone number on a CRITICAL Nagios notification

Here is what you need:

  1. A https://www.twilio.com/ trial/free account.
  2. root access to the Nagios server, because you will put a simple bash script, which uses “curl” to open an URL

The idea here is to make a phone call on a really CRITICAL issue to be able to wake up the person on duty to check the monitoring system and the SMS messages he received! So the phone call might not be accepted, at all – the ring of the phone (or continues vibrating of your smart band on your wrist!) should be enough to get a second different type of notification after the first (only and most cases only?) SMS messaging (which has the infromation for the problem!).

STEP 1) Make a https://www.twilio.com/ trial/free account.

Just go to the site and register. We are going to use an API call to make a phone call and because it should be very easy and the Twilio API supports it we are going to use a single API call using “curl”, which will open an Internet URL and after that, you’ll get a phone call. So it is simple as that

open an Internet URL and you get a phone call

No SDK or any software installation and dependencies. “curl” command line tool is available for all Linux distributions and even for MS Windows.

STEP 2) Phones’ numbers verification

When logged in you should go to verify at least two real telephone numbers, the verification process is to accept an SMS or a voice call.

main menu
Twilio phone number verification

After you have verified your telephone numbers you can use them in the API. You need two different numbers because you cannot use FROM and TO with the same number your telephone carrier will always give a busy signal and you will not receive a phone call. If you do not have telephone numbers or you do not want to use your personal one you can always purchase from Twilio, they sell telephone numbers for as low as 1$ per month for different countries, of course.
There is a catch if you are in a country (or your telephone number) is at high risk (the company maintains a list of such countries) you should promote your account to regular and buy a credit (which you may not use at all for this setup!) with PayPal or credit card or other of their payment methods. But if you are in a country with the lower risk you could do everything here with the trial/free account (at the moment of writing this article, the company could change their policy).
To verify your numbers go to “Programmable Voice” on the left (https://www.twilio.com/console/voice/numbers) and click on “Manage Numbers” button. Then click on “Verified Caller IDs” on the left and last click on “Add new number” (a red plus button). Add your numbers and verify them to continue.

STEP 3) Get your authentication token and account ID (“Auth Token” and “Account Sid” in Twilio terms).

They are shown on the main window after logging in Twilio – the dashboard https://www.twilio.com/console

main menu
Auth Token and Account Sid in Twilio

STEP 4) Simple Nagios bash script.

Here is a simple Nagios bash script to make a phone call to your phone. Note you could use FROM and TO to be the same, but probably you’ll get busy every time (it was the case with our tests), so that’s why you need at least separate verified numbers – one your and one, which will be the caller.
The name of the file is nagios-call-twilio.sh.lock and you must set 755 permissions.

#!/bin/bash

#1. Account settings
account_id="AC11111111111111111111111111111111"
account_auth="22222222222222222222222222222222"
phone_from="+3333333333"
phone_to="+4444444444"

#2. Script settings
lock_expire=300

#3. Advnced script settings
lock_file="/tmp/nagios-call-twilio.sh.lock"
lock_missing=0;

if [ ! -f "$lock_file" ]
then
    touch "$lock_file"
    lock_missing=$((lock_expire+1))
fi

date_lock=`stat -c '%Y' "$lock_file"`
date_now=`date +%s`

date_diff=$((date_now - date_lock + lock_missing))

if [ "$date_diff" -gt "$lock_expire" ]
then
    curl -X POST "https://api.twilio.com/2010-04-01/Accounts/$account_id/Calls.json" --data-urlencode "Url=http://demo.twilio.com/docs/voice.xml" --data-urlencode "To=$phone_to" --data-urlencode "From=$phone_from" -u "$account_id:$account_auth" &>/dev/null
    touch "$lock_file"
fi

The minimal configuration for the script above is:

account_id="AC11111111111111111111111111111111"
account_auth="22222222222222222222222222222222"
phone_from="+3333333333"
phone_to="+4444444444"

And you should received them from Twilio account above – STEP 1, 2 and 3.
The variable “lock_expire” is to set a minimum limit duration between calls. The next phone call will be after this number in seconds if there are is a CRITICAL respectively the script is invoked by the Nagios service.

Technically you can use only the “curl” line, but probably it is a good idea to have a sort of synchronization, when there are multiple requests to this file like multiple down servers at the same time – bad things happen and connectivity issues, too! Here is the Twilio Rest API “make a call” functionality: https://www.twilio.com/docs/voice/make-calls

STEP 5) Nagios configuration

The minimum Nagios configuration would have:

  1. Nagios command, which will invoke your bash script for making phone calls.
  2. Nagios contact, the person, which will accept the phone call.

Nagios command
We used the generic command file in /etc/nagios/objects/commands.cfg and we added the following:

# 'notify-host-by-sms' command definition
define command{
        command_name    notify-host-by-call
        command_line    /usr/local/bin/nagios-call-twilio.sh >>/var/nagios/nagios-call-twilio.log 2>&1
        }

Note we want only host criticals, thats why we have only one command and for the simplicity we do not use the Nagios Variables here.
Nagios contact
The contact file in Nagios /etc/nagios/objects/contacts.cfg

define contact{
        contact_name                    smith-call                      ; Short name of user
        use                             generic-contact                 ; Inherit default values from generic-contact template (defined above)
        alias                           Agent Smith GSM Call            ; Full name of user
        email                           smith@ahelpme.com               ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******
        host_notification_commands      notify-host-by-call             ; The notification command
        host_notification_options       d,u,f                           ; ONLY server down, unreachable or flapping
        service_notification_options    n                               ; No notifications
}

Now include the smith-call in the contact groups you use in your Nagios specific host configurations (for example):

define contactgroup{
        contactgroup_name       support
        alias                   Support Servers
        members                 nagiosadmin, onsite-support, smith-email, smith-call
}

And in your host definition you must include the group in “contact_groups” (multiple members of this variable are accepted):

define host{
        use                     generic-host            ; Inherit default values from a template
        host_name               srv101                  ; The name we're giving to this host
        alias                   srv101 Media            ; A longer name associated with the host
        address                 1.2.3.4                 ; IP address of the host
        hostgroups              linux-servers           ; Host groups this host is associated with
        check_command           check-host-alive
        max_check_attempts      10
        notification_interval   120
        notification_period     24x7
        notification_options    d,u,r
        register                1
        contact_groups          support
}

Restart your Nagios.

Improvements

An easy improvement it to pass the curl command the name of the server/service or generated text based on the Nagios command variables. So when you accept the call the Twilio will use speech synthesis to tell you what/which server/service is in a bad state. But we wanted to keep this article as simple as possible so probably in another article, we can make some improvements and complex modifications.
Of course, you can make a link between the Nagios contact and numerous values it offers

Bonus – Testing the Twilio service

You can test the service only calling the curl from the command line with:

myuser@local ~ $ curl -X POST https://api.twilio.com/2010-04-01/Accounts/AC11111111111111111111111111111111/Calls.json --data-urlencode "Url=http://demo.twilio.com/docs/voice.xml" --data-urlencode "To=+4444444444" --data-urlencode "From=+3333333333" -u AC11111111111111111111111111111111:22222222222222222222222222222222
{"answered_by": null, "price_unit": "USD", "parent_call_sid": null, "caller_name": null, "group_sid": null, "duration": null, "from": "+3333333333", "to": "+4444444444", "annotation": null, "date_updated": null, "sid": "AC55555555555555555555555555555555", "price": null, "api_version": "2010-04-01", "status": "queued", "direction": "outbound-api", "start_time": null, "date_created": null, "from_formatted": "+3333333333", "forwarded_from": null, "uri": "/2010-04-01/Accounts/AC11111111111111111111111111111111/Calls/AC55555555555555555555555555555555.json", "account_sid": "AC11111111111111111111111111111111", "end_time": null, "to_formatted": "+359887099135", "phone_number_sid": "PN57e1f5f63a941c213816e270aa381bad", "subresource_uris": {"notifications": "/2010-04-01/Accounts/AC11111111111111111111111111111111/Calls/CA1a9995222436cd6c43769aebf988b511/Notifications.json", "recordings": "/2010-04-01/Accounts/AC11111111111111111111111111111111/Calls/AC55555555555555555555555555555555/Recordings.json", "feedback": "/2010-04-01/Accounts/AC11111111111111111111111111111111/Calls/AC55555555555555555555555555555555/Feedback.json", "feedback_summaries": "/2010-04-01/Accounts/AC11111111111111111111111111111111/Calls/FeedbackSummary.json"}}

As you can see all IDs are fake.

One thought on “The simplest nagios setup to make a phone call on a critical notification – using twilio service”

Leave a Reply

Your email address will not be published. Required fields are marked *