Monitoring custom attributes via SNMP

There comes a time in every network admin’s career when management/developers/dbas need to monitor some obscure attribute on a particular server. Invariably, there is no pre-existing plugin in nagios to monitor this attribute.

The solution of course is to write your own plugin to monitor this attribute. There are at least a couple of ways of doing this, such as running the script remotely via SSH on the target server from the monitoring server. But there is another way which I haven’t seen documented much. This method uses the snmp “exec” directive. The basic method is the following:

1. Write your custom script to monitor the attribute in question. Ensure that the output conforms to the Nagios’ API rules (http://nagios.sourceforge.net/docs/3_0/pluginapi.html)

2. Copy this script over to the target server, and ensure it is executable

3. Add the following line to your snmpd.conf file:

exec custom_oid name_of_script path_to_script [script_arguments]

make sure the custom_oid is not already in use by some other MIB

As a very simple practical example, let’s say you would like to monitor the date on a server. This is a very simplistic example with not many uses, but keep in mind the same reasoning is used for musch more useful, custom built solutions (for example in our environment we’ve used it to determine a runaway database process, or to monitor the mailflow through our mail gateways. Following the steps above:

1. Writing the script

This would be a simple script stored someplace, which i’ll call sample_date.sh:

#! /bin/bash

echo `date`

2. Place the script on the target server. Make sure the target server has SNMP installed and is responding to SNMP get requests, and the script is given execution permissions

3. An example of the relevant line in snmpd.conf:

exec .1.3.6.1.4.1.27654.3    check_date    /home/david/sample_date.sh

Restart SNMPD and you should be all set. Test your new “OID” by running snmpwalk against your server, for example:

snmpwalk -c snmp_community_string -v 2c hostname_of_server .1.3.6.1.4.1.27654.3

This should return at least 7 lines, similar to the following:

SNMPv2-SMI::enterprises.27654.3.1.1 = INTEGER: 1
SNMPv2-SMI::enterprises.27654.3.2.1 = STRING: “check_date"
SNMPv2-SMI::enterprises.27654.3.3.1 = STRING: "/home/david/sample_date.sh"
SNMPv2-SMI::enterprises.27654.3.100.1 = INTEGER: 0
SNMPv2-SMI::enterprises.27654.3.101.1 = STRING: "Sat Oct  8 18:33:27 CEST 2011"
SNMPv2-SMI::enterprises.27654.3.102.1 = INTEGER: 0
SNMPv2-SMI::enterprises.27654.3.103.1 = ""

The line you are normally interested in is the one in bold, namely, the one with the result. Make a note of the OID, in this case it is .1.3.6.1.4.1.27654.3.101.1

Lastly, to monitor this via nagios, you’d just need to use the above OID in the check_snmp plugin already present in nagios. You can monitor almost anything using the above method.