FirstServed Technical Blog

Home

Help

Technical Blog

Monitoring disk usage with Nagios passive checks and SNMP traps

11th Apr, 2007
Why passive checks and SNMP traps?

The standard Nagios plugins include both a generic snmp plugin check_snmp and one specifically designed to monitor disk usage, check_disk_snmp. Setting up disk usage monitoring with those requires that you activate disk usage monitoring on the host to be monitored, and set up the appropriate service and command on your Nagios server.

You have two options when configuring Nagios: either you set up a service with active checks, where Nagios polls an SNMP OID regularly to check if disk usage isn’t too low, or you set up your service with passive checks only, which means Nagios will not check disk usage itself, but will be fed with alerts sent out from the monitored host. Though the second setup is much more complex than the first one, we prefer it because it allows us to delegate the checks to the monitored servers, and helps keep the load on the Nagios server down.

Setting up SNMP on the monitored host

First, create an SNMP v3 user that will be used for snmpd’s internal checks. This is done by adding a line to SNMP’s persistent data file, which is usually to be found at /var/net-snmp/snmpd.conf. Do NOT put the line below in your main snmpd configuration file!

createUser disman MD5 asecretpassphrase

Note that this line will disappear from the file when next starting the SNMP agent. This is nothing to worry about!

Next, add the UCD-SNMP MIB to your system-wide SNMP settings ( found at /etc/snmp/snmp.conf or /user/local/share/snmp/snmp.conf ). This will allow you to refer to the needed SNMP OID’s by name instead of by number.

mibs IF-MIB:UCD-SNMP-MIB

After this, add the following information to snmpd’s configuration file ( found at /etc/snmp/snmpd.conf for a Red Hat distro, or /usr/local/share/snmp/snmpd.conf if you installed net-snmp from source ):

# readonly user needed for disman monitoring
rouser disman auth

# security name used for internal checks
agentSecName disman

# the default trapcommunity - better change this from the default ‘public’
trapcommunity public

# choose whether to send SNMP v1 traps, SNMP v2c traps or SNMP v2c informs to your Nagios server
# do not send multiple trap types to the same destination
# change 10.10.10.1 to your Nagios server’s address

#trapsink 10.10.10.1
trap2sink 10.10.10.1
#informsink 10.10.10.1

# add the drives to monitor, and define what consists low disk space
# specify the disk space in kilobytes or as a percentage
# you can also use ‘includeAllDisks’ to monitor all available disks

#includeAllDisks 10%
# monitor ‘/’ for 10G available disk space
disk / 10000000
# define the monitoring frequency, what will be monitored, and what information will be added to the SNMP trap
# the line below monitors the dskErrorFlag every 60 seconds, and will send out an SNMP trap when the value reaches its lower or upper threshold - 0 or 1.
monitor -u disman -t -r 60 -o dskPath -o dskAvail -o dskTotal "dskTable" dskErrorFlag 0 1

Restart the snmpd daemon to complete the setup of the agent. Don’t forget to add snmpd to the services started at boot if you haven’t already done so.

Setting up snmptrapd on the Nagios server

For your freshly configured SNMP traps to reach your Nagios installation, you’ll have to configure the snmp trap handling daemon to send the traps to SNMPTT ( SNMP Trap Translator ), which is a nifty little bit of software that permits you to process SNMP traps in a detailed manner.

First add a line like this to /etc/snmp/snmptrapd.conf:

authCommunity log, net, execute public

traphandle default /usr/sbin/snmptthandler

Then modify the snmptrapd stratup script, so the trap handler daemon no longer translates numeric OID’s to their symbolic form:

#   OPTIONS="-Lsd -p /var/run/snmptrapd.pid"
   OPTIONS="-On -p /var/run/snmptrapd.pid"

Start snmptrapd and add it to the startup services.

Setting up SNMPTT on the Nagios server

SNMPTT can be downloaded at http://www.snmptt.org/. Follow the readme file to install the software, add snmptt to the startup services, and then edit snmptt.ini as follows:

# enable dns reverse translation of ip addresses to host names
dns_enable = 1
# assuming your nagios hosts are defined as short host names, strip the domain part of the dns answer
strip_domain = 1
# add a new file to the snmptt conf filessnmptt_conf_files = <<END
/etc/snmp/snmptt/disman-event
END

Next create the /etc/snmp/snmptt/disman-event file, with the content below. Note that the PREEXEC lines normally wouldn’t be necessary, but the version of net-snmp we’re using seems to be buggy when reverse mapping ip addresses, and always passes <UNKNOWN> as the agent’s host name.

EVENT fsDiskSpaceLow .1.3.6.1.2.1.88.2.0.2 "Status Events" Warning
MATCH MODE=and
MATCH $1: (^dskTable$)
MATCH $5: 1
PREEXEC /usr/bin/dig +short -x $aA | awk -F. ‘{ print $$1 }’
FORMAT Disk Space on $p1 low ( $7 / $8 on disk $6 )
EXEC /usr/lib/nagios/plugins/eventhandlers/submit_check_result $p1 disk-usage 1 "Disk Space low ( $7 / $8 on disk $6 )"
SDESC
Disk Space low
EDESC

EVENT fsDiskSpaceNormal .1.3.6.1.2.1.88.2.0.3 "Status Events" Normal
MATCH MODE=and
MATCH $1: (^dskTable$)
MATCH $5: 0
PREEXEC /usr/bin/dig +short -x $aA | awk -F. ‘{ print $$1 }’
FORMAT Disk Space on $p1 normal ( $7 / $8 on disk $6 )
EXEC /usr/lib/nagios/plugins/eventhandlers/submit_check_result $p1 disk-usage 0 "Disk Space normal ( $7 / $8 on disk $6 )"
SDESC
Disk Space normal
EDESC

What does all the above mean? Two OID’s are defined, corresponding to the Disman events mteTriggerRising and mteTriggerFalling. Additionally, a further restriction is made on the variable ‘dskTable’ and the dskErrorFlag value. This is necessary because the Disman event OID’s are generic events also used for load average monitoring, log file monitoring, etc… This way, we can define other events with the same OID’s in SNMPTT and map them to a different Nagios service.

The first event generates a passive check result for Nagios for host $p1, service disk-usage and status WARNING. The second event does the same, but with an OK status.

Restart your snmptt daemon after modifying the file, and proceed to the next step, adding the /usr/lib/nagios/plugins/eventhandlers/submit_check_result file, which looks like this:

#!/bin/sh
# Write a command to the Nagios command file to cause
# it to process a service check result

echocmd="/bin/echo"

CommandFile="/var/spool/nagios/cmd/nagios.cmd"

# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`

# create the command line to add to the command file
cmdline="[$datetime] PROCESS_SERVICE_CHECK_RESULT;$1;$2;$3;$4"

# append the command to the end of the command file
`$echocmd $cmdline >> $CommandFile`

Setting up Nagios for passive checks

Create a service like this:

define service {
        host_name               <your host>
        service_description     disk-usage
        check_command           check-disk-usage-linux!<your host’s ip address>!<snmp community>

        active_checks_enabled           0
        check_freshness                 1
        check_period                    24×7
        contact_groups                  sysad
        event_handler_enabled           0
        freshness_threshold             86400
        is_volatile                     1
        max_check_attempts              1
        normal_check_interval           86400
        notifications_enabled           1
        notification_interval           0
        notification_options            w,u,c,r,f
        notification_period             24×7
        passive_checks_enabled          1
        retry_check_interval            86400
        stalking_options                w,u,c

}

The explanation for all options can be found in the Nagios documentation describing passive-only services. The check_command is used by Nagios when no passive check data have been received within the freshness_threshold, one day in this example. The command should look like this:

define command {
        command_name    check-disk-usage-linux
        command_line    $USER1$/check_snmp_disk -H $ARG1$ -C $ARG2$
}

Check your Nagios after modifying the configuration files, and restart the daemon. You should now be able to test your disk usage monitoring by modifying the low disk space threshold in the target host’s snmpd.conf file, and restarting the snmpd daemon.

Troubleshooting

If the above setup doesn’t work out of the box, which is pretty probable considering how easy SNMP is to set up, try the following:
- Set up SNMPTT to log traps to syslog. If you get no syslog output, the problem lies with your target host’s snmpd configuration.
- Set up SNMPTT to debug to a log file. You’ll see detailed information about the translation of the traps.
- Set up SNMPTT to log unknown traps to a log file. If your events are not correctly defined, you’ll see your traps appear in that file.
- Add ‘traphandle default /usr/bin/traptoemail -s localhost -f <your email> <your email>’ to your snmptrapd.conf file and restart the daemon. This will send all SNMP traps to your e-mail address so you can analyze them in further detail.
- Check your Nagios event log. Do the service check results appear in the log, with the correct hostname and service name?
Good luck!
No Comments
Getting Xen networking to work on a Dell Poweredge 2950

6th Apr, 2007

What happened to our networking?

Here we were, admiring our brand new Dell Poweredge 2950 server that we were going to use as a Xen host. Fedora Core 6 installed without a hitch, but when we tried booting the Xen kernel, we lost connectivity on the first network card for Dom0. The second NIC, which had not been tweaked by Xen’s bridging setup, still worked as a charm. So, regretfully, we set out to learn once again things we’d rather left unlearned…

Dell, Broadcom and the art of technical support

After googling around a bit, we found that the problem was hardly a new one. It appears that the Broadcom Netextreme II BCM5708 NIC’s that ship with the Dell Poweredge 2950 do not play ball with Xen’s bridging setup, at least not when support for Dell’s IPMI card ( DRAC ) is enabled. Xen does some internal tweaking with your network interfaces, and ends up assigning eth0’s mac address to a new virtual interface, which is Dom0’s eth0. The original eth0 gets renamed to peth0, and gets a mac address of ff:ff:ff:ff:ff:fe, since in normal circumstances this interface should not be visible on the LAN.

It appears that the Broadcom BCM5708 NIC’s filter out all traffic that is not directed at the NIC’s physical mac address, at least when the NIC’s management function is enabled. The funny thing is that this problem has been acknowledged by Dell as far back as the summer of 2006, but they haven’t seen fit to bring out a firmware update ro solve this problem. Calling Dell Support brought us no further: there’s nothing known about an eventual release of a patch. How the guys are going to cope with the new Suse Linux Enterprise Server 10 and Red Hat Enterprise Linux 5, both of whom proudly announce the wonders of Xen virtualisation, I don’t know.

Two solutions for the problem

The first, quick and dirty workaround, is to disable the management functions on your Broadcom NIC’s. Since this means going without IPMI support, and forgoing remote reboot and serial text console functionality , that’s more of a choice between Scylla and Charibdis. Anyway, this workaround is documented at:

How to disable the management function on Broadcom NetXtreme II BCM5708 NIC

We chose for a second solution, which is slightly more complex, but has the advantage of keeping IPMI connectivity intact. Since the Broadcom NIC’s require that the physical NIC’s mac addresses remain unchanged, we had to tweak Xen’s bridging scripts to allow this. Dom0’s network interfaces, which normally are assigned the physical NIC’s mac addresses, get a locally administered mac address out of Xen’s mac address range instead. Beware: you should take care to assign different addresses to all Xen servers on a same subnet, or you may experience connectivity problems to your Dom0’s.

So here’s what we did:

In you Xen config file, found at /etc/xen/xend-config.sxp in our installation, change the default network script to a custom one:

# comment this one out
#(network-script network-bridge)
# add this one
(network-script my-network-bridge)

Then you have to create your custom bridging script. Place it in /etc/xen/scripts/my-network-bridge - or the directory your xen scripts reside in. The script should contain the following lines - the second one is not strictly necessary if you only want top bridge one network interface:

#!/bin/sh
dir=$(dirname "$0")
# The mac addresses you assign here should be different for every server.
# Also, watch out for conflicts with your DomU’s mac addresses.
# Use the 00:16:3e:xx:xx:xx range, which is Xen’s assigned range
"$dir/network-bridge" "$@" vifnum=0 macaddr=00:16:3e:00:00:00
"$dir/network-bridge" "$@" vifnum=1 macaddr=00:16:3e:00:00:01

Then finally, modify your network-bridge script like this - the changes are outlined in bold:

#!/bin/sh
#============================================================================
# Default Xen network start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates a bridge (default xenbr${vifnum}), adds a device
# (default eth${vifnum}) to it, copies the IP addresses from the device
# to the bridge and adjusts the routes accordingly.
#
# If all goes well, this should ensure that networking stays up.
# However, some configurations are upset by this, especially
# NFS roots. If the bridged setup does not meet your needs,
# configure a different script, for example using routing instead.
#
# Usage:
#
# network-bridge (start|stop|status) {VAR=VAL}*
#
# Vars:
#
# vifnum     Virtual device number to use (default 0). Numbers >=8
#            require the netback driver to have nloopbacks set to a
#            higher value than its default of 8.
# bridge     The bridge to use (default xenbr${vifnum}).
# netdev     The interface to add to the bridge (default eth${vifnum}).
# antispoof Whether to use iptables to prevent spoofing (default no).
#
# Internal Vars:
# pdev="p${netdev}"
# vdev="veth${vifnum}"
# vif0="vif0.${vifnum}"
#
# start:
# Creates the bridge
# Copies the IP and MAC addresses from netdev to vdev
# Renames netdev to be pdev
# Renames vdev to be netdev
# Enslaves pdev, vdev to bridge
#
# stop:
# Removes netdev from the bridge
# Transfers addresses, routes from netdev to pdev
# Renames netdev to vdev
# Renames pdev to netdev
# Deletes bridge
#
# status:
# Print addresses, interfaces, routes
#
#============================================================================

dir=$(dirname "$0")
. "$dir/xen-script-common.sh"
. "$dir/xen-network-common.sh"

findCommand "$@"
evalVariables "$@"

vifnum=${vifnum:-$(ip route list | awk ‘/^default / { print $NF }’ | sed ’s/^[^0-9]*//’)}
vifnum=${vifnum:-0}
bridge=${bridge:-xenbr${vifnum}}
netdev=${netdev:-eth${vifnum}}
antispoof=${antispoof:-no}
# add a new macaddr parameter to the script
macaddr=${macaddr:-ff:ff:ff:ff:ff:ff}
pdev="p${netdev}"
vdev="veth${vifnum}"
vif0="vif0.${vifnum}"

get_ip_info() {
    addr_pfx=`ip addr show dev $1 | egrep ‘^ *inet’ | sed -e ’s/ *inet //’ -e ’s/ .*//’`
    gateway=`ip route show dev $1 | fgrep default | sed ’s/default via //’`
}

do_ifup() {
    if ! ifup $1 ; then
        if [ ${addr_pfx} ] ; then
            # use the info from get_ip_info()
            ip addr flush $1
            ip addr add ${addr_pfx} dev $1
            ip link set dev $1 up
            [ ${gateway} ] && ip route add default via ${gateway}
        fi
    fi
}

# Usage: transfer_addrs src dst
# Copy all IP addresses (including aliases) from device $src to device $dst.
transfer_addrs () {
    local src=$1
    local dst=$2
    # Don’t bother if $dst already has IP addresses.
    if ip addr show dev ${dst} | egrep -q ‘^ *inet ‘ ; then
        return
    fi
    # Address lines start with ‘inet’ and have the device in them.
    # Replace ‘inet’ with ‘ip addr add’ and change the device name $src
    # to ‘dev $src’.
    ip addr show dev ${src} | egrep ‘^ *inet ‘ | sed -e "
s/inet/ip addr add/
s@$[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+/[0-9]\+$@\1@
s/${src}/dev ${dst}/
" | sh -e
    # Remove automatic routes on destination device
    ip route list | sed -ne "
/dev ${dst}$ \|$$/ {
s/^/ip route del /
p
}" | sh -e
}

# Usage: transfer_routes src dst
# Get all IP routes to device $src, delete them, and
# add the same routes to device $dst.
# The original routes have to be deleted, otherwise adding them
# for $dst fails (duplicate routes).
transfer_routes () {
    local src=$1
    local dst=$2
    # List all routes and grep the ones with $src in.
    # Stick ‘ip route del’ on the front to delete.
    # Change $src to $dst and use ‘ip route add’ to add.
    ip route list | sed -ne "
/dev ${src}$ \|$$/ {
h
s/^/ip route del /
P
g
s/${src}/${dst}/
s/^/ip route add /
P
d
}" | sh -e
}

##
# link_exists interface
#
# Returns 0 if the interface named exists (whether up or down), 1 otherwise.
#
link_exists()
{
    if ip link show "$1" >/dev/null 2>/dev/null
    then
        return 0
    else
        return 1
    fi
}

# Set the default forwarding policy for $dev to drop.
# Allow forwarding to the bridge.
antispoofing () {
    iptables -P FORWARD DROP
    iptables -F FORWARD
    iptables -A FORWARD -m physdev –physdev-in ${pdev} -j ACCEPT
    iptables -A FORWARD -m physdev –physdev-in ${vif0} -j ACCEPT
}

# Usage: show_status dev bridge
# Print ifconfig and routes.
show_status () {
    local dev=$1
    local bridge=$2

    echo ‘============================================================’
    ip addr show ${dev}
    ip addr show ${bridge}
    echo ‘ ‘
    brctl show ${bridge}
    echo ‘ ‘
    ip route list
    echo ‘ ‘
    route -n
    echo ‘============================================================’
}

op_start () {
    if [ "${bridge}" = "null" ] ; then
        return
    fi

    if ! link_exists "$vdev"; then
        if link_exists "$pdev"; then
            # The device is already up.
            return
        else
            echo "
Link $vdev is missing.
This may be because you have reached the limit of the number of interfaces
that the loopback driver supports. If the loopback driver is a module, you
may raise this limit by passing it as a parameter (nloopbacks=<N>); if the
driver is compiled statically into the kernel, then you may set the parameter
using loopback.nloopbacks=<N> on the domain 0 kernel command line.
" >&2
            exit 1
        fi
    fi

    create_bridge ${bridge}

    if link_exists "$vdev"; then
        mac=`ip link show ${netdev} | grep ‘link\/ether’ | sed -e ’s/.*ether $..:..:..:..:..:..$.*/\1/’`
        preiftransfer ${netdev}
        transfer_addrs ${netdev} ${vdev}
        if ! ifdown ${netdev}; then
            # If ifdown fails, remember the IP details.
            get_ip_info ${netdev}
            ip link set ${netdev} down
            ip addr flush ${netdev}
        fi
        ip link set ${netdev} name ${pdev}
        ip link set ${vdev} name ${netdev}

#       do not use the default bridging function as this resets the mac address
#       setup_bridge_port ${pdev}

#       instead use the code below, which leaves the mac address intact
        ip link set ${pdev} down
        ip link set ${pdev} arp off
        ip link set ${pdev} multicast off
        ip addr flush ${pdev}
        setup_bridge_port ${vif0}

#       do not assign pethX’s mac address to Dom0’s ethX
#       ip link set ${netdev} addr ${mac} arp on

#       instead use the $macaddr parameter we passed to the script
        ip link set ${netdev} addr ${macaddr} arp on
        ip link set ${bridge} up
        add_to_bridge ${bridge} ${vif0}
        add_to_bridge2 ${bridge} ${pdev}
        do_ifup ${netdev}
    else
        # old style without ${vdev}
        transfer_addrs ${netdev} ${bridge}
        transfer_routes ${netdev} ${bridge}
    fi

    if [ ${antispoof} = 'yes' ] ; then
        antispoofing
    fi
}

op_stop () {
    if [ "${bridge}" = "null" ]; then
        return
    fi
    if ! link_exists "$bridge"; then
        return
    fi

    if link_exists "$pdev"; then
        ip link set dev ${vif0} down
        mac=`ip link show ${netdev} | grep ‘link\/ether’ | sed -e ’s/.*ether $..:..:..:..:..:..$.*/\1/’`
        transfer_addrs ${netdev} ${pdev}
        if ! ifdown ${netdev}; then
            get_ip_info ${netdev}
        fi
        ip link set ${netdev} down arp off
        ip link set ${netdev} addr fe:ff:ff:ff:ff:ff
        ip link set ${pdev} down
        ip addr flush ${netdev}

# do not reassign pethX’s mac address, since it hasn’t changed
#       ip link set ${pdev} addr ${mac} arp on
        brctl delif ${bridge} ${pdev}
        brctl delif ${bridge} ${vif0}
        ip link set ${bridge} down

        ip link set ${netdev} name ${vdev}
        ip link set ${pdev} name ${netdev}
        do_ifup ${netdev}
    else
        transfer_routes ${bridge} ${netdev}
        ip link set ${bridge} down
    fi
    brctl delbr ${bridge}
}

# adds $dev to $bridge but waits for $dev to be in running state first
add_to_bridge2() {
    local bridge=$1
    local dev=$2
    local maxtries=10

    echo -n "Waiting for ${dev} to negotiate link."
    ip link set ${dev} up
    for i in `seq ${maxtries}` ; do
        if ifconfig ${dev} | grep -q RUNNING ; then
            break
        else
            echo -n ‘.’
            sleep 1
        fi
    done

    if [ ${i} -eq ${maxtries} ] ; then echo ‘(link isnt in running state)’ ; fi

    add_to_bridge ${bridge} ${dev}
}

case "$command" in
    start)
        op_start
        ;;

    stop)
        op_stop
        ;;

    status)
        show_status ${netdev} ${bridge}
        ;;

    *)
        echo "Unknown command: $command" >&2
        echo ‘Valid commands are: start, stop, status’ >&2
        exit 1
esac

Finally, comment out the mac addresses in the ifup scripts, else your OS will complain about setting up eth0 and eth1 with different mac addresses than those mentioned in the ifup scripts:

# Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.255.255
#HWADDR=00:1A:A0:05:12:AB
IPADDR=192.168.0.11
NETMASK=255.255.0.0
NETWORK=192.168.0.0
ONBOOT=yes

Restart your services as needed, and you’ll be able to connect to your network interfaces without a hitch!

No Comments

Monitoring disk usage with Nagios passive checks and SNMP traps

Why passive checks and SNMP traps?

Setting up SNMP on the monitored host

Setting up snmptrapd on the Nagios server

Setting up SNMPTT on the Nagios server

Setting up Nagios for passive checks

Troubleshooting

Getting Xen networking to work on a Dell Poweredge 2950

What happened to our networking?

Dell, Broadcom and the art of technical support

Two solutions for the problem

Recent Articles

Categories

Archives

Meta