Using cisco’s embedded event manager to aid in troubleshooting

In my previous post I explained a setup in which multiple cisco nodes send syslog messages to a centralised syslog server which network admins would check during troubleshooting. As an extension to that, we had another requirement proposed. To facilitate troubleshooting, we wanted to display the interface description when an error occurred on an interface. The description would let us know which node / user was connected to the interface, and help to pinpoint where the exact problem was. Till now this was done manually, i.e. when errors occurred on an interface the admins would login to the cisco box and use the sh int gig0/1 or similar commands to see the description. We wanted to display this information automatically in the syslog messages sent to the webmin site.

This was possible by using the embedded event manager (EEM) that is present on later versions of cisco IOS. There is a high level overview of EEM here:

http://www.cisco.com/web/about/security/intelligence/embedded-event-mgr.html

In a nutshell, the EEM enables you to react to anything on that occurs on the control plane of a cisco node. So you can do things such as deny and log commands, send SNMp traps, syslog, emails and so on whenever the cisco detects an event.

below i’ll present the EEM script I used to send the interface description when an interface event is detected.

The script is loosely based on the one on this website:

http://wiki.nil.com/Report_interface_loss_based_on_OSPF_neighbor_loss

I found the tricky part in my script was to understand the use of regex… so having a quick read of the following would help you understand:

http://wiki.nil.com/Regular_expressions_in_Embedded_Event_Manager_applets

Here’s the script used:

event manager applet syslog-int-test
event syslog occurs 1 pattern "GigabitEthernet" period 1
action 100  regexp "GigabitEthernet([0-9/]+)" "$_syslog_msg" ifname
action 210  cli command "show interface $ifname | include Description:"
action 230  regexp "Description: (.*)\r" "$_cli_result" desc
action 240  syslog msg "$desc"

– The first line defines the name of the “applet”. syslog-int-test in my case

– The second line informs the cisco when to execute the applet. In this case, the applet should be run when the cisco generates a syslog message with the word “GigabitEthernet”

action 100 : this action uses a regex (“GigabitEthernet([0-9/]+)") to detect which interface is quoted in the syslog message (“$_syslog_msg” : this is a default environment variable in EEM), and stores this in the variable “ifname”

action 210 : this action runs the cli command containing the interface and uses the “include” command to output only the description line

action 230 : this action, similar to action 100, uses a regexp ("Description: (.*)\r") to search the command line result ("$_cli_result") and places the description in a variable named “desc”

action 240 : this action then instructs the cisco to send a syslog message containing the contents of the “desc” variable

Here are the results:

result

You can see that whenever an interface event occurs, we now see the interface description in the subsequent line (note the name of the applet “syslog-int-test” is also displayed)

🙂