Sunteți pe pagina 1din 84

Embedded Event

Manager

Joe Horgan, CCIE #16542


email: joehorga@cisco.com
Cisco Advanced Services

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1
Destination Check
This session is not
an introduction to NMS concepts
engineering internals of SNMP
a session on programming or scripting
(although we do assume some understanding of basic
programming/scripting any knowledge of Tcl and XML will be
advantageous)
This session is
a technical session about a really cool embedded
instrumentation built into Cisco devices
full of examples and sample scripts to help you understand
the power of these features

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2
Agenda
Introduction
Embedded Event Manager
Practical Applications
Summary and Conclusion

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 3
Embedded Event Manager (EEM)
Overview
Service running in Cisco IOS
Provides the opportunity to do self-monitoring reduce or eliminate polling
Let the network elements monitor themselves and send the right fault management events
when there is a problem

Allows us to watch for specific events and then take specific actions when
those events occur
Actions can be relatively simple or complex
Advantages:
Ability to take corrective actions automatically when a problem is detected
Build custom automation right on the device
Can reduce dependence on external NMSs (or make them more powerful!)
Reduce network bandwidth by doing local event monitoring

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 4
Embedded Instrumentation Model
vs. External View
Connectivity to external systems may not always Si
Si Si
Si Si
Si

be available or reliable
Internal scripts are distributed and localized: SiSi SiSi

Each router or switch in the network has its own set of


policies
Information can be distilled and forwarded to external
monitoring systems Interpret
Problems can be handled locally from
Polling vs. event notification outside
Policies are triggered without having to constantly poll Vs.
No need to listen and filter error messages
See from
Instant reaction when a condition is detected within
No need to relay the information outside the system and
wait for response

Combining embedded instrumentation with


external management tools creates a powerful
NMS solution!
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 5
EEM Basic Architecture

Policies
Actions to take
(Tcl script or applet)

EEM Server
brains of the system

Event Detectors
Watch for events of interest

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6
EEM Event Detectors and EEM Policies
All of this is internal to Cisco IOS

Think of a policy
as an action ED notifies EEM
registered to an Server; which
event triggers interested
policies

Applet-based policies Tcl-based policies


Defined via CLI Programmed in Tcl
Simpler As complex as you want

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7
So what Can I Do with EEM?
Simple Example:
Write a syslog message when CPU utilization goes above 80%
An Implementation (there are others):
Determine the SNMP OID to monitor
Configure an EEM applet in IOS CLI
Define the SNMP event detector with the appropriate threshold
Define the syslog action to generate a syslog message
Optional: include the actual value of the OID when event detected
Result:
SNMP Event Detector will continually monitor the SNMP OID configured and
when the utilization crosses 80%, a syslog message will be generated
We now have proactive monitoring on the box!

But we are only scratching the surface of what is possible

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 8
EEM 1.0 Event Detectors
SNMP
Monitors a MIB object and generates an event when threshold crossed

Syslog
Matches a syslog message based on a regular expression match
Available options, an event is raised when:
1 message is detected, or
n messages are detected, or
n messages are detected within a specific period of time

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 9
EEM 2.0 Event Detectors
Application-specific
Allows IOS applications or EEM policies to define and publish
application specific events
Counter
Provides persistent EEM counters that can be set by policies
A policy can be triggered when a specific counter crosses a threshold
A way for policies to communicate and trigger other policies. Example:
Policy A increments an EEM counter when it runs. Policy B is triggered
when the EEM counter crosses a threshold
Interface Counter
Generates an event when a specific IDB port generic statistics counter
crosses a threshold, up or down.
Provides an easier way to track interface statistics than through the
SNMP Event Detector

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 10
EEM 2.0 Event Detectors (Cont.)

Timer
Generates an event at an absolute time (absolute).
Generates an event after a specific period (countdown).
Generates a recurring event (watchdog - a countdown timer that
is automatically re-armed at the event).
Generates an event using a Unix CRON specification (cron -
Similar to Unix cron job

Watchdog
Triggers policies based on certain conditions relative to a
certain Cisco IOS process or subsystems activity

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 11
EEM 2.1 Event Detectors
CLI
Screens CLI commands for a regular expression pattern match.
Publishes an event upon a valid match after the command is
successfully parsed and before it is executed.
Can be used to augment or replace a command.
None
Used as a placeholder for policies that are manually triggered
via the event manager run <policyname> command
Online Insertion and Removal (OIR)
Publishes events when one of the following hardware
insertion/removal events occurs:
When a card is removed
When a card is inserted

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 12
EEM 2.1.5 Event Detectors

Generic Online Diagnostic (GOLD)


Provides generic HW fault detection framework (currently
implemented on Cat6k)

System Manager
Generates events for IOS Modularity process start,
normal/abnormal stop, and restart events.
Allows the policy to change the default behavior of process
restart

Watchdog
Generates events when IOS Software Modularity Watchdog
System Monitor (WDSYSMON) detects infinite loops,
deadlocks, and memory leaks in Cisco IOS Software Modularity
processes
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 13
EEM 2.2 Event Detectors
Redundancy Framework
Generates events for all RF (Redundancy Framework)
notifications and state transitions
Resource
Interfaces with the IOS Embedded Resource Manager (ERM)
sub-system.
Shared resource threshold notifications both in the up and down
direction.
Resources to monitor:
Memory
CPU
IPC
Buffers
File System
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14
EEM 2.2 Event Detectors (Cont.)

Enhanced Object Tracking (EOT)


Interfaces with the Enhanced Object Tracking sub-system
(EOT) to trigger policies based on tracked object state changes
Tracking objects are defined via CLI commands
Tracking subsystem polls objects for status
Clients (HSRP, GLBP, VRRP) register with tracking subsystem
Status changes are communicated to clients:
Immediately or
After specified delay

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15
Embedded Event Manager Versions
IOS EEM 1.0 IOS EEM 2.3
Small footprint Single Source

CLI based policies (Applet


IOS EEM 2.4
Interface) Multi-Event support
Limited Event Detectors Two new EDs
IOS EEM 2.0 Bytecode support
Tcl based policies IOS EEM 3.0
Additional Event Detectors Four new EDs
Performance enhancements
IOS EEM 2.1
Variable logic for Applets
User Tcl based policies Digital signature support
IOS EEM 2.2 Distributed EEM
Two new EDs IPv6

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 16
Security Of The Embedded Event
Manager

All config commands are privileged global config mode


commands.
If users modify Cisco scripts, they need to be run from
the user directory in user mode.
When EEM policies execute CLI commands, the event
manager session cli username <username> command
specifies the username that is sent to TACACS+.

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 17
EEM Policies
An EEM policy is self-contained:
Contains execution criteria
Contains actions
Policies are viewed as short-lived (default elapsed run
time is 20 seconds)
Two Engines:
CLI Based (Applet Interface)
Script Based (Tcl)
Two Policy Types:
Synchronous policy can affect the outcome of the event
Asynchronous policy runs asynchronously with the event

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18
EEM Built-In Actions
An Embedded Event Manager Policy can:
Execute an IOS CLI command and receive the result
Send a CNS event
Increment or decrement an EEM counter
Force a switchover to the standby in a redundant configuration
Request system information
Send an e-mail
Cause another EEM policy to be executed
Publish an application specific EEM event
Reload the box
Send an SNMP trap with custom data
Log a message to Syslog
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 19
EEM Environment Variables

Variables that are referenced within policies that can be


set in the config
Can be used to customize policies
Example:
Router#config t
Router(config)#event manager environment _email_server
email.cisco.com
Router(config)#event manager environment _email_from
soandso@somecompany.com

Note: environment variable names that start with _ are


reserved for Cisco use only

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 20
Some Environment Variables
Environment Variables Available for All Events
$_event_type The event type that triggered the event.
$_event_pub_time The time at which the event type was published.
Environment Variables Available for SNMP Events
$_snmp_oid The Simple Network Management Protocol (SNMP) object ID that caused the event to be published.
$_snmp_oid_val The SNMP object ID value when the event was published.
Environment Variables Available for Syslog Events
$_syslog_msg The syslog message that caused the event to be published.

When an applet is entered


Cisco defined read-only environment variables called built-in
variables are pre-set with the characteristics of the event that
triggered the policy to run
These environment variables can be used in msg text
Will be replaced with the relevant text
Can be checked by policies
There are others see the docs
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 21
Lets Configure a Simple EEM Policy

When we see a particular syslog message, write a


special syslog message.
For example:
When someone leaves config mode, this message is seen:

Lets generate a message:


Configuration event occurred
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 22
Configure an EEM Applet Policy

We can make an Applet policy to match on the syslog


message
Uses a regular expression match
We will register our policy SYSLOG EVENT
Syslog Event Detector will notify the EEM Server when the
message string match occurs
Our policy action will be invoked SYSLOG ACTION
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 23
Applet Configuration Detail
event manager applet CFGMSG

Define applet
named
CFGMSG

Event type
will be
syslog

iin-rtr1(config-applet)#event syslog

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 24
Syslog REGEXP Match Pattern

iin-rtr1(config-applet)#event syslog pattern "%SYS-5-CONFIG_I:"

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 25
Add the ActionSyslog Action

iin-rtr1(config-applet)#action 1.0 syslog

Label - used to
sort actions

Alphabetic sort
on the label

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 26
Complete Applet Policy Definition
event manager applet CFGMSG
event syslog pattern "{%SYS-5-CONFIG_I:}"
action 1.0 syslog priority warnings msg
"Configuration event occurred"

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 27
Tcl Policies or Scripts
Tcl Policies can do everything that applets can, and more!
Standard Tcl 8.3.4 script support
Tcl Manual available at http://www.tcl.tk/man/tcl8.3/
This is the same support available in IOS for tclsh, ESM (Embedded Syslog
Manager), and IVR
There are several Tcl language extension dialects and some differences among
them
More details and examples on Tcl and tclsh available in the Appendix

EEM keyword extensions as documented in the Guide To Writing


EEM Policies documentation implement event specification,
system information requests, and built-in actions
EEM Tcl library support is available for some common functions
such as CLI, SMTP, and Tcl global variable state checkpointing.
EEM uses the ::cisco::eem Tcl namespace
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 28
Cisco Built-In Scripts
sl_intf_down.tcl
Monitors for a specific syslog message and can run CLI
commands
Originally designed to run CLI commands upon receipt of an
interface-down syslog messages
tm_cli_cmd.tcl
Allows a CLI command (e.g. show command) to be run
periodically and have the results emailed to someone
tm_crash_reporter.tcl
Triggers 5s after boot up and sends crash information, if
relevant, to a specified URL
To see list of built-in (system) scripts:
show event manager policy available system

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 29
EEM Tcl Policy Structure
Begin with EEM Event
Register keyword
Required Next is any input
variables or required
environment variables
to control the script
Names space imports
Required
Entry criteria for the
policy
Body (logic of the
Required
script)
Exit status

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 30
Example of Tcl Policy Structure
Registration command
(Tcl extension)
Tcl namespace
(namespace import)
::cisco::eem
This namespace includes
all Tcl commands closely
related to Embedded Event
Manager
::cisco::lib
This namespace includes
auxiliary library commands
that are not necessarily
specific to the Embedded
Event Manager
Body
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 31
Getting Started:
Create Policy Directory

ashcroft#mkdir ABCCoTclPol
Create directory filename [ABCCoTclPol]?
Created dir disk0:ABCCoTclPol

ashcroft#dir
Directory of disk0:/
1 drw- 1 Oct 26 2003 13:37:42 +00:00 sys
6 drw- 1 Oct 30 2003 12:56:04 +00:00 ABCCoTclPol
47843328 bytes total (29356032 bytes free)

ashcroft#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ashcroft(config)#event manager directory user policy disk0:/ABCCoTclPol
ashcroft(config)#^Z

ashcroft#sh event man dir user policy


disk0:/ABCCoTclPol
ashcroft#

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 32
Getting Started:
Copy Tcl Policy to Router

ashcroft#copy tftp disk0:


Address or name of remote host []? 88.1.88.9
Source filename []? sl_cfgSaveRemT.tcl
Destination filename [sl_cfgSaveRemT.tcl]? ABCCoTclPol/sl_cfgSaveRemT.tcl
Accessing tftp://88.1.88.9/sl_cfgSaveRemT.tcl...!
1232 bytes copied in 0.620 secs (1987 bytes/sec)

ashcroft#dir
Directory of disk0:/
1 drw- 1 Oct 26 2003 13:37:42 +00:00 sys
6 drw- 1 Oct 30 2003 12:56:04 +00:00 ABCCoTclPol
47843328 bytes total (29351936 bytes free)

ashcroft#cd ABCCoTclPol
ashcroft#dir
Directory of disk0:/ABCCoTclPol/
8 -rw- 1232 Oct 30 2003 14:14:58 +00:00 sl_cfgSaveRemT.tcl
47843328 bytes total (29351936 bytes free)
ashcroft#

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 33
Getting Started:
Register the Policy

ashcroft#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ashcroft(config)#event manager policy sl_cfgSaveRemT.tcl type user
ashcroft(config)#

ashcroft#sh event manager policy registered


No. Type Event Type Trap Time Registered Name
1 user syslog Off Thu Oct30 14:54:17 2003 sl_cfgSaveRem.tcl
occurs 1 pattern {%SYS-5-CONFIG_I: Configured}
nice 0 priority normal maxrun 90.000

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 34
Some Other Ideas for Using EEM
Every few days you lose all OSPF neighbors
EEM script could be triggered on the OSPF messages in SYSLOG, capture the information,
save it to flash and reload the router (if it doesnt correct itself)
Every few weeks a router is running low on memory around 2am
EEM script could be triggered based on the memory utilization, capture the memory
information and send the output with Syslog
One of the (redundant) links has problems with occasional high error
rates, but does not go down
EEM script could be triggered on the interface errors, remove the link from using it and send
a notification by SNMP or Syslog or email
You need to automatically update a ACL every night at 2am
EEM script could be triggered by cron timer, grab new version of ACL from a web server
and apply it (and send a confirmation by syslog/email/snmp)
One of the BGP peers sometimes drops with hold timeout, but
recovers within seconds. How do you troubleshoot?
EEM script could be triggered by Syslog message and verify connectivity to other BGP peer
with a ping at the time of the problem and possible look at the interface status and save a
traceroute output to the other BGP peer

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 35
EEM Security
Cisco scripts run in full Tcl mode
User scripts run in Safe-Tcl mode
Allows Cisco to disable/customize specific Tcl commands
Provides restrictions to ensure system integrity
Built-in throttle that periodically suspends execution
User scripting disabled by disallowing command:
event manager directory user
All config commands are privileged global config mode
To control the user that runs an EEM policy, use:
event manager session cli username <username>
username sent to TACACS+ for command authorization

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 36
Embedded Event Manager (EEM)
Debugging and Show Commands

Debug commands
debug event manager tcl cli_lib
debug event manager tcl commands
debug event manager tcl smtp_library

Note: EEM delivers debug to syslog at the debugging level

Show commands
show event manager policy available
show event manager directory user policy

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 37
EEM Version Comparison
EEM Version 1.0 2.0 2.1 2.1.5 2.2
12.3(14)T1,
12.2(18)SXF4
12.3(4)T, 12.2(28)SBC, 12.4(2)T,
IOS Version Introduced 12.0(26)S
12.2(27)SBC
12.2(18)SXF5,
(IOS with
12.2(33)SRB1
modularity)
12.2(33)SRA
Syslog, SNMP EDs X X X X X
Syslog, SNMP Actions X X X X X
Watchdog, Counter,
Interface Counter,
X X X X
Timer, Application-Specific
EDs
Counter Modification,
System Info, X X X X
Email Actions
OIR, CLI EDs X X X
User and System Tcl
X X X
Policies
GOLD, System Manager,
X X
WDSysMon EDs
Resource, RF, EOT EDs X

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 38
EEM Today and Tomorrow
2.3
Latest release available on 12.4(11)T
No new features added
Single source code used across products
2.4
Coming soon
Multiple event support
Allows event correlation where multiple events can trigger a policy
SNMP Proxy ED
Allows device to receive an SNMP trap/inform and execute a policy
Policy has access to varbind info in the trap
RPC ED
Allows external entity to send a SOAP message over SSHv2 to
invoke a policy and return the result to the external entity
And more!
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 39
CISCO IOS EMBEDDED EVENT MANAGER
EEM VERSION - PRODUCT MATRIX

6/8/07 4:39 PM

Legend Shipping In EFT EC Planning N/A


CISCO ACCESS ROUTERS
Cisco
EEM Cisco 800 Cisco 1800 Cisco 2800 Cisco 3800 Cisco 1700 Cisco 2600 Cisco 2691 Cisco 3600 Cisco 3700
2600XM
Version Series Series Series Series Series Series Series Series Series
Series
1.0 12.3(11)T 12.3(11)T 12.3(11)T 12.3(4)T 12.3(4)T 12.3(4)T 12.3(4)T
2.0
2.1 12.3(14)T1 12.3(14)T1 12.3(14)T1 12.3(14)T1 12.3(14)T1 12.3(14)T1 12.3(14)T1 12.3(14)T1 12.3(14)T1
2.1.5
2.2 12.4(2)T 12.4(2)T 12.4(2)T 12.4(2)T 12.4(2)T 12.4(2)T 12.4(2)T 12.4(2)T 12.4(2)T 12.4(2)T
2.3 12.4(11)T 12.4(11)T 12.4(11)T 12.4(11)T 12.4(11)T 12.4(11)T 12.4(11)T 12.4(11)T 12.4(11)T 12.4(11)T
2.4 12.5(pi-1)T 12.5(pi-1)T 12.5(pi-1)T 12.5(pi-1)T Planning Planning Planning Planning Planning Planning

CISCO 5000 SERIES & UP


EEM Cisco 7200 Cisco 7600 Cisco 12000 Cisco XR Cisco Cisco 7500 Cisco 5000
Cisco 7301 Cisco 7304 Cisco 10000
Version Series Series Series 12000 CRS-1 Series Series
1.0 12.0(26)S 12.0(26)S
See IOS-XR See IOS-XR
2.0 12.2(27)SBC
Fault Mgr Fault Mgr
12.2SB See IOS-XR See IOS-XR
2.1 12.3(14)T1 12.3(14)T1 12.2(28)SB 12.2(18)SXF5 12.4M
2H06 Fault Mgr Fault Mgr
See IOS-XR See IOS-XR
2.1.5
Fault Mgr Fault Mgr
See IOS-XR See IOS-XR
2.2 12.4(2)T 12.4(2)T1
Fault Mgr Fault Mgr
12.2SB 12.2SB 12.2SR 12.2SB
See IOS-XR See IOS-XR
2.3 12.4(11)T 1H07 1H07 1H07 1H07 12.4(11)T
Fault Mgr Fault Mgr
12.2(rls5)SB 12.2(rls5)SB Barracuda 12.2(rls5)SB
12.2(SR) 12.2(SR) 12.2(SR) See IOS-XR See IOS-XR
2.4 12.5(pi-1)T 12.2(rls6)SB Planning
Dragon Dragon Dragon Fault Mgr Fault Mgr

CISCO CATALYST SWITCHES


EEM Cisco 3750 Cisco 4500 Cisco 6500
Version Switches Switches Switches
1.0
2.0
IOS w/o
2.1 Modularity
12.2(18)SXF5
w/ Modularity
2.1.5
12.2(18)SXF4
2.2
2.3 Whitney 1
12.2(??)SE 12.2(x)SG
2.4 Whitney 2
Fall '07 1Q08
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 40
IDEEM
The EEM Development And Management Tool

Developed by Nidus Software Corp.


Provides an integrated tool for development, test,
management EEM policies
http://www.nidussoft.com/

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 41
Embedded Event Manager (EEM)
Additional References
EEM Starting Point:
http://www.cisco.com/go/eem
EEM Overview:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios12
2/122newft/122limit/122sx/122sxf18/evnt_mgr/nmb_eemo.htm
Writing EEM Policies Using IOS CLI:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios12
2/122newft/122limit/122sx/122sxf18/evnt_mgr/nmb_eemc.htm
Writing EEM Policies Using Tcl:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios12
2/122newft/122limit/122sx/122sxf18/evnt_mgr/nmb_eemt.htm

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 42
Cisco BeyondProduct Extension Community
EEM Scripting Community

New on Cisco.com
Open source
scripts, share,
upload, download,
learn by example
Categories include:
Ntwk mgmt, routing,
QoS, High
availability, User
interface, etc
Comments, ratings,
community
managed forum

http://cisco.com/go/ciscobeyond

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 43
Example 1
Using EEM to Collect Interface Error History

Requirement:
Collect interface errors regularly and append output to a file on flash
CLI command to use
remote-pe#sh int fa2/0 | inc errors
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 3 interface resets

Solution:
Write a Tcl policy that:
Uses Timer Event Detector watchdog timer will provide a recurring
event
Appends a timestamp and command output to a history file
Uses the following environment variables as parameters:
Run Period (time between successive samples)
Output Filename
Interface (to run show command on)
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 44
The Tcl Policy
Watchdog Timer ED called errimt that
runs every $errim_period seconds

::cisco::eem::event_register_timer watchdog name errimt time $errim_period


# _________________________________________________________________________#
# | |#
# | errim |#
# | Script name |#
# | by: Rick Williams |#
# | 2006, Cisco Systems, Inc. |#
# | |#
# | Version 01.0 05/05/2006 |#
# |_______________________________________________________________________|#
#
# errim - timed interface error monitor
#
# desc: The purpose is to monitor the output from cli command:
# "show interface <int> | inc errors" and save the output to a
# history file on flash disk along with a timestamp
#
# trigger: Watchdog timer - set from env variable
# action: Write history file
# env vars: errim_period - time interval for execution (seconds)
# errim_outfile - file to write history to
# errim_int - interface to monitor
# This script can be
downloaded from

Cisco Beyond
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 45
The Tcl Policy (Cont.)
# Check for required environment variables
# If any required vars are not available, print error and quit
#
if {![info exists errim_period]} {
set result "EEM Policy Error: variable errim_period has not been set"
error $result $errorInfo
}
if {![info exists errim_outfile]} {
set result "EEM Policy Error: variable errim_outfile has not been set"
error $result $errorInfo
}
if {![info exists errim_int]} {
set result "EEM Policy Error: variable errim_int has not been set"
error $result $errorInfo
}
#
# namespace imports Check that mandatory variables have been set
#
namespace import ::cisco::eem::*
namespace import ::cisco::lib::*
#
Import namespaces to include EEM
and auxiliary Tcl commands

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 46
The Tcl Policy (Cont.)
Opens a channel and returns the
channel handler, fd and tty_id
# issue the cli command
#
if [catch {cli_open} result] {
error $result $errorInfo
} else {
array set cli1 $result Execute CLI command using channel
} handler, fd returned by cli_open
# save exact execution time for command
set time_now [clock seconds]
# execute command
if [catch {cli_exec $cli1(fd) "show int $errim_int | inc errors"} result] {
error $result $errorInfo
}
set cmd_output $result Save the command output
#
if [catch {cli_close $cli1(fd) $cli1(tty_id)} result] {
error $result $errorInfo
}
# Close the channel using channel handler,
action_syslog msg "errim command executed" fd and tty_id returned by cli_open
#
#

Send a syslog message

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 47
The Tcl Policy (Cont.)
# prepare to write line to the history file Open the output file for appending
# and return a file descriptor
if [catch {open $errim_outfile a+} result] {
error $result
} Set the fileD to the file descriptor
set fileD $result
#
# save timestamp of command execution
# (Format = 00:53:44 PDT Mon May 02 2005)
set time_now [clock format $time_now -format "%T %Z %a %b %d %Y"]
puts $fileD "%%% $time_now, errin executed"
#
# Write command output
# Write the timestamp
puts $fileD $cmd_output
#
close $fileD Write the command output

Close the output file

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 48
Configuration Required
Set the environment variables

event manager environment errim_period 15


event manager environment errim_outfile disk2:errim_history.dat
event manager environment errim_int FastEthernet2/0
!
event manager applet goeim Create an applet to register policy
(Uses None ED)
event none
action 1.0 syslog msg "enable errim"
action 2.0 cli command "enable"
action 2.1 cli command "conf t"
action 2.2 cli command "event manager policy errim.tcl type user"
action 2.3 cli command "exit"
! Create an applet to de-register policy
event manager applet stopeim (Uses None ED)
event none
action 1.0 syslog msg "disable errim"
action 2.0 cli command "enable"
action 2.1 cli command "conf t"
action 2.2 cli command "no event manager policy errim.tcl type user"
action 2.3 cli command "exit"
! Aliases allow easy start/stop control
of EEM policies from command line
alias exec goerrim event manager run goeim
alias exec stoperrim event manager run stopeim
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 49
Example Run
remote-pe#goerrim
remote-pe#
May 5 13:51:45.937: %HA_EM-6-LOG: goeim: enable errim
May 5 13:51:46.053: %HA_EM-6-LOG: config_audit: Config mode entered via
configure terminal
May 5 13:51:46.177: %SYS-5-CONFIG_I: Configured from console by on vty2
remote-pe#
remote-pe#
May 5 13:52:01.941: %HA_EM-6-LOG:
system:/lib/tcl/eem_scripts_registered/errim.tcl: errim command executed
May 5 13:52:16.805: %HA_EM-6-LOG:
system:/lib/tcl/eem_scripts_registered/errim.tcl: errim command executed
May 5 13:52:31.805: %HA_EM-6-LOG:
system:/lib/tcl/eem_scripts_registered/errim.tcl: errim command executed
May 5 13:52:46.805: %HA_EM-6-LOG:
system:/lib/tcl/eem_scripts_registered/errim.tcl: errim command executed
remote-pe#stoperrim
remote-pe#
May 5 13:52:55.045: %HA_EM-6-LOG: stopeim: disable errim
May 5 13:52:55.157: %HA_EM-6-LOG: config_audit: Config mode entered via
configure terminal
May 5 13:52:55.181: %SYS-5-CONFIG_I: Configured from console by on vty2
remote-pe#
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 50
History Output File

remote-pe#more errim_history.dat Timestamp


%%% 13:52:01 EDT Fri May 05 2006, errin executed
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 3 interface resets
remote-pe>
} Command
Output

%%% 13:52:16 EDT Fri May 05 2006, errin executed


0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 3 interface resets
remote-pe>
%%% 13:52:31 EDT Fri May 05 2006, errin executed
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 3 interface resets
remote-pe>
%%% 13:52:46 EDT Fri May 05 2006, errin executed
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 output errors, 0 collisions, 3 interface resets
remote-pe>

remote-pe#
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 51
Q and A

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 52
Recommended Reading

BRKNMS-3021
Available at www.CiscoPress.com
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 53
Recommended Reading

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 54
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 55
Appendix

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 56
Tool Command
Language (Tcl)

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 57
Tool Command Language (Tcl)
Overview

Tcl is a multithreaded interpreted


scripting language
Language resources found at:
http://www.tcl.tk/

Tcl 7.x has been in Cisco IOS since 1994


Tcl 8.3.4 first released in Cisco IOS in 12.3(2)T
and merged into 12.2(25)S

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 58
Tool Command Language (Tcl)
Features

Support for scripts compiled with the TclPro


bytecode
Support for Tcl namespaces
Allows execution of exec commands and
Cisco IOS configuration

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 59
Tool Command Language (Tcl)
Uses within Cisco IOS

Build custom show commands


Access SNMP objects
Integrate with the Embedded Syslog Manager
and Embedded Event Manager
Build Interactive Voice Response (IVR) scripts
Consolidate complex configuration commands
Autoconfiguration

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 60
Tool Command Language (Tcl)
Starting the Interpreter

Router#tclsh
Router(tcl)#

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 61
Tool Command Language (Tcl)
Configuration

Router(config)#scripting tcl ?
encdir Specify path for Tcl character encoding files
init Specify path for Tcl initialization script
low-memory Configure low water memory mark

The encdir and init values can be any Cisco IOS URI
(e.g. disk:, slot:, tftp:, etc.)
Use the low-memory command to avoid crashes due
to memory allocation (do not go less than 10% of total
available memory)

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 62
Tool Command Language (Tcl)
Configuration (Cont.)
Interactive Shell
Tcl Cisco IOS
Extended Commands
Tcl Built In Command
Cisco IOS Command

Router#tclsh
Router(tcl)#puts "Hello Networkers"
Hello Networkers

Router(tcl)#exit
Router#

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 63
Tool Command Language (Tcl)
Configuration (Cont.)
Running Cisco IOS Commands
Tcl Cisco IOS
Extended Commands
Tcl Built In Command
Cisco IOS Command
Router(tcl)#set output [exec "show interface fa0/0 description"]
Interface Status Protocol Description
Fa0/0 up up FlashNet
Management Connection
Router(tcl)#log_user 0
0
Router(tcl)#set output [exec "show interface fa0/0 description"]

Router(tcl)#puts $output
Interface Status Protocol Description
Fa0/0 up up FlashNet
Management Connection

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 64
Tool Command Language (Tcl)
Configuration (Cont.)
Tcl and CLI Configuration Commands
Tcl Cisco IOS
Extended Commands
Tcl Built In Command
Cisco IOS Command
Router(tcl)#ios_config "interface fa0/0" "description Networkers
Uplink"

Router(tcl)#set output [exec "show interface fa0/0 description"]

Router(tcl)#puts $output
Interface Status Protocol
Description
Fa0/0 up up Networkers
Uplink

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 65
Tool Command Language (Tcl)
Configuration (Cont.)
Writing to the Input Buffer
Tcl Cisco IOS
Extended Commands
Tcl Built In Command
Cisco IOS Command

Router(tcl)#typeahead "show run\n"

Router(tcl)#show run
Building configuration...
Current configuration : 8245 bytes
!
! Last configuration change at 22:05:49 CET Sat Mar 10 2005
!
version 12.0
no service pad

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 66
Tool Command Language (Tcl)
Configuration (Cont.)
Capturing Cisco IOS Errors
Tcl Cisco IOS
Extended Commands
Tcl Built In Command
Cisco IOS Command
Router(tcl)#set line "snmp server community RO"
Router(tcl)#if {[catch {ios_config $line} result]} {
+>puts "Bad config command: \"$line\""
+>}
Bad config command: "snmp server community RO"

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 67
Tool Command Language (Tcl)
Configuration (Cont.)
Loading External Scripts
Tcl Cisco IOS
Extended Commands
Tcl Built In Command
Cisco IOS Command
Router(tcl)#source slot0:myscript.tcl

Router(tcl)#source tftp://10.10.10.10/myscript.tcl

Router#tclsh tftp://10.10.10.10/myscript.tcl

Tip: Keep Common Scripts in a Central


TFTP Archive

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 68
Tool Command Language (Tcl)
SNMP Support
Requires an SNMP community to be configured
on the router
Provides easy access to SNMP objects and commands
snmp_getbulkretrieves a large section of the MIB tree
snmp_getidretrieves the system table
snmp_getnextretrieves the next object in the MIB tree
snmp_getoneretrieves one object in the MIB tree
snmp_setanysets an object in the MIB tree
Data is returned in an XML format
First introduced in 12.3(7)T

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 69
Tool Command Language (Tcl)
SNMP Example
Tcl Cisco IOS
Extended Commands
Tcl Built In Command
Cisco IOS Command
Router(tcl)#snmp_getid public
{<obj oid='system.1.0' val='Cisco IOS Software, 7200 Software
(C7200-JS-M), Version 12.3(14)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Fri 25-Mar-05 14:01 by yiyan'/>}
{<obj oid='system.2.0' val='products.108'/>}
{<obj oid='sysUpTime.0' val='71184284'/>}
{<obj oid='system.4.0' val=Dan Jerome'/>}
{<obj oid='system.5.0' val=dj.cisco.com'/>}
{<obj oid='system.6.0' val=Networkers 2005'/>}

Router(tcl)#snmp_setany private system.6.0 -d "Networkers 2006"


{<obj oid='system.6.0' val='Networkers 2006'/>}

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 70
Tool Command Language (Tcl)
Limitations

The following is a list of differences between


the Tcl 8.3.4 standards and Cisco IOS:

Command Keyword Argument Supported


after ms script Yes
file atime atime No
file mtime mtime No
fileevent Yes*
history !n No
load No

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 71
Tool Command Language (Tcl)
Script Debugging
Use a UNIX or Windows Tcl 8.3 interpreter
to sanity check code
Make sure log_user is set to 1 to get all
possible errors
Use Control+Shift+6 to interrupt a runaway script

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 72
Tool Command Language (Tcl)
Caveats

Use 12.3(14)T or later for best results


Use low-memory to prevent malloc failures
Tcl process runs at medium priority, so be careful with
loops

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 73
Tool Command Language (Tcl)
Security Concerns

No implied trust with Tcl scripts


Load scripts from network servers with care
Use privilege levels to control access to the tclsh
Router(config)#username admin privilege 7 password cisco
Router(config)#username dan privilege 3 password cisco
Router(config)#privilege tcl all level 7 tclsh
Router(config)#line vty 0 903
Router(config-line)#login local

NMS_server% telnet Router


Trying 10.10.10.10...
Connected to Router.cisco.com.
Escape character is '^]'.
User Access Verification
Username: dan
Password:
Router#tclsh
Translating "tclsh"...domain server (10.10.10.10)
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 74
Tool Command Language (Tcl)
Additional References
General language resources:
http://www.tcl.tk/

Cisco IOS scripting with Tcl:


http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/produ
cts_feature_guide09186a00801a75a7.html

Cisco Open Source Initiative (COSI) with scripts found


in this session:
http://sourceforge.net/project/showfiles.php?group_id=25401&p
ackage_id=154317&release_id=332786

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 75
Other Examples

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 76
Tcl Access-List Editor
The Code

proc get_acl { acl } {


set command "show running-config | include ^access-list $acl"
return [exec $command]
}

Extract the Desired Access-List


from the Running Configuration

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 77
Tcl Access-List Editor
The Code (Cont.)

proc paginate { l } { Display the ACL in


set i 0 24 Line Pages
while { $i < [llength $l] } {
set new_page 1
for { } { $i < [llength $l] && ($i == 0 || ($i % 24 != 0)
|| $new_page == 1) } { incr i } {
set new_page 0
set num [expr {$i+1}]
set lentry [lindex $l $i]
puts "$num: $lentry"
}
if { $i < [llength $l] } {
puts -nonewline "Hit enter to continue..."
flush stdout
gets stdin key
incr i
}
}

return $i
}
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 78
Tcl Access-List Editor
The Code (Cont.)

proc add_acl { acl acllist } {


puts "\n"
set i [paginate $acllist]
set aclend [expr {$i + 1}]

puts -nonewline "Insert before which line number ($aclend to append): "
flush stdout
gets stdin choice

if { $choice <= 0 || $choice > $aclend } {


puts "Invalid line number, $choice.\n"
return $acllist
}

puts -nonewline "Enter body of ACL rule to insert (without the access-list
$acl portion): "
flush stdout Insert a New ACL Entry
gets stdin body
Within the Existing ACL
regsub -nocase {^access-list\s[^\s]+\s} $body "" body

return [linsert $acllist [expr {$choice - 1}] "access-list $acl $body"]


}
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 79
Tcl Access-List Editor
The Code (Cont.)

proc delete_acl { acl acllist } {


puts "\n"
set i [paginate $acllist]

puts -nonewline "Enter ACL entry number to delete: "


flush stdout
gets stdin lineno

if { $lineno <= 0 || $lineno > $i } {


puts "Invalid entry number, $lineno.\n"
return $acllist
}

return [lreplace $acllist [expr {$lineno - 1}] [expr {$lineno - 1}]]


}

Delete a Specific
ACL Entry
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 80
Tcl Access-List Editor
The Code (Cont.)
proc commit_acl { acl acllist orig_acllist } { Check for Errors to Ensure We Do
ios_config "no access-list $acl"
Not Leave the Router Unprotected
foreach line $acllist {
if { [catch { ios_config $line } result] } {
puts "Error committing access-list entry \"$line\" ($result)"
puts "Re-adding the original access-list..."
ios_config "no access-list $acl"
foreach origline $orig_acllist {
if { [catch { ios_config $origline } result] } {
puts "DANGER! Error committing original
access-list entry \"$origline\" ($result)"
puts "Investigate this immediately!"
return
}
}
return
}
}

puts "Access-list $acl was committed successfully."


}

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 81
Tcl Access-List Editor
The Code (Cont.)
while { $done == 0 } {
puts "Access-list Editor"
puts "------------------\n" Present the Access-List; Edit
puts "1. View access-list $aclno" Options to the User in a Menu
puts "2. Add access-list entry"
puts "3. Remove access-list entry"
puts "4. Quit and save changes"
puts "5. Quit without saving changes"
puts ""
puts -nonewline "Enter option: ";
flush stdout

gets stdin choice

switch $choice {
1 { view_acl $acllist }
2 { set acllist [add_acl $aclno $acllist] }
3 { set acllist [delete_acl $aclno $acllist] }
4 {
set done 1
set save 1
}
5 {
set done 1
NMS-3301 set save 0
BRKNMS-3021
11152_05_2005_c2
13717_05_2007_c1 2005
2007 Cisco Systems, Cisco
Inc. Systems,
All rights Inc. All rights
reserved. Ciscoreserved.
Confidential 82
82
Tcl SNMP Security Fix Script
The Code
Determine the High UDP Port Dynamically by
Inspecting the Output of Show ip Sockets

proc snmp_fix { } {
snmp_unfix
set sockets [exec "show ip sockets"]
set socket 0
foreach line [split $sockets "\n"] {
set line [string trim $line]
if {[regexp {^17\s+--listen--} $line] || [regexp {^17 0\.0\.0\.0}
$line]} {

set tsocket [lindex $line 3]


if {$tsocket > 49152 && $tsocket < 65535} {
set socket $tsocket
break
}
}
}

BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 83
Tcl SNMP Security Fix Script
The Code (Cont.)
if {$socket > 0} {
set myacl [find_acl 100 200]
if {$myacl == 0} { Find a free IP ACL, then
set myacl [find_acl 2000 2700]
apply the necessary
}
control plane policing
if {$myacl == 0} {
puts "Failed to find a free access-list." configuration changes
return
}
ios_config "class-map match-all matchsnmp" "match access-group $myacl"
ios_config "policy-map dropsnmp" "class matchsnmp" "drop"
ios_config "access-list $myacl permit udp any any eq 162"
ios_config "access-list $myacl permit udp any any eq $socket"
ios_config "access-list $myacl deny ip any any"
ios_config "control-plane" "service-policy input dropsnmp"

puts "SNMP control plane access now denied to ports 162 and $socket"
puts "using access-list $myacl. Use ``snmp_unfix'' to remove this"
puts "configuration."
} else {
puts "Failed to find a listening socket for SNMP."
}

}
BRKNMS-3021
13717_05_2007_c1 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 84

S-ar putea să vă placă și