Redbooks Paper

Steve Strutt

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management
Introduction
This IBM Redpaper outlines how it is possible to integrate management and reporting in IBM Tivoli Storage Manager (TSM) environments into an enterprise-wide IT monitoring and management solution. It shows how IBM Tivoli Storage Manager Operational Reporting can be used to simplify the monitoring and reporting on the status of a Tivoli Storage Manager environment and how this information can be displayed centrally on an enterprise event management console. IBM Tivoli Storage Manager Operational Reporting provides a significant step forward in monitoring and reporting on the current state of one or more TSM servers. Summary reports and automatic notification of issues simplify administration and reduce the time required to manage TSM environments by enabling a management-by-exception approach to be adopted. In environments where enterprise monitoring and event consoles are used, these Operational Reporting status events can be used to simplify the management of TSM environments and integrate them into the wider picture of management of the whole IT infrastructure. Accompanying configuration files are provided to implement the features described in this paper. These can be downloaded from the following URL:
ftp://www.redbooks.ibm.com/redbooks/REDP3850/

Alternatively, you can go to the IBM Redbooks Web site at:

http://ibm.com/redbooks

Select Additional materials and open the directory that corresponds with the Redpaper form number, REDP3850.

Copyright IBM Corp. 2004. All rights reserved.

ibm.com/redbooks

Event management and operational reporting

The TSM Operational Reporting management capabilities are provided by reports on the Microsoft Windows-based reporting server, by Web pages, and also by sending alerts using e-mail and the Windows Net Send capability. However the reports are delivered, they are used to notify storage administrators that their TSM environment is either running smoothly or needs attention. This provides good management for either (1) standalone TSM environments or (2) applications in which there is no requirement to manage TSM in conjunction with other pieces of the IT infrastructure. To simplify and minimize the administration of IT infrastructure, many environments use central event management consoles to monitor and manage the whole IT environment. This practice gives visibility of the availability and status of all components of the IT infrastructure from a single point, along with visibility of the impact on other dependant parts of the infrastructure. Use of an event console allows a single consistent approach to alert and event management to be used irrespective of the application, hardware, or vendor. This approach simplifies administration and reduces costs. It also opens up the possibilities of using other alerting mechanisms such as pagers, SMS text messages, and voice response systems to alert administrators to issues out of hours. In environments where these event and management processes already exist, it is desirable to integrate TSM events and Operational Reporting events into these processes to provide a single point of management and further reduce the effort required to administer TSM. Using an event management approach enables TSM to be managed and monitored in conjunction with its supporting hardware environment. Figure 2 on page 4 shows how TSM and Operational Reporting can be integrated into an event management environment. In addition to event management, it shows how the Web-based management features of TSM and Operational Reporting can be used to provide remote management of a TSM environment. On receipt of an event, a Web browser can be launched to access the related TSM server or Operational Reporting report or monitor to provide more detail about the current operational status and address any issues.

TSM W e b a d m in
Launch w eb a d m in

E v e n t C o n s o le
Laun ch w eb r e p o rt

T SM O R W e b s e rv e r

TSM O R C o n s o le

E v e n ts

T S M O R S e rv e r T S M S erver H a rdw a re

Figure 1 TSM and Operational Reporting integrated into event management

TSM Operational Reporting intelligently monitors the results of TSM operations, schedules, and status. It can then alert exception conditions to the event console. The alert thresholds can also be set to meaningful values for specific user environments, such as the known minimum number of scratch tapes to complete the overnight backup schedules. As a result, the number of events that must be forwarded from a TSM server to monitor its activities can 2
Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

be reduced significantly because Operational Reporting can more precisely summarize the results. TSM is an extremely verbose product and an out-of-the-box implementation of TSM results in many messages being sent to the event console. With Operational Reporting monitoring schedule results and server status, the number of TSM messages forwarded to the event console can be restricted to those critical events that indicate severe problems and those issues that must be dealt with immediately With the 5.2.2 release of TSM for Windows, Operational Reporting provides an event log that can be exploited to forward status events to an IBM Tivoli Enterprise Console (TEC) or other SNMP Event Managers. This paper shows how this event log can be used to forward TSM Operational Reporting status events on to TEC using a TEC logfile adapter and how the Web versions of the reports can be launched and viewed from TEC. This approach to central monitoring of TSM servers could also be implemented for SNMP-based event management solutions that provide adapters to read log files.

Reporting operational status

Within the Custom Summary section of each report exists the capability to define Notification Rules that determine which aspects of TSMs operations to monitor and report on. These rules define which out-of-line situations are included in the Issues and Recommendations section of each report. When items are included in the Issues and Recommendations section, the status of a report or monitor becomes needs attention, otherwise it is running smoothly. This status information is also used to determine the classification of e-mail and Net Send alerts sent out into needs attention or running smoothly categories. This status information is also used to drive the events that Operation Reporting writes to the event log file and that are picked up and passed to TEC. Each Notification Rule in a report or monitor can generate an issue and is written out as a separate event log entry. In this paper these are termed as issue events. When no out-of-line situations are detected and the report or monitor status is running smoothly, the event is classed as a success event.

The Tivoli logfile adapter

The logfile adapter for Windows platforms is the Windows Event Log Adapter. It gathers events from the six Windows event logs (System, Application, Security, DNS Server, File Replication service, and Directory service) and also any other ASCII log files residing on the Windows server. It reads any messages in these logs and forwards them via TCP/IP to a TEC Event Server for further processing. More details about the adapter and its configuration can be found in the manual, IBM Tivoli Enterprise Console, Adapters Guide, Version 3.8 GC32-0668. Figure 2 on page 4 shows the architecture of the logfile adapter and the constituent components making up the Operational Reporting event management integration. Operational Reporting writes success and issue events to a log file, from which the logfile adapter checks for new events on a periodic basis. New events are formatted as TEC events and sent on to a TEC server for display on a TEC console.

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

TSM Operational Reporting Server

Config file Class definitions

TEC Server

Write Message
TSM_Monitor_Warning version "1.0" timestamp "06/13/2003 08:59:04" file "SSTRUTTServer1MonHourlyMonitor20030613 085904.htm" type "1" typename "Monitor" computer "SSTRUTT" instance "Server1" serverurl "http://SSTRUTT:1580" report "Hourly Monitor" begin "2003-06-13 07:33" end "200306-13 08:33" status "1" statusname "Needs attention" "message There are not enough scratch volumes available. condition 0 < 5 recommendation Check in some scratch tapes."

Check for new messages added to file Read Message

Read and tokenize message

Formatted TEC event

tecinfo.txt Log file adapter

class file (baroc) rules file

Figure 2 Logfile adapter architecture

Installation and configuration

The following section describes the installation and configuration of the logfile adapter and configuration of TEC. To configure TEC alerting, the Tivoli Logfile adapter for Windows 2000 must be installed on the Windows Server where Operational Reporting is installed. This is not supplied with TSM and can be found on the TEC installation CDs. This should be installed as per the Tivoli installation instructions for a Tivoli-managed host (if a Tivoli endpoint is installed) or as a non-TME managed host. The instructions included here assume that the Windows Server running Operational Reporting does not have a Tivoli endpoint installed and that the non-TME version of the adapter is installed. The accompanying sample configuration files are for the TEC 3.8 version of the logfile adapter and assume a TEC 3.8 console.

Installation of the non-TME Windows Logfile adapter

You must install the TEC logfile adapter from a local directory on the server where the adapter will be installed. Create a local installation directory, and copy the Windows version of the non-TME logfile adapter from the w32-ix86\installwin directory on the TEC non-TME CD to this directory. To install the Windows logfile adapter, run setup.exe from this installation directory. The installation process then takes you through installing and configuring the logfile adapter. Select the installation directory in the first dialog of the installation process. The default is C:\tecwin, as shown in Figure 3 on page 5.

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

Figure 3 Destination location of logfile adapter

Enter the fully qualified host name of the computer where the Tivoli Enterprise Console resides on the Server Configuration panel, as shown in Figure 4.

Figure 4 Server Configuration dialog for the logfile adapter

Enter the port name used by Tivoli Enterprise Console to receive events in the Server Port panel. This is usually 5529 for a TEC installed on a Windows server, but it should be the value used in your environment. This is shown in Figure 5 on page 6.

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

Figure 5 Server port dialog for the logfile adapter

When this is complete, the logfile adapter is installed. However, it must be configured to forward events from Operational Reporting to TEC.

Configuring Operational Reporting to create events

The success and issue events generated by Operational Reporting are written to an ASCII file, tecinfo.txt, in the console\tec directory under the TSM installation directory. Other event managers that have adapters to read log files could also use this file to collect Operational Reporting events and send them to the event console. The full path and file name is:
C:\Program Files\Tivoli\tsm\console\tec\tecinfo.txt

This file is created when Operational Reporting is configured to create TEC events, and all subsequent events are written into this file. The writing of events to this file is enabled in the Operational Reporting properties panels. To enable this process launch the (Windows) TSM Management Console and select the Tivoli Storage Manager menu item in the left-hand pane, as shown in Figure 6 on page 7. Right click the menu item to configure Operational Reporting.

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

Figure 6 TSM Management Console

Writing to the TEC log file is enabled on the Summary Information tab of the Properties panel. Check the Create a Tivoli Enterprise Console (TEC) log file box and click OK, as shown in Figure 7.

Figure 7 Operational Reporting Properties panel

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

Configuring the logfile adapter

By default, the TEC logfile adapter only monitors the Windows Event Log. It needs to be configured to monitor the ASCII log file created by Operational Reporting. The adapter configuration is held in the file c:\tecwin\etc\tecad_win.conf file. This file must be edited to specify the location of the tecinfo.txt file and the frequency to scan the log file. In this example, it is assumed that monitoring and reporting of events in the NT Event Log is not required and is disabled. Add the following lines to the end of tecad_win.conf file. The LogSources variable will need to be edited to reflect the location where Operational Reporting is installed as in Example 1.
Example 1 Entries for tecad_win.conf file LogSources=c:/Program Files/Tivoli/TSM/console/tec/tecinfo.txt PollInterval=30 WINEVENTLOGS=none

Two files are required, tecad_win.fmt and tecad_win.cds, for the logfile adapter to recognize the Operational Reporting events and format them correctly as TEC events. The format of the messages that the adapter looks for is defined in the .fmt file. The sample configuration files provided as additional materials to this Redpaper include .fmt and .cds files. The four Operational Reporting events can be found at the end of the sample .fmt file. The .cds file is generated from the .fmt file and read by the logfile adapter. A pre-generated .cds file is provided. The tecad_win.cds file should be copied to the c:\tecwin\etc directory. The logfile adapter uses the locale of the Windows Server. The sample .cds and .fmt files assume a locale of English. If this is the case, the sample tecad_win.fmt file should be copied to the c:\tecwin\etc\c directory. If the locale is set differently and monitoring of the Windows Event log is needed in addition to the tecinfo.txt file, the four Operational Reporting message format statements in the sample fmt file should be copied to the correct locale version of the tecad_win.fmt file and a new .cds file generated. The logfile adapter installs as a Windows service. It should be stopped and restarted after updating the configuration files, as in Example 2.
Example 2 Commands to stop and restart TEC Logfile Adapter net stop TECWinAdapter net start TECWinAdapter

Configuring TEC to display Operational Reporting events

The event class definitions for the events in the tsmor_msg.baroc class file found in the accompanying sample files need to be loaded into the TEC DB to enable the events to be formatted and displayed on the TEC server. Details of how to load a class file can be found in the manual, IBM Tivoli Storage Manager for Windows: Administrators Guide, GC32-0782, in Chapter 20, Monitoring the IBM Tivoli Storage Manager Server, under the heading Logging IBM Tivoli Storage Manager Events to Receivers. A sample set of TEC rules are also provided in the file tsmor.rls to remove duplicate events and unnecessary historical events. Operational Reporting status events are issued hourly for monitors or daily for reports. Over the period of a day, this would mean 24 events for just one hourly monitor. If the status remains the same, this will result in a large number of duplicate messages for each report or monitor every day. If, for example, the status changes from running smoothly to needs attention, new issue events are created, but the old running smoothly events will still be displayed. Two rules are supplied to eliminate these duplicate 8
Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

messages and to automatically close preceding success or issue events if the status of a report or monitor changes. This leaves just one success event or a small number of issue events for each report or monitor on the TEC view, which significantly reduces the number of events that must be monitored. As a result, (1) the TEC event views show only that information that is both current and relevant, and (2) the views are automatically updated with the latest status information. This simplifies management when multiple TSM servers are monitored because the number of events to be reviewed is relatively small. Rules files are loaded in the same fashion as outlined for class files, with the rules file being imported instead of a class file. After the class and rules files have been loaded and the TEC server stopped and restarted, it should display the Operational Reporting events in the all events view.

Using the Tivoli Enterprise Console

The TEC console provides a flexible environment to manage events from multiple products across the whole IT infrastructure from hardware to business applications. To meet varying operational requirements, it can be configured to show just the events and views that operators require to manage all or part of the IT infrastructure. From a storage perspective, it can be configured to show only those events that are related to the storage infrastructure.

TEC Event Groups and Views

TEC Event Groups enable only selected event classes to be displayed in a specific view, removing events that would otherwise make analysis of the received events more difficult. Figure 8 shows a TEC console configured to show only storage events.

Figure 8 TEC Summary Chart View

Event Groups have been used to separate out events into the different storage management disciplines. These include the SAN fabric that TSM is potentially dependant on, along with the storage hardware device status for tape libraries and disks. An Event Group has been

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

created for Operational Reporting events. This group makes locating these events easier and enables you to see the status of multiple TSM servers at a glance from this event view. Figure 9 shows the Operational Reporting TEC Event View with events showing the status of multiple monitors and reports for two TSM servers, DUBLIN and Azure. The TSM server name can be seen in the Hostname field in the top pane of the window.

Figure 9 Operational Reporting TEC View

TEC Event information

The following TEC event classes are used to classify the success and issue events: Logfile_TSMMON_Success: OK status for monitors Logfile_TSMMON_Warning: One event of each issue identified by the monitor. Logfile_TSMREP_Success-: OK status for reports Logfile_TSMREP_Warning-: One event of each issue identified by the report. For the issue events, the issue identified is passed, along with the Notification Rule or condition that triggered it and the associated recommendation. Figure 9 shows examples of all the Operational Reporting events on TEC. The Daily Report has identified two issues. As a result, there are two events with a severity of warning, one for each issue. The message column shows the event type, the TSM Server name, the report or monitor name, and the issue. A number of the reports and monitors are showing that everything is running OK and these have a severity of harmless. The message column shows the event type, the TSM Server name, the report or monitor name, and the words running smoothly. In an environment where multiple TSM servers are managed, these harmless events can be suppressed from being displayed to reduce the number of events that must be scanned to locate issue events. The event view then only shows events when there are issues that must be addressed.

10

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

Using Operational Reporting events with TSM

TSM intentionally writes out many messages because it has to log all operations and events that occur in order to aid the management and resolution of problems. However, this is not particularly helpful when an event console is being used to monitor TSM activities, due to the large number of messages created. Therefore, a set of message defaults are provided in the ibmtsm.mac file in the TSM server directory to filter and limit the number of events that are sent to TEC. In many environments, this default filtered set of events, however, is not enough. There can still be too many messages to easily identify what is happening on a TSM server and monitor the current status. Fortunately, Operational Reporting status events that alert you to out-of-line situations can be used to significantly reduce the number of TSM messages that must be sent to TEC to monitor operations. The default set of messages to be sent to a TEC console, defined in the file ibmtsm.mac in the TSM server directory, includes all server messages relating to the scheduling of client operations. Thus, the default results in many scheduling messages in the TSM Activity Log. Alternatively, monitors and reports can be configured to check the status of client schedules on a regular basis. Specific monitors can also be created to look for specific critical schedules. If these monitors are in place, Operational Reporting status events can be used to check schedule status. Then, the TSM scheduling messages can be filtered out and not sent to TEC. Using Operational Reporting for status events in this fashion avoids the need to actively monitor the TSM messages coming into TEC. The filtered set of messages could be reduced to just the Critical and Fatal messages that indicate significant problems with TSM and need to be dealt with immediately to avoid a significant impact on TSM operations. Warning and informational messages can be summarized by Operational Reporting into more concise status updates, with the Operational Reporting views being monitored for these status events. Daily Reports created by Operational Reporting can then be used to review all messages on a daily basis, if desired. Figure 8 on page 9 shows just this configuration. The TEC Summary View only shows a small number of critical TSM server messages and a few Operational Reporting status events. One of the customization features of TEC is the ability to define custom buttons. These buttons can be used to launch other applications. If Operational Reporting is configured to create Web pages and a Web server used to host the HTML versions of the reports and monitors, this TEC custom button can be used to launch a Web browser pointing at the specific monitor or report. Figure 1 on page 2 shows this capability being exploited in a remote management environment. On receipt of an Operational Reporting status event, the related Operational Reporting HTML version of a monitor or report can be launched in a Web browser. This gives the administrator at the TEC console full access to all of the information in the monitors or reports. The related TSM administrator Web interface also can be launched directly from TEC or from the monitor or report. Figure 10 on page 12 shows a custom button, Launch Storage, configured on the TEC console to launch the Web browser interfaces of the IBM storage management products, which includes TSM, Operational Reporting, IBM Tivoli Storage Resource Manager (ITSRM) and IBM Tivoli SAN Manager (ITSANM). Figure 10 on page 12 shows the TEC console properties configuration dialog. Up to three custom buttons can be created from here. The figure shows the Launch Storage button configured to execute the custombutton.cmd script. This script is supplied in the accompanying sample files, but it may need customizing to the specific environment to point to the Web pages configured for each IBM storage product.

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

11

Figure 10 TEC console properties showing custom button configuration

Summary
TSM Operational Reporting provides the ability to significantly reduce the administrative effort to manage a TSM environment by summarizing the results of operations and alerting users to specific out-of-line situations. When integrated with centralized event management, it enables IT administrators to manage their whole environment by exception and only get involved when the situation warrants it. It also simplifies the management of multiple TSM servers by bringing all status reports of all servers to a single point and presenting it on one screen. Exploiting the custom button feature of TEC and TSMs Web-based administration enables a TEC event management console to become a central focal point for a storage administration console. This focal point is a place where all relevant management interfaces can be launched and products can be administered to address any identified issues.

The team that wrote this Redpaper

Steve Strutt is a Certified IT Specialist in the U.K., working in Technical Sales Support. He holds a bachelor's degree in Electronic Engineering and has more than 18 years of experience in IBM storage and storage management tools from z/OS to open systems storage. His areas of expertise include enterprise storage architecture, tape storage systems, Tivoli Storage Manager, Storage Area Networking, and Storage Resource Management and systems management. Thanks to the following people for their invaluable contributions to this Redpaper: Charlotte Brooks International Technical Support Organization, San Jose Center Andy Robinson Software Group, U.K.

12

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

Stuart Swain International Technical Support Organization, San Jose Center

Become a published author

Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at:
http://www.ibm.com/redbooks/residencies.html

Comments welcome
Your comments are important to us. Send us your comments in one of the following ways (be sure to include the document number, ): Use the online review redbook found at:
http://www.ibm.com/redbooks

Send your comments in an Internet note to:

mailto://redbook@us.ibm.com

Mail your comments to: IBM Corporation, International Technical Support Organization Dept. QXXE Building 80-E2 650 Harry Road San Jose, California 95120-6099 U.S.A.

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

13

14

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management

Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

Copyright IBM Corp. 2004. All rights reserved.

15

Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:
Eserver Eserver Redbooks (logo) Eserver ibm.com z/OS IBM Redbooks Tivoli Enterprise Tivoli Enterprise Console Tivoli TME

The following terms are trademarks of other companies: Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others.

16

Integrating IBM Tivoli Storage Manager Operational Reporting with Event Management