Whether business is conducted locally or globally, companies depend on their technology to keep them up and running. End users expect 100% availability, rapid and consistent response times and no excuses. The systems that support e-business, airline reservations, utility services and retail point-of-sale are but a few examples of critical systems whose users require immediate response – and downtime can spell disaster.

The availability of these highly visible systems relies on the proper management of hundreds or thousands of individual servers, applications, databases, middle-ware and Web technologies. Failures in these environments must be detected quickly and efficiently, with accurate details being conveyed to the proper personnel. According to analyst studies, as much as 80% of system downtime is spent identifying a problem prior to even formulating a solution. When customers are on the line, that's time companies can't afford. Maximizing availability is key to maintaining a competitive position in the marketplace.

Why is event management and notification important?

Management approaches vary from company to company, but in general they include three basic components: 1) tools to detect failures, 2) personnel to handle recovery and 3) the procedures that outline how a failure should be managed. These three components must be synchronized, collaborating with each other to achieve the ultimate goal – service and application availability.

Failure detection is the starting point that ties together the other two components. If the failure can't be detected, neither personnel nor company procedure can fix it. However, failure detection alone does not reduce downtime.

Automatic recovery features can help to some extent, but even the most sophisticated recovery tools cannot automatically handle all recoveries. In cases where automatic recovery is not possible, IT personnel must take manual action to resolve the problem and maintain availability. A challenge many organizations face in these instances is managing the overwhelming number of events detected by their monitoring tools. To overcome this challenge, the failure management process must include the right event management component.

An effective event management system:

• Helps personnel (operators) deal with the problems that require human intervention by assisting in the identification and resolution of problems before they adversely affect end users
• Allows operators to concentrate on those events that need their immediate attention
• Notifies the appropriate IT staff quickly and efficiently when problems arise

Without an event management system in place, any IT environment may eventually become too complex and unstable to manage, resulting in availability issues, end-user dissatisfaction and high support costs. Companies need all of their events managed simply and efficiently, in a manner that optimizes their IT staff's ability to react quickly and correctly to problems as they arise. A company's large investments in technology mean nothing to its end users unless the services are available when they need them.

What should an event management system do?

The objective of an event management system is to ease the task of managing numerous events. This makes the operator's job easier and more manageable.

An event management system must provide enough information to allow operators to act quickly as the failure occurs, while reducing the need for expert involvement. At the same time, the event management system must be very specific when presenting information to operators to avoid overwhelming them with too many events or with unimportant information.

An event management system is dependent on the ability of the monitoring tool to detect events, allowing the system to then translate those events into information that operators and management personnel can use to make appropriate decisions. In general, an event management system must provide:

• Flexible tools to reduce the overwhelming number of events presented to operators
• Powerful and scaleable graphical interface to present the right information to the right people at the right time
• External notification methods (page, email, phone) to immediately involve the technical support group, the management hierarchy and/or proactively inform customers of an impending problem
• Tools like correlation of events to reduce failure isolation time and relieve the burden on technical experts
• Helpful event enrichment to quickly identify the impact of a failure on the business (services or customers)

The right event management component will help companies move away from the reactive failure management approach and switch to a proactive one that will maximize customer satisfaction and loyalty.

BMC Software's event management initiatives

PATROL EM easily accommodates networks, telecommunication devices, mainframes, PABX and any other technology used to run the business. It gathers the information from the individual management tools (also referred to as element management systems or EMS), providing the event management functionality at the enterprise level.

In the case of distributed systems, customers need an EMS as well. BMC Software meets this need with a new, enhanced event management and notification component built into the basic PATROL product. This new component helps operators deal with those failures that cannot be recovered automatically and that require human intervention. This is done by:

• Providing a powerful operator-oriented graphical interface
• Implementing a middle-tier server for event management and notification

With this enhancement, BMC Software brings all the benefits derived from event management into PATROL-only environments, providing a domain-level management or EMS bundled right into PATROL.

This new event management component complements BMC Software's strategy on application management. Now with PATROL, customers receive all the tools they need to immediately detect failures, automate recovery actions and effectively manage events in the distributed system environment. In summary, they receive a complete solution to manage the two components of the availability equation – failure detection and event management.

New PATROL architecture

There are at least four major improvements with the introduction of this new event management component in PATROL:

• Effective event filtering, de-duplication and automatic closing
• Powerful and scaleable operator-oriented console
• Flexible external notification methods (page, email, phone)
• Quick problem isolation through correlation of events

From the architectural point of view there are two new components: a middle-tier server for event management, and an operator-oriented GUI to visually identify the failures (Figure 1). These changes make PATROL a three-tier architecture product, maintaining the advantages of direct communication between the PATROL Console and the PATROL Agent.

The PATROL OM server allows total customization of the events coming from the PATROL KM. Customers can change event attributes, such as severity or message text, and can add information about the event's impact on the business, such as services or customers affected. This event customization is defined at the server without changing the PATROL KM, enabling customers to tailor the operations to their specific IT and business needs. The attribute changes happen "on the fly" once the PATROL OM server receives the event from the PATROL KM. The translated events are then stored in an Oracle database, which houses a wealth of historical data.

The heart of PATROL OM is the rule engine. It is responsible for processing the events according to user specifications. Users use rules to identify what events to process and what actions to execute.

Once the event is defined, users specify what will happen when the event arrives. Many actions can take place. An alert can be created, a notification process started or even a condition created to be used in correlation. All these actions occur at the PATROL OM server, where the user can redefine the message (severity, message text, etc.) to make the event more understandable (Figure 4).

It can also execute multiple actions once the event is recognized. Figure 5 shows how to create an alert and send a page when PATROL detects a problem with a local disk on Microsoft Windows NT.

The paging notification itself is very powerful. Pages can be sent using TAP or SMS protocol. Additionally, an easy-to-use scripting mechanism is available to customize the notification to individual protocols. Pages can be sent to individuals or groups, and the acknowledgment can be done over the phone. If the acknowledgment is not received, an escalation procedure is started that involves other persons or groups. This entire process is centralized from the PATROL OM server.

Benefits of PATROL's new event management and notification component

The introduction of the PATROL OM as the component for event management and notification is a big leap forward in the PATROL evolution. It brings many benefits, all of them targeted to reduce downtime and increase service and application availability. The PATROL OM server component:

• Increases efficiency by eliminating the event overflow. PATROL now provides event de-duplication, automatic closure and advanced event filtering that helps the operator locate the failure quickly. The operator sees only the current failures related to his area of responsibility.
• Informs IT and non-IT management, providing the right information at the right time. This also empowers first-level operators, giving them a broader spectrum of control. Using PATROL Explorer, users can create tailored maps for a simplified view of problems for both operators and management. Management levels have access to maps with services and customers affected, and operators have access to maps with all the information they require for supporting rapid action.
• Reduces time to resolution by rapidly notifying the identified hierarchy of management/operators as soon as the failure occurs. PATROL now uses emails, paging or phone calls to contact operators once an event is defined for notification. It can reach a single user or a group by using paging, escalation, phone acknowledgment and notification based on calendars. The notification process is centralized and can be initiated using multiple conditions from different servers or applications.
• Reduces failure isolation time and unburdens technical experts. PATROL now uses correlation to help operators quickly comprehend the situation, and diagnose and resolve the problem appropriately. This reduces the number of times the technical expert is required to intervene.
• Increases the ability to understand the failure's impact on the business (such as services and customers affected). The new event management component allows management to quickly relate the PATROL event to the services or customers. The translated event in PATROL is automatically enriched with the user-defined information, making it easier to identify what customers or services went down when a specific PATROL event is received.
• Integrates with PATROL EM, providing a simple and natural step up as the enterprise grows (no migration or upgrading is required, simply "plug in" ). Customers can start managing their distributed system environment with PATROL, implementing filtering, notification and correlation at the domain level. Then, when it's necessary to manage other technologies beyond distributed systems, PATROL EM can be implemented to manage the whole enterprise. PATROL OM simply connects to, and feeds the distributed system information to, PATROL EM (Figure 7).
• Minimizes installation and maintenance time. Out-of-the box, PATROL offers predefined rules to begin receiving and translating events immediately after installation. It discovers PATROL Agents at installation to minimize set-up time (approximately one hour). Additionally, no modifications or additional software is required at the agent.
• Eliminates the need to retrain IT staff or change the GUI, since PATROL OM and the PATROL EM server use the same PATROL Explorer as the operational console. Likewise, there is no need to upgrade or migrate from the domain level.

Summary

An effective event management and notification system is key to maximizing business-critical application availability. By offering a powerfully robust solution, PATROL assures this availability, combining failure detection, automatic recovery and robust event management and notification in a single product.

BMC Software's PATROL solution for event management and notification addresses the IT professional's needs with event filtering, notification and tailored views of the enterprise. This eases the normally labor-intensive task of identifying and reacting to system failures, thus increasing IT staff productivity.