General Donald Walker oversees a seven-processor mainframe environment (4400 MIPS' worth), a handful of midrange servers (320) and a few Windows NT desktops (25,000 or so). Walker is CIO of United Services Automobile Association (USAA) in San Antonio, Texas and occasionally the sheer size of the insurance and financial services organisation creates headaches. For example, take software updates.
Some time ago, USAA planned a software rollout that went smoothly "until we ran into instability at about the 7000-desktop mark", Walker says. Problems cropped up that hadn't appeared in the test lab, where roughly 500 PCs were running test scripts. All the individual components worked, but they didn't work well together at the necessary enterprise scale. Developers, contractors and operations workers wound up in crisis mode, scrambling to fix the problem for a very peeved business-side client.
Sound familiar? Any CIO with a client/server set-up knows that software distribution is just one of many systems management hassles. Enter ESM, enterprise systems management, the science -- or art -- of managing distributed computing in big companies.
USAA bought IBM's Tivoli Enterprise framework as the software foundation for an ESM project. Like all ESM efforts, USAA's project incorporated lots of systems and network management functions, but it is Tivoli's automated software distribution capability in particular that best illustrates the potential of ESM. Now Walker can push an 80MB load of software out to 7500 desktops in about five hours. Fast distribution means Walker's group can make a few bulk changes instead of many incremental ones. That in turn means less integration testing, more time to do it, less potential for error and lower testing costs. By creating a centralised, real-time mechanism for collecting systems feedback, "Tivoli also helps us anticipate problems instead of recovering well from the train wreck," says Walker.
Fast, smooth software distribution is just one example of ESM's promise. ESM vendors make a raft of others: Better uptime, higher application performance, more efficient storage and security, increased visibility into problems.
However, along with the promises comes ESM's bad rep. Some early attempts at ESM framework implementations hit more than the usual share of IT project snags because both the management software and the client/server environment were new.
That scared off a few folks. Instead of taking the risk, they preferred to stick with solutions that handle various administrative tasks like security, backup and performance management but don't exchange information and thus provide no overall view of what's going on enterprisewide.
But that was then and this is now. "We've certainly found significant customers who have gotten excellent return on their investment," says Paul Mason, vice president of infrastructure software research at IDC (US). "I suspect the problems [in ESM implementations] are institutional as much as anything else." In other words, there's nothing wrong with the frameworks, but companies buying and using them have to handle the implementation correctly and must be prepared for corporate changes in order to make the software pay off. If we add disciplined project management, process change and organisational remodelling, ESM can help keep the networks humming, the software up to date and the security sound.
Phase It In
Before implementing a framework, CIOs must define which specific management functions the project will address. The basic architecture of an ESM suite is this: The framework provides a central repository of information about the devices attached to the enterprise network; a set of software agents resides on those distributed devices and keeps the repository up to date on any changes in the status of the devices; an interface graphically displays that status.
Software modules that plug into the central repository and make use of its information handle various management functions. ESM suites offer a long list of potential modules and functions, but not every piece is necessary for every company.
Veterans universally recommend breaking the implementation process into phases.
Done correctly, ESM rollouts require a lot of time -- a year or more is quite appropriate. Trying to roll out the whole convoluted kit and caboodle in one heroic effort is no more likely to work with ESM than it is with enterprise resource planning (ERP) applications-that's what gave both of them a black eye in the first place.
PrimeCo Personal Communications LP, a PCS-wireless communications network provider based in Westlake, Texas, is going about its implementation the right way. With 3500 PCs, 180 servers and 2 IBM SP2 parallel processors to manage, PrimeCo started its ESM effort with the Tivoli Enterprise framework in July 1998. The first phase of the implementation included the security configuration and software distribution management modules, according to vice president for IT and CIO Jim Myckleby. Fault management -- tracking and fixing network and systems problems, which is typically regarded as the central function of ESM -- was relegated to phase two, scheduled for completion in July. Production management (including issues such as scheduling of processing jobs) is also a part of phase two. Any further functionality will happen in a third phase.
Expect to Integrate
Even though ESM suites cover a lot of ground, they do not cover all the ground.
Third-party applications frequently plug into the framework to handle niche functions.
"If you go in with the mentality that the framework is going to manage everything, you're going to fail," says Ken Klindworth, vice president and CIO of Graybar Electric, a $US3.7 billion wholesale distributor of electrical and communications equipment based in Clayton, Missouri. Graybar dove into ESM about five years ago.
The company uses the OpenMaster framework from Bull Worldwide Information Systems' BullSoft division to manage 2550 devices (including printers and other peripherals) attached to 300 local area networks. While OpenMaster is the overarching system, Graybar connects software from several other vendors to handle specific functions. The Graybar network's 3Com hubs and switches, for example, are monitored by 3Com's Transcend Management software; Nortel Networks's Optivity management suite keeps track of the routers and ATM devices. Graybar also uses BMC Software's Patrol management software to keep tabs on its BEA Systems Tuxedo transaction monitoring system. Some vendors' suites are more tightly integrated than others. A highly integrated suite provides smooth connection of one module to another and can make network problem-solving easier. The trade-off is that it can be more difficult to integrate third-party applications. It's up to the CIO to find the framework with the right balance of integration and openness. "For us, OpenMaster has a high degree of integration where we need it, but it isn't so integrated that you can't add things to it," says Ted Hoffman, director of IS for Graybar.
Nail the ROI
It's an old IS conundrum: how do you quantify the dollar return on a process that adds no new capabilities but facilitates other processes? Most ESM veterans say that although they do not have hard payback figures, they are convinced that the frameworks ultimately save their companies money.
ESM's return on investment can come in two forms. One form is enhanced systems availability. For example, one CIO who requested anonymity reportedly increased his uptime from 95 per cent to over 99 per cent and multiplied that increase by a "cost of lost sales due to an inability to book orders" figure supplied by the marketing department to determine his ROI. Others confirm this kind of return.
"I don't have figures on the ROI; I can't prove the payback," says Luc Verhelst, the Flemish CIO of Kind en Gezin, a Belgian government child-care agency in Brussels. "But when your mail system goes down for one hour, that costs a lot of money." Kind en Gezin's ProVision ESM system from Platinum Technology (recently acquired by Computer Associates International) gives Verhelst an early warning of those kinds of problems and helps resolve issues before a server crash. Verhelst estimates that the total cost of ownership for maintaining Kind en Gezin's network has dropped to two-thirds the industry average since installing the software.
The other form of payback is in staff reduction. But PrimeCo's Myckleby argues that the prospect of cutting staff through ESM could lead the IS group to resist the project. PrimeCo emphatically stated upfront that its ESM implementation would not result in staff cuts. Employees were told that they would be retrained and given new responsibilities if their old job functions were automated away. "We justified [the software expense] in terms of running the business more efficiently," Myckleby says. "That takes the fear of people losing their jobs off the table." Whether the company insists on concrete dollar justification for the ESM purchase or settles for an ROI based on system performance, expect the payback to be a long time coming. Graybar counts better problem isolation, increased uptime and higher companywide satisfaction with IS among the results of ESM but says it took a year before these benefits became apparent. "You have to have some faith to stick with it," says Hoffman. He notes, however, that Graybar jumped into ESM early and that the maturity of today's products may allow for a faster payback.
ESM is more than software; it is a corporate discipline. In this respect it resembles ERP software. Simply pulling it out of the box and plugging it in won't work. To reap any benefits, CIOs must examine their companies' management processes and policies before embarking upon an implementation. If there is a skeleton key to making ESM work, advance planning is it.
For example, about 700 mobile PCs and 300 desktop PCs throughout Belgium need to plug into Kind en Gezin's systems. In addition to off-the-shelf Windows applications and analytical software, those computers run a custom-developed application called Ikaros, which helps the agency's visiting nurses share patient files. Like USAA, Verhelst's agency wanted to automate the process of distributing software updates to their roaming client systems.
But once they got the ProVision ESM system running, they discovered that one-button software distribution can create chaos. If the help desk, the applications development team and the network manager can all push the button for software distribution, the result can be incompatible versions, not to mention a support nightmare. "We had to examine this process very carefully.
How do we change our client configurations?" asks Verhelst.
Kind en Gezin responded by limiting the authority for software distribution to a single change manager who would ensure appropriate compatibility tests were performed and documented before any changes were rolled out to the client systems. To avoid conflicts, Verhelst's agency went through a similar procedure to define and document its automatic job scheduling procedures.
Other veterans echo the call to examine, document and change processes. When PrimeCo decided to implement Tivoli's ESM solution, Myckleby brought in change management consultants to run two-day classes for the IS staff.
Keep It Staffed
With ESM, appropriate staffing means full-time staffing. "This system is not shrink-wrapped software," says Graybar's Hoffman. "We put one guy in charge of this thing from day one, thinking that once he got it installed he could go do other things. We figured out pretty quickly that it was never going to happen." The complexity of ESM software, and the networks it manages, demands a dedicated team for implementation and for ongoing maintenance.
A criticism that some have levelled against ESM frameworks is that they lack proactive capabilities. In other words, while the framework provides a central console for monitoring problems, it does not sufficiently increase the IS group's ability to anticipate them.
Vendors are striving to respond to this criticism. Most notably, last year Computer Associates announced the addition of neural network functionality for its Unicenter TNG product, allowing the framework to do more sophisticated pattern recognition and recognise earlier the events or sequences that can lead to performance deterioration.
However, Graybar's team says the issue of proactivity also can be addressed through appropriate staffing. "It's as proactive as you want it to be," says Don Ludwinski, the company's manager of networking and open systems. Ludwinski says ESM frameworks can be set up to simply page the technical staff when performance drops below acceptable levels, or they can be used proactively by putting someone in front of the console full time to probe and look for potential trouble.
Nobody claims ESM is easy. The goals of the practice remain ambitious, and the software is still a work in progress. But CIOs can avoid many of the pitfalls into which ESM pioneers have plunged. Coordinating not only the software but also the people and processes that make it work can result in a harmonious operation. In today's jumbled client/server computing world, that's beautiful music indeed.
ESM: What Is It?
ESM stands for enterprise systems management. Say what you will about the days when the frames were main and the terminals dumb, it was a fairly easy environment to manage. ESM is the discipline of trying to create a similar level of control and automation in complex distributed computing environments.
ESM encompasses many management functions, including but not limited to:* Automated software distribution * Distributed applications management and performance tuning * Database administration * Network management * Output management * Storage management * Automated backup and recovery * Help desk * Security The idea behind the software frameworks is to gather these various functions under a single umbrella. Sundry modules control and automate the functions listed above. All the modules plug into the framework, which provides a centralised management console to help pinpoint trouble spots, whether the problem is a server in need of a reboot or a fried board on a router.
-- D Slater
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.