You Know It Makes Sense to Respond

When the grasslands of the marketplace are constantly shifting, organizations will only prosper if they can react - fast - to both internal and external events. That is why sense and respond (S&R) systems will inevitably - and probably in just a few short years - become commonplace.

In organizations, just as in organisms, responding suitably to critical threats can be a matter of life and death.

The zebra that fails to detect the stalking lion becomes that lion's dinner. Yet a zebra continually fleeing imagined threats would be just as vulnerable, risking death by exhaustion or a blunder into hostile territory in which it simply could not survive.

Matters are not so different on the corporate savannah. Events - many of them impossible to anticipate - can threaten from both inside and outside the organization. Competitors, governments, news organizations and markets can create both opportunities and threats.

When the grasslands of the marketplace are constantly shifting, organizations will only prosper if they can react - fast - to both internal and external events. That is why sense and respond (S&R) systems will inevitably - and probably in just a few short years - become commonplace, according to Dr K Mani Chandy, professor of Computer Science at the California Institute of Technology (Caltech). These systems, as their name suggests, sense, and then respond to opportunities and threats, then expand an enterprise's capacity to react.

"Animals that respond to too many false positives die," Chandy says. "A false positive is an unreal opportunity or threat, so responding to false positives wastes energy. An animal dies if it spends more calories hunting than it gets by eating.

"The stork gets streams of information from its senses: eyes, ears and touch. Much of the time it stands around doing nothing. Yet for perhaps five minutes in every hour, responding to a rare event - threat, opportunity or other critical situation - may be a matter of life and death," Chandy says.

"So for the remaining 55 minutes the basic functions - breathing, heart beating and the flowing of digestive juices - continue while the central nervous system filters out the other data streams, telling the stork that no critical event has occurred. When an opportunity appears, such as a meal in the form of a frog, the central nervous system alerts the mechanism, and this alert is crucial."

As it is with creatures, so it is with corporations. Think of evolution as largely being about honing each organism's ability to tell the difference between real and imagined threats and opportunities. Now organizations need to take a leaf out of these organisms' books to survive, Chandy says, by replicating those sense and respond capabilities. The good news is that your company is almost certainly event driven ("If it weren't it would be dead," Chandy says). Now it is time to prepare the ground for the time when you will want to sharpen that ability to respond rapidly to change.

"All enterprises have to respond to events to succeed," Chandy says. "We must recognize the IT organization in general, and you in particular, aren't going to make your enterprise an event-driven enterprise - it already is. The question for us in IT is whether we can help make our enterprises more agile."

Out of Your Control

The not so great news is that when it comes to monitoring your external environment, essential data is almost never in your control. Neither competitors nor collaborators structure data in schemas to suit your enterprise. They do not deliver data to your schedule either. That means errors will inevitably creep in. Also, data painstakingly gathered from your environment may be imprecise (as in natural-language text), wrong or delayed. Worse, systems can fail to detect genuine threats or opportunities, or raise false alarms about non-existent threats and opportunities.

That is where S&R comes in, Chandy says. Caltech's Infospheres Group defines such systems as "systems that detect critical conditions in an extended distributed environment and respond proactively".

S&R systems can embrace the reality of corporate data limitations by highlighting asynchrony and recognizing imprecision. They expand the human ability to respond to threats and opportunities. They monitor events both outside and inside an organization and create the changing big picture by integrating information from multiple sources about events. They identify new opportunities and threats as the big picture changes, and respond by invoking applications and sending alerts to devices.

S&R systems rely on individuals and software applications to define the conditions that will produce alerts. Such conditions may be rare, or based on sensitive information.

"The system monitors data sources within collaborating institutions and from non-collaborating (and potentially hostile) agencies. Data from multiple sources is normalized to a common vocabulary and integrated or 'fused'. The system detects when any of the specified conditions begin to hold. When a condition holds, alerts are sent securely to specified destinations authorized to receive the information," Chandy explains.

Page Break

Sense and respond systems are already commonplace in many sectors. In some areas, like foreign exchange, commodities or stock trading, events happen rapidly and the software has to respond just as fast. S&R systems also exist in military command and control, which has been devising these systems for more than 50 years. There dealing with the enemy infers looking outside your own forces and "dealing with all the noisy information that they get from the outside world". S&R has also increasingly influenced process control, control of the electrical power grid and control of chemical plants, where S&R applications have been in play for many years. But now, Chandy says, it is time for other enterprises to start moving to achieve increasingly fast response to situations, both expected and unexpected.

That is not to say they need to do anything specific - yet - about their products and tool sets. It does mean laying the groundwork conceptually, he says, and understanding the event-directed architecture (EDA) these systems will eventually sit on relies on asynchronous operations.

As the Simon Ramo Professor of Computer Science at the California Institute of Technology, Chandy is conducting research on distributed computing. Currently he is working on development of an EDA to sit as a thin layer on top of service-oriented architectures (SOAs) to take advantage of S&R, as well as on applying S&R systems to crisis management. He says an event-directed architecture can help organizations to capitalize on the event Web (EW), a thin layer on top of the World Wide Web developed by Caltech's Infospheres Group. Continuously active, the EW monitors information sources and responds fittingly as conditions change. Organizations can build new S&R applications to respond continuously to critical conditions on top of the EW.

"The event Web is an S&R utility that helps people respond to critical conditions in their environments. EDA is the architecture that allows for the systematic structuring of S&R applications. Event-directed architecture components monitor the environment, process events and respond to changing conditions continuously," he says.

Change of Focus

We have had 40 years of enterprise IT focus on services within the enterprise. That is no longer good enough, Chandy says. External events demanding an immediate reaction continually buffet today's corporations. They must alert government agencies in short order should certain situations occur. They have to react fast to competitive threats, breakdowns in extended supply chains and vagaries in electronic markets.

That is why S&R systems differ from traditional enterprise information systems, which focus only on information from within the organization where the time, place, form and accuracy of events are controlled. S&R systems monitor events both inside and outside the enterprise. In an S&R application, finely tuned actuators or responders execute actions in response to detected critical conditions.

Chandy says it is time for CIOs to recognize that they will eventually need to build an event-directed architecture to capitalize on S&R. An event-directed architecture lets the enterprise monitor, correlate and respond to both internal and external events. While many organizations are still struggling with SOA, they should start laying the groundwork for EDA now, he says.

"There are many companies, including IBM and Oracle, that have existing products or are modifying products to focus on this event-driven architecture and thus make their companies more sense and respond companies," he says.

"A service-oriented architecture is where the client calls the server when the client needs something. In an event-directed architecture a server gets called on when something happens at any time. It's not a synchronous operation between a client and server: it's more like things happen when they happen and you react when you have to react. [CIOs] must keep that in mind as they build out this SOA architecture," he says.

Chandy says the business proposition of EDA is that it helps enterprises respond to events, whether threats, opportunities or other kinds of situations demanding timely response. In preparing the ground for S&R, CIOs should keep some questions about the business proposition in mind. How important is it for the enterprise to respond to situations as they occur? How might EDA help the organization respond in a proper and timely fashion?

Page Break

"The organizations working to implement an EDA so far tend to be in niche application areas. I think it will remain with niche areas [the aforementioned foreign exchange; commodities or stock trading; military command and control; control of the electrical power grid; and control of chemical plants] for the next few years and then maybe in 2008 or so it will maybe be picked up by a broader set of critical functional areas," he says.

The good thing, he says, is you already have the components of EDA in your enterprise stack, and that EDA and SOA are compatible, despite the differences in their primary objectives. You can build EDA on top of SOA using SOA components.

"Developing IT to help your organization become more agile does not imply that your organization needs to respond in seconds," Chandy argues. "There are very few applications where sub-second response is required though there are many where responses in a few minutes rather than days can make a huge difference. Now IT has a chance to help the organization become even more effective in responding to events," he says. "The company already has the architectural pieces it needs to build EDA and you need only an incremental effort and cost to put it together for a different goal."

The question for each organization is whether the incremental benefits of IT for EDA exceed its incremental costs. Or more completely, do the incremental benefits of IT in helping your company become more agile exceed the incremental costs of building event-driven applications?

Growing Reliance

Society is coming to increasingly rely on sense and respond systems in areas from national security, health-care, finance, supply chains and energy and environmental protection, to security and IT infrastructure management.

Steve Haeckel first coined the term "sense and respond" as a business concept in a 1992 Management Review article. At the time he was director of Strategic Studies at IBM's Advanced Business Institute.

Haeckel has written that what began as a label describing a desirable type of organizational behaviour evolved over the next six years into "an adaptive managerial framework incorporating a set of concepts, principles, prescriptions and tools for creating and managing an adaptive enterprise". Now, he says, it has become a "comprehensive, scalable and internally consistent recasting of industrial age strategy, structure and governance to cope with the post-industrial environment of unpredictable change".

That is because a rapidly changing marketplace makes it impossible for any business to thrive for long just by making products and selling them.

"It does not matter how good you are at making widgets if the market for widgets disappears or if your competitors offer dramatically new and improved widgets faster than you can," Haeckel writes in a recent book, Adaptive Enterprise: Creating and Leading Sense-And-Respond Organizations.

In today's environment, he says, a company can only succeed by knowing how to adapt to customers even before those customers themselves know what they want. The answer is to build a sense and respond approach to promote rapid response amid change. That means using innovative ways to gather information about customer needs. Some examples: car manufacturers used video cameras in airport parking lots to discover that people often struggle to lift heavy suitcases over the high lower edges of boots. In mall parking areas, cameras showed shoppers had nowhere to put the soft drinks they bought. As a result, low boot edges and cup holders are standard features in almost every car.

Yet many still treat the term S&R as synonymous with "agile", "lean", "flexible", "resilient", or "near real-time", or to characterize adaptive technologies like agent-based modelling or to name tools and methods that support human adaptability (for example dashboards). This is wrong, Haeckel says: without a managerial model to leverage and exploit such capacities systematically and coherently, the tools' and methods' impact will be limited and operational, rather than transformational and strategic.

At, Haeckel summarizes an S&R managerial framework, noting that what is missing from most attempts to transform businesses into adaptive organizations is an adaptive management model.

"Adaptive people, technologies and infrastructures are necessary but insufficient, because the legacy industrial age management system systematically discourages the exploitation of adaptive capabilities," he writes. " . . . Those firms that have made a degree of progress in becoming more adaptive (or at least more agile or resilient) have by and large relied on process improvement, adaptive technologies, and on what Bruce Harreld of IBM has called 'the heroic model' of management - counting on exceptionally talented people to break the rules without breaking too much glass.

"Sense and respond fills the adaptive management gap. It is a fundamentally different framework for on-demand, customer-back businesses; one that systematically leverages adaptive individuals, technologies and infrastructures to produce and scale adaptive organizational behaviour," Haeckel writes.

Haeckel points out any organization that cannot trust its ability to predict future requirements must eschew planning, process designs, hierarchies of authority, and command and control. An adaptive enterprise should be able to sense and respond effectively to events on the ground, rather than relying on operational excellence to efficiently make and sell products and services that customers are predicted to want. Its core competencies must include the capabilities to: know earlier (anticipate) the meaning of what is happening now; dynamically dispatch modular capabilities in response (pre-empt); and express strategy as a systems design of roles and accountabilities.

"This transforms strategy from a plan of action into a structure for action," Haeckel notes. "The organizational design becomes the main strategic document, and policy executives must therefore acquire competence as business architects. In an adaptive sense and respond enterprise, leadership controls context, not process and activity."

Page Break

Model Behaviour

An enterprise sense and respond system comprises a network of distributed S&R systems, each serving a specific role. "S&R applications share many of the behaviours of groups that respond collectively to their environments. You can think of a pride of lions as an S&R system consisting of multiple lions, each of which, in turn, is an S&R system," Chandy wrote in a recent research paper.

"Living things form models for the ways in which components of their environments behave: a zebra has an innate model of a lion's behaviour; cattle that graze near a railroad track learn a model of a train's behaviour. Similarly, S&R systems have models of their environments, which may be either specified by users or learned by the system. Machine learning algorithms help the systems learn critical conditions," he noted.

"Devices such as pocket PCs, PDAs and mobile phones, and technologies such as e-mail and instant messaging, generate increasing amounts of data," Chandy wrote with co-author Jonathan Lurie Carmona in an article entitled The Event Web: Sense and Respond to Critical Conditions. "Think of these data items as dots. To understand the changing big picture, you have to connect these dots as they arrive. Opportunities and threats are visible in the big picture, not in the individual dots. S&R systems collect dots, create the changing big picture, identify new opportunities and threats as the big picture changes, and respond by invoking applications and sending alerts to your devices.

"A great deal has been written about information utilities that provide information services just as electric utilities provide electricity. A valuable service is one that responds or helps you respond to threats and opportunities. The event Web (EW) is just such a utility. (Articles that appear later in this series will discuss the design and implementation of the EW)," they wrote.

Examples of S&R applications that can be built on top of the EW include those that would help with the following activities:

  • CEOs respond to possible violations of Sarbanes-Oxley compliance rules
  • operations managers respond when service level agreements are likely to be violated
  • information assurance officers respond to inappropriate patterns of access to sensitive information and applications
  • purchasing managers respond to purchasing and logistics opportunities
  • public health officials respond to the potential onset of a pandemic
  • financial traders take advantage of arbitrage opportunities
  • teenagers get alerts when favourite bands tour nearby locations (at times that are open on their calendars and when ticket prices are within their budgets).

Software architectures developed as compositional structures based on request/response calls, where components like procedures, objects, servers and services reply to synchronous requests with a response. Just like that stock, those components remain quiescent until they receive a request.

Chandy says that although SOA is the framework for systematic composition based on synchronous request/reply calls, event-driven applications are continuously active rather than remaining quiescent until they receive a request. In fact event-driven architectures deal with components and compositional structures that facilitate continuous monitoring and response.

Practical Barriers

Not every company will need event-driven applications: it all comes to a consideration of incremental benefits and costs.

CIOs should consider which existing business processes already respond to critical situations that might benefit from automation. They should also evaluate whether the responsibility for responding to some critical situations should live within a clearly identified group. Where it does, Chandy says, it can be much easier to prove business value. They should also ask whether the organization is meeting its current performance requirements and how much they might improve performance with IT support. Is this incremental improvement in performance valuable to the business activity of responding to critical conditions?

"You will find that developing an app is easier if the number of data sources is small, and there are a small number of rule templates that can be tailored and used by a large number of users," he says. "Keep in mind that a large part of the costs is the time required from business users. Yes, eventually this time pays off. But, getting the business user's time can be a hurdle.

"Building an effective event-driven application is mostly effort from businesspeople. Certainly building a sense and respond enterprise or an agile enterprise is very much a business issue rather than an IT issue," he says.

Chandy says it is one thing for CIOs to promote adoption of S&R, another to get the organization to take advantage of the technology. Unless the company is ready to exploit the IT system once it is in place, CIOs might as well not bother. The US government's dismal response to hurricane Katrina was not due to lack of technology, but to organization failures, he says. "Agencies in the government had information, IT was working okay, but the organization couldn't respond in a timely fashion. This is not primarily a technology problem, this is primarily a management and organizational problem.

"And that is a considerable hurdle. If you're used to doing things one way and you have to start doing things in a much more proactive way, it might be difficult. You won't get much out of your sense and respond technology unless you can achieve that change in culture," he says. There is no point spending a lot of money to make your technology ahead of your organization's ability to adapt to it.

He warns CIOs to beware the hype. Vendors are selling SOA as if it were the answer to any CIO's prayers, and hype is just as likely to dog efforts to understand the benefits and limits of EDA, he says.

"I think they can both be problems: dealing with organizational changes that are required and dealing with the hype, which is why we are moving gradually over the next few years," he says.

"I think it will take [CIOs] at least two more years to digest service-oriented architectures, and plan for sense and respond and then carefully evaluate the return on investment."

For now, Chandy advises CIOs to hasten their efforts to gather knowledge about S&R. Go to conferences and read plenty of material first. CIOs should evaluate their organizations, identify the S&R applications that can add value to their organizations and decide if their organizations and management are ready to adapt to the new technology. Then they can evaluate the components in the enterprise stack that they already have that may be enough for the applications, carry out a careful cost-benefit analysis and only then consider buying new S&R-specific products, he advises.

Come 2008 or so, by which time Chandy fully expects Gartner and other organizations to be holding event-driven architecture conferences, the time may be right to make your move. Doing so will just be plain good sense, he says.