Integration is no longer a choice - it's an obligation. And that means far more than merely connecting systems. It means marrying IT strategy to business goals. Not an easy task, but once achieved, the payoff is huge.
Despite years of advancement in application connectors and data interchange technology, integration between databases remains a major problem within both government and commercial sectors. The average organisation spends nearly 40 per cent of its IT budget on integration, according to some estimates, and difficulties bringing this down over time confirm the extent of the difficulties CIOs face in wrestling with the complex machinations of inter-database communication. The challenge is particularly tough in the government sector, where decisions about just what is allowed to connect to what are dictated not only by natural commercial imperatives for efficiency, but by constant external pressure from ministers, legislators, budget makers and others in powerful places.
What may seem like a simple collaboration between departments, announced with the usual pomp and circumstance by seemingly well-meaning ministers, can easily become an extra nightmare that government CIOs simply do not need. And in places where integration might normally make sense, political mores and legal restrictions on intermingling data can make it difficult for commonsense projects to proceed.
Legally enshrined methods for maintaining the independence of the judiciary, for example, have long-hindered the technical integration of court and law enforcement interests. In South Australia, even though the police, Attorney-General and Corrections are being linked in a developing integrated justice program, special approval was needed to allow the South Australian Court Administration Authority (CAA) to progress with development of a service that allows lawyers to go online to lodge civil claims directly with the CAA.
According to project director Steven Taylor, the organisation's ageing ObjectStar-based case management system, which handles more than 70,000 cases a year, would have required "unnatural acts and cost us an absolute fortune" to be upgraded in order to allow real-time transaction processing. But by putting validation forms online and sourcing information from claimants in XML format, the CAA developed a "workaround" whereby new claims are received throughout the day, then entered into the system in a nightly batch process.
This integration alone has shaved turnaround time for lodging such claims - which comprise 85 per cent of all claims heard every year - from three or four days to less than 24 hours. Better yet, the CAA has extended the life of the current case management system by one or two years, at which point Taylor believes a new database (Oracle or DB2 are likely candidates), fed by standards such as the evolving LegalXML, will make data exchange far easier to manage.
Despite the clear benefits of integration within and between departments, getting approval for such projects is not easy. "There are a lot of restrictions," Taylor says. "That's why integrated justice systems have never got off the ground. You've always got your technological issues with old databases, and doing real-time data integration from the Web has its own issues. But usually there's very sound and tried technology [to fix these]; it's not the technology, it's the bureaucracy and political climate. There's a lot of conservatism here."
Local councils have the benefit of being relatively free of the political performance pressure put on state and Commonwealth departments by optimistic ministers. And while the imperative to adopt e-government has forced many councils to look at ways of exposing their core systems to customers, integration with different levels of government has taken a back seat to simply consolidating and clarifying internal IT structures.
That process has forced IT decision makers first to address the often harrowingly complex heterogeneous infrastructure that has grown organically over the years. New South Wales' Sutherland Shire Council (SSC), the second-largest in the state, has been busy phasing out no fewer than seven different legacy databases to standardise on Microsoft SQL Server and Intel-based servers.
SSC's previous environment was, by all accounts, an integration nightmare. A decade ago it chose to base its core finance and administration system on a Fujitsu mainframe and IDB/2 database because, as IT manager Sid Curry explains, "other platforms we saw were developed for smaller councils, and we weren't feeling confident that other platforms could do what we wanted to".
Since then, SSC's IT panoply has grown to include applications based on UniVerse, Oracle and a number of proprietary Unix databases attached to systems handling land information, library, mapping and other functions. Those core systems that already offer native support for SQL Server are being configured to do so; others are being replaced with different applications or updated versions that are SQL Server-ready.
The new environment went live in time for the new financial year on July 1. While conceding that the change has introduced a major culture change and necessitated considerable retraining, Curry is optimistic that it will both save money and eliminate the complexities of the council's previous silo-based database architecture.
"We can't see any other way out of it; it's just too much of an expense trying to keep the level of skills and people required" to continue managing so many platforms, he explains. "We believe we need to get our systems in a satisfactory state. In the 10 years since we took the direction we had, there were more vendors out there. Now there's less variety, and councils will be coming together in terms of what they use."
Integration by consolidation is a luxury unique to local governments, however, since it is far easier to muster the top-down political will to force cooperation between business units. When inter-departmental issues come into play, the situation becomes far more complicated, and any IT strategist knows that complexity can mean delays and difficulties in driving the technical case for integration.
Planning, Planning and More Planning
Even when technical integration can be achieved, CIOs must plan and execute a massive data cleansing operation that can become a major sticking point for seamless interchange of data. That is because, database standards aside, each department will have its own way of representing data and may represent data about two different people in a different format. Even different naming or spelling conventions can make it difficult to cross-match data between departments, compromising the efficacy of whatever database integration is put in place.
Noting the increasing imperative for organisations in every sector to provide bridges between disparate applications, analyst firms envision strong and continuing demand for integration services of all types.
IDC, for one, anticipates the worldwide market for network consulting and integration services will grow from $US18.2 billion in 2002 to $US39.8 billion by 2005, with the Asia-Pacific region leading the world with annual growth of 31 per cent. At the same time, IDC has predicted, demand for enterprise integration software will lead the rest of the software industry with a compound annual growth rate of 43.9 per cent through to 2005.
While these figures include private enterprise as well as government bodies, the imperative to integrate is increasing equally across every sector. IDC identified a number of factors driving this push: the need to increase ROI on corporate information systems; the continuing requirement to link new systems with legacy environments; employee expectation of a consistent Web-based interface to departmental information systems; and structural changes brought about by politically driven forces such as departmental restructuring or IT outsourcing.
How, then, can CIOs reconcile the intrinsic complexity of integration with the service demands and ideological limitations of their political masters? One way out will be the widespread adoption of Web services - that evolving class of software standards that allows departments to hide the complexity of their internal systems while providing for the seamless exchange of data as necessary.
Web Services to the Rescue?
Web services standards are still very much works in progress - particularly those dealing with security, the first of which was only launched earlier this year by Web services architects IBM and Microsoft. Yet their core enablers - XML for data representation, SOAP (Simple Object Access Protocol) for moving that data between Web services, and UDDI (Universal Description, Discovery and Integration) and WSDL (Web Services Description Language) for directing services how to find and communicate with each other - are well enough advanced that workable Web services can now be built and integrated very effectively.
That means individual departments can continue to set their own technology agendas while collaborating with others as necessary or desirable. And that ability may ultimately be more appealing to departmental decision makers than the perceived sense that they are compromising their departmental strategies to meet the demands of others.
Analysts caution that early authentication and security standards mean departments should only trial Web services internally for now, but one organisation taking a more proactive approach is the Australian Taxation Office. Recognising the need to standardise representation and exchange of data across various departments, it was recently revealed that the ATO has spent nearly two years working with Microsoft and consulting firm Accenture to build the massive Australian Business Register (ABR).
Designed from the ground up around Microsoft's evolving .Net architecture, Microsoft spokespeople indicate the ABR is going to be positioned as a centralised repository of information about Australian businesses that will underscore new services from the ATO and other departments.
By keeping a single version of the truth and offering appropriate parties the opportunity to access that truth using a Web services interface, the ATO could single-handedly provide both the technical infrastructure and political assurances to propel government past its current integration stalemate.
ATO representatives declined to offer more detail on the ABR until it is formally launched, but as described so far, its potential for facilitating integration is clear. Other departments will eventually take the ATO's lead, abandoning expensive proprietary integration projects to rework their systems around XML. Economic downturn or no downturn, IDC anticipates that increasing takeup of XML-enabled databases will drive the feverishly-paced XML business-to-business market to rocket past $US3.7 billion by 2005.
Much of this growth will come at the expense of traditional enterprise application integration (EAI) middleware, which has long been used to mediate between disparate platforms but is becoming less critical as XML provides similar benefits through an open and universal standard.
Many departments are already making the shift towards XML in pilot projects or more ambitious deployments. Web services takeup among government bodies will no doubt mirror or even lead that in the private sector, where research firm Gartner has predicted the technologies will dominate deployment of new applications by 2004.
And, true to form, it appears that large Commonwealth departments will lead this integration push within the overall government sector. "I don't see a lot of [integration] at the state level," says Gartner research director Greta James, "and I rarely get an enquiry from local government outside Queensland."
A Time for Cooperation
In Queensland, which is now publicly billing itself as "the smart state", an accepted imperative to improve services is forcing some open discussions about the political and technical challenges of interdepartmental integration. Queensland Transport, for one, is weighing up the complexity of providing a smartcard-based driver's licence that also serves as a de facto identity card to support applications relevant to other government departments.
Data protection and consistency issues mean attempts to foist such services onto other departments are almost certain to fail without strong inter-departmental cooperation. But with the right approach, Gartner suggests, gradual introduction and expansion of the program would build public confidence in the system and therefore increase political will to support its expanded use.
The Queensland government's philosophical support would bolster back-end integration efforts, quickly opening up new service opportunities as other departments came on board. This process would be particularly well executed if Queensland Transport took the ATO's lead and eventually exposed its licence management system as a carefully controlled Web service securely accessible by other departments online.
Successfully executing such a change is a major step towards greater interoperability of data. Once that initial interoperability can be demonstrated, it will become easier to sell stakeholders on the benefits of ever-tighter database integration. In Queensland Transport's case, this could effectively create a cottage industry for the department, turning it into an authentication service provider and positioning the driver's licence as a statewide identity card able to identify Queenslanders for all manner of online and offline transactions.
The key to success is to spread awareness proactively and demonstrate the potential of early victories, says James. "Many organisations have struggled to take a formal platform approach to integration because it is infrastructure," she explains. "That's meant it's traditionally very difficult to get those projects approved, particularly when you're competing with projects bringing new products or services to market.
"It tends to be strings and sealing wax under the covers in many organisations, but to do this effectively there needs to be a common infrastructure using common technologies. Government has an important need in being able to deliver services more efficiently and effectively, but there's a lack of awareness as to what this takes.
"As with so many things, it takes IT people to be able to articulate these technology issues in business terms," says James. "You're putting the architecture in place to be able to give you agility, and to create linkages that you haven't envisaged yet." vMarching in Sync INTEGRATION is difficult in the best of circumstances. When you're the US Joint Forces Command (JFCOM) and your integration project involves four branches of the military and dozens of government agencies, it's an almost insurmountable challenge.
It helps when the mandate for integration comes from the US secretary of Defence and the head of the Joint Chiefs of Staff. JFCOM began its multipronged approach in 1998, when the secretary of Defence issued a charter mandating interoperability among the military branches. First, JFCOM created a governing body and drafted system standards to which all branches must adhere. The goal of the project, says David Ozolek, assistant director of joint experimentation for JFCOM, was to create a rapid response capability in which all branches of the military communicate via integrated systems.
"The Department of Defence has a history of problems with interoperability," Ozolek says. "Our initiative has been more than defining a set of systems. It's been overhauling the culture within the DoD." It's no secret the four branches haven't worked together well. For example, if a Navy pilot was flying over a group of Marines who were about to be ambushed, the pilot couldn't warn them because the Navy plane's radio equipment wasn't compatible with the walkie-talkies carried by the Marines. The same went for their computer systems.
"Each service had systems developed to meet their particular needs," Ozolek says. "It made us less effective and efficient."
JFCOM has established three approaches to creating a joint military context. The first, called Unified Vision, is a coalition of military bodies, government agencies, industry leaders and academics whose goal is to examine new technologies and organisational structures, and reconcile the needs and goals of the service as they move toward interoperability. JFCOM then changed its organisational structure. The Joint Interoperability and Integration (JI&I) structure creates and enforces system requirements, says Navy Captain Alex Urrutia, deputy director of joint interoperability and integration at JFCOM. JI&I also holds the purse strings for all integration-related expenses.
Because of security concerns, JFCOM can't provide details on the new standards, Ozolek says, but he can say they are extensive, covering everything from radios to data tags. Now, JFCOM is charged with ensuring all new and existing systems meet interoperability criteria.
The third leg of the project, called the Interoperability Technology Demonstration Centre (ITDC), is scheduled for completion in 2003. The ITDC will be the testing arena for systems, software and procedures to ensure compliance with JI&I standards. Eventually, the ITDC will set the standards.
Another concept called Rapid Decisive Operations (RDO) is now being used in Afghanistan. RDO involves networked command that lets all military personnel communicate and collaborate under a central command post. "It took us eight months to build up ground forces and plan for Desert Storm," Ozolek recalls. "Within the RDO structure, when the US went to Afghanistan, we had amassed and deployed troops to the area and had significant enemy contact within three weeks."
Getting the four branches to agree to work together was difficult, Ozolek says. What finally convinced them was a chilling projection devised to anticipate the circumstances of the next major conflict. The scenario involved domestic terrorist attacks by extremist Middle Eastern groups that escalated into a foreign invasion. "After September 11, they were totally convinced, and we were able to deploy to the Middle East as one military force, rather than four disparate groups," Ozolek says.
- Simone Kaplan
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.