How does an enterprise deal gracefully and effectively with such unwieldy mountains of information?

With the costs per megabyte of storage plummeting even as actual storage costs soar, organisations are starting to pay careful attention to all matters relating to e-mail, data storage and storage policy management. There is growing realisation that until the organisation has a storage strategy and clearly defined storage policies, it has little hope either of getting control of exploding storage costs, or of dealing with the multiple headaches the demand for storage can create.

That became clear to Royal Australian Mint IT manager John Power as the organisation grappled with the multiple problems caused by a growing volume of data and e-mails - 16GB of e-mails alone - stored on its network servers. Not only was the explosion in storage demands creating a server-side performance issue, the Mint was finding it impossible to put in enough disk space to keep up with the volume of e-mails. As for the hidden costs of managing that storage - well just how long is a piece of string?

"Initially what we did was to just keep pestering people to delete their old superfluous e-mails, and try to whack as much storage as we could into the server. And it didn't work. It was a temporary fix, like a bandaid - that was all it really was," Power says. "We also found that there was a hell of a lot of our corporate knowledge tied up in these internal and external e-mails, and the issue was that because we couldn't store them, we were losing them. We were losing all that corporate knowledge."

Things began to change after a seminar on e-mail storage, rules and regulations that introduced Power to Legato's e-mail management product, which the Mint has now fully implemented. He says implementing product and process has reduced the time he spends on maintenance of the e-mail database from an average of 20 to 30 minutes a day, to perhaps 30 or 40 minutes a month. Time spent retrieving stored e-mails has also been dramatically reduced. The solution has freed up around about 8GB or 9GB on the e-mail server, with most stored material now sitting on a high-end PC. And now all work-related material gets stored, and corporate knowledge is no longer endangered.

Robert Hillard, managing director, data and knowledge management with BearingPoint, says between a host of regulatory drivers such as the US Sarbanes-Oxley Act and Basel II, which are forcing organisations to store much more data, and commercial trends like the new customer expectation of information integration, there has been a "phenomenal" explosion in the growth of information being stored on the corporate information network information. At the same time, clients at the C-level want to know how to cope in a world where physical storage costs have plummeted, and yet the cost to the organisation of storage continues to soar.

"Now what that means is: Okay, I've got all the information, I've stored it on my network, but now I've got to put this management layer over the top of it to try and make sense of it, to organise it, along with what's becoming a phenomenal amount of noise," Hillard says.

There is growing concern among CIOs about the hidden cost of storing information - that is, the additional management overhead that comes with every piece of information that has to be stored, according to Hillard. So CIOs have to find ways to retain all information deemed significant under regulatory requirements, plus as much other information as possible that may prove be relevant to the business, while getting rid of as much as possible of the information that is currently being retained, which has absolutely no business value while just creating "noise".

A survey by US CIO conducted in February 2002 found that on average 22 per cent of a company's total IT budget will be allocated to storage this year, and the magazine reports some analysts as estimating the budget bite can go as high as 50 per cent. More than half of the survey respondents reported a new focus on storage by senior management.

According to some estimates, although the cost of storage capacity plunged by a factor of 100 from 1990 through to 2002, the cost of managing that storage rose by a factor of seven. And Gartner Group believes just 20 per cent of storage TCO amounts to acquisition costs, with the rest including "hidden costs" related to lack of scalability, performance degradation and stored data availability.