With the growth of critical data rapidly increasing, the Salvation Army’s Southern Territory arm was running out of “prime real estate” in its storage area, according to its IT manager, Larry Reed.
The not-for-profit organisation, which encompasses the Northern Territory, South Australia, Tasmania, Western Australia and Tasmania, has 4910 employees and about 9000 volunteers. The IT team operates 3500 computers over the organisation’s private wide area network, and its data centre is located in Blackburn near Melbourne where its head office is based.
Back in 2009, the organisation was suffering at the hands of both unstructured and structured data from Microsoft SQL databases, IBM Lotus Noted email and collaboration software and client records, case notes and payroll which its ageing EMC Clariion CX300 could no longer handle.
After spending six months looking closely at Dell’s Equal Logic PS storage system, Reed said a consideration of standing investment in EMC technology resulted in a change of direction.
“We’re a Dell shop and we looked at the EqualLogic as a potential solution. In fact, we were pretty much going to go down that track but we ended up looking at our investment in the EMC equipment,” he said.
“The fact that we had all the fibre channel switches and all the infrastructure in place and we got a very good price from EMC changed our mind.
“We had experience with it and it was a known technology that we had used and had no problems with previously, so it was a safe bet.”
The implementation of two CX4-120 network storage systems in late 2010 was initially to use one in production and one for disaster recovery (DR), Reed said, but demand soon meant that both were needed in production.
Hand-in-hand with the networked storage growth, were issues with the organisation’s daily tape backups, which were running into business hours and affecting the performance of servers.
“We were starting at 8pm and finishing at 11am or later the following day, and this time was only set to increase,” he said. “With some backups running up to 16 hours, we didn’t see a big future in expanding our use of tape.”
Reed said the tape also compromised the ability to quickly retrieve backed-up information as it was stored in an offsite storage facility where IT staff would have to go to locate the information and restore it to production systems, often taking days at a time. Most importantly, he said, was the archiving of information and being able to retrieve that information quickly.
After close consideration of backup systems from various vendors, including Commvault and Quantum, the organisation again opted for what was known — implementing EMC’s Avamar for data deduplication, backup and recovery.
The interim period was tough, he said, with staff having to archive old emails on their personal computer hard drives which could not be searched, meaning they would be lost if the drive was corrupted.
“Our Territorial Commander wanted us to have an effective email archiving system, and we wanted to expand or eliminate altogether the quotas on the organisation’s 3200 email accounts,” he said. “We also wanted to retrieve at will any email that had come into or out of the organisation.”
The implementation of the Clariion CX4-120s in early 2011 and the Avamar system later that year were done by the Salvation Army in conjunction with EMC and implementation partner Thomas Duryea in parallel over a six-month period.
“Most of the implementation was done by Thomas Duryea and EMC providing the specifications because there was a requirement to understand what we were backing up, how much we were backing up and getting the capacity and scripting right,” Reed said.
“The biggest issue we had was putting in the Avamar here [in Blackburn] and also at the Bourke Street premises, where our 2 megabyte link wasn’t big enough so we ended up upgrading to 10MB.”
The IT team can now complete nightly backups between 10pm and 6am, eliminating performance problems and file lock-outs, and store 30 days’ worth of data.
“Coupled with the EMC CLARiiON CX4-120 systems, EMC Avamar has made a considerable difference — we certainly don’t have people complaining about performance,” he said.
“The organisation now only conducts one tape backup a week, and has eliminated the administration and cost of copying data daily onto tape.
“We have cut out about 16 tapes a week, and we can restore data in minutes rather than days.”
The entire project was in the order of about $220,000, Reed said, which was covered by his annual IT budget and supported by the organisation.
“The organisation recognises that IT enhances the running the organisation, we couldn’t run it without the IT being involved.
“For any of these type projects, we put forward a business case which spells out what we’re doing, why we’re doing it, and the rationale behind the expenditure and it all came down to.
“We’ve got a duty of care to follow and consequently putting in a technology such as the Avamar gives us that peace of mind that in the unfortunate event our data centre is corrupted we have a complete replica at an offsite location.”
According to Reed, although the Avamar platform can have information restored within half an hour or less, they have not eliminated tape completely. Instead, the organisation keeps a weekly ‘tape out’ for archival storage.
He noted the importance of carrying out extensive planning to ensure the process is documented and to ensure there is an understanding of current issues.
“Bring in a trusted partner who can analyse the current situation and can provide for the recommended solution according to your requirements,” he said. “Do all the pre planning that you can do possible because that will end up paying dividends in the end result so you need to ensure that’s done.
Follow Chloe Herrick on Twitter: @chloe_CW
Follow Computerworld Australia on Twitter: @ComputerworldAU
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.