When project manager Kerrie Reyn was told she had to find a new home for Department of Employment and Workplace Relations’ entire computer hub and backbone — and do it within six months — she thought her bosses must be kidding. They weren’t, and so began an epic race against time.
Everyone knows it. Government is traditionally an area where outcomes can take some time. Red tape, bureaucracy, the sheer size of the operations — all conspire to make change a slow business. But sometimes the pace hots up. Such an example was the decision late last year by the Australian Taxation Office (ATO) to give the computer operations centre of the Department of Employment and Workplace Relations (DEWR) its marching orders. The reason was space, or more specifically the ATO’s wish to expand its computer operations, and that meant that DEWR’s massive computer hub had to find a new home.
Kerrie Reyn, whose “daytime job” is director of customer relations at DEWR, was tasked with moving the department’s operations to new premises, which at that time had still to be found. And it all had to be achieved within six months, a time frame many would consider impossible.
“When I was told, I didn’t go, ‘you are joking’. I’m afraid it was a little stronger than that,” says Reyn. “Moving the system was daunting enough but to be told it had to be done within six months was, frankly, frightening.”
Frightening not simply because of the physical move itself — and the need to find suitable accommodation for the system as swiftly as possible — but also because any interruption, any problem with DEWR’s computer centre, could have sent the Jobs Network and Centrelink operations into a death spiral. Both of those major “clients” are served and supported 24 hours a day every day of the year by the DEWR computer operation.
“External delivery of these services is primarily through the Internet,” says Reyn, “So in a business perspective we are significantly externally focused. Apart from providing policy provision for ministers and the government, and supporting our own internal mechanisms to facilitate that, the bulk of our business delivery is external to the jobs network agencies.”
This meant that Reyn was in charge of ensuring that a computer network and database that supported and fed around 30,000 branches of the Jobs Network had to keep working. The DEWR operation also links organizations like the Salvation Army into the jobs network.
“For example,” says Reyn, explaining how the operation works, “we provide a plug-in to Centrelink as well as all the job network agencies like the Salvos, so we are the data repository. We are the hub, the centre. We are the main database for these operations. We would hold information like the status of job network members, the subsequent payment set the job member may get, the allowance that an unemployed person would get, and the training, when they undertake that — we also link to agencies offering training — and the outcomes. So, we carry and hold a lot of very important information.”
The big issue for Reyn and her team was not only a purely logistical one: how to move all that equipment safely and effectively without any disruption to operations; but also a practical one: how would you find a purpose-built building capable of housing the operation?
“That was a real challenge,” Reyn says. “We had to remain in Canberra, so that limited our options for a new home. Given the scope and complexity of the equipment, and what we were being asked to do in the time frame, we considered it at the time to be a near impossible mission. In fairness, the pure logistics in doing it were a known entity and something that we fully appreciated. So what we had to move, we were comfortable with.
“But the issue first and foremost was the availability of sites within Canberra. We were reliant on them being in Canberra because of part of our infrastructure, and we already knew there were not a lot of computer facilities readily available that we could simply just move into.”
So Reyn started to build a team around her, starting with a small management group to make daily decisions and to set up the approach and strategy they were going to follow. Then they sent out a Request For Information (RFI) to the marketplace to assess which vendors could assist in the project management of the relocation exercise, establish the centre, get it up and operational, assist in moving into the new centre and then provide ongoing facilities management.
“As a department we’d already made a decision that we wanted to move ourselves away from the facilities management role of the data centre,” says Reyn. “The ATO had facility-managed their arrangements for us and we wanted to continue that type of relationship because it worked well. We didn’t want to facility-manage the new operation ourselves because it’s not something we have a core expertise in.”
Partly because of the tight time frame the RFI went out on a restricted basis to companies the department already knew, and who had carried out similar exercises for them in the past.
Rather than treating the move as a real estate problem, the team chose to treat it as an IT issue. It was natural to turn for advice to its key strategic partners, all seven of which had long-term connections with the department, and which might be expected to be well aware of all the facilities in Canberra that might suit.
“Now, cost was a factor but in terms of the very tight time frame, demonstrated ability to deliver was the big one for us.”
Three of the companies contacted said they could do the job.
“The real estate they were offering was consistent across the three responses we got,” says Reyn. “So, then it came down to the project management fee and everything else on top. They all had a base outlay that was consistent and so we could compare easily. In the end we went with HDS because the quote was competitive and they had relocated our computer centre on two other occasions, successfully and on time. We have an ongoing relationship with them — though we did with the other two as well — but HDS had a demonstrated ability and so we were comfortable and confident that they could achieve the outcome.”
The main requirements were not only that the operation had to be in Canberra but also that the building chosen had to have approximately 750 square metres of space — and here was where Reyn and her team faced their first hurdle.
“We wanted to get one site but we ended up eventually securing two sites because there simply wasn’t one site available that would have met our needs. So we ended up with 360 square metres in the suburb of Bruce and we also ended up with another 350 square metres about 10 kilometres away.”
Although initially this was not seen as a good thing, clouds can have a silver lining, and so indeed did this one. In the end, it turned out that having two sites gave DEWR the ability to make one site predominantly a production facility while earmarking the other as a development test operation with the option of a backup facility for the production facility.
Still, there were additional problems to overcome. The main site in Bruce was effectively a large computer room. It had a raised floor, and there was no power, no data cabling and no live telephone connections. It was simply a bare computer room. And the other site was an office building that needed complete refurbishment and kitting-out before equipment could be moved. DEWR had been told by the ATO on April 14 that it had six months to make the move and it was not until July 23 that the buildings had been earmarked. The clock was ticking . . .
“There were quite a number of issues involved, aside from the timing, which obviously put a lot of pressure on us,” says Reyn.
“The first thing we had to decide was what equipment went to the computer room site and what went to the secondary site. Then we had to go through an analysis to see what the requirements were for each site and we had to identify what the power drain was going to be because of that split of the equipment. Then we had to look at revised overheads for the Uninterruptible Power Systems (UPS) and the motor generators for that equipment. Meanwhile, at the second site — which essentially became a building site — UPS and motor generators did not exist. They had to be imported, they had to be configured and they had to be installed and the entire main distribution boards and everything else electrical had to be done from scratch. It was an absolutely massive job.”
While Reyn took over the project management and computer siting side of the operation, HDS organised all of the subcontracting, the army of electrical and data contractors, and took on overall management of the two sites.
“We had complex issues to be managed as well as developing the relationship with these two primary sites,” says Reyn. “For example, we had Storage Tek equipment, we had HP equipment, IBM equipment, we had HDS equipment and in our contracts with these companies only they could move their own equipment.”
But again there was a bright side, because Reyn and the team could afford to look at equipment replacement and also at installing leading edge technologies. It was an opportunity that few established IT operations ever get.
With many of the racks holding servers and all communications equipment up for asset replacement, DEWR brought in new racks as it established the new computer room. It was able to standardize the rack types and install all new data cabling and all new electrical cabling.
DEWR also moved to innovative blade technology and that presented a number of significant technical issues. One was the heavy weight of the blades, which led to discussions with technical experts as to the weight-bearing capacity of the floors.
“We had to recognise that staff would be under those blade racks,” Reyn says, “so there was an OH&S issue in case the floors collapsed. There was also a heat issue because the blades produce a lot of heat, so we had to make sure that the appropriate clearance and air circulation was factored in. Given that we were condensing 96 blades into a given rack there were cabling issues too: being able to cable back to a communications switch centre was a technical solution that had to be devised in-flight, as it were.”
The team also considered the many options concerning how to configure the power beneath the floor. The technical team had multiple views about the most appropriate course, which they had to work through to reach an agreed solution.
In the middle of all this, there was a change of minister, which meant the team responsible for communications had to go elsewhere to configure a new minister’s office and electoral office.
“And our staff who were looking at servers became heavily involved in backing up data for the departing minister. We had actually joked about it earlier and said this is a possible project, but all we’d need would be a couple of changes, like a new minister. We stopped jokes from that point.”
Fitting It All In . . .
When all the basics were in place, the move began.
DEWR took a staggered approach to the move, broken down into six weekends of relocation. The first four weekends were dedicated to moving into 90 East, deemed the production facility, after the team made the strategic decision to bring the 90 East facility directly into the network.
“We brought it in as an additional node and the way we configured it was that our then still existing facilities at ATO and our new facilities at 90 East effectively looked as one,” Reyn says.
“This meant end users of the system certainly would not have had any awareness as to where the equipment was housed during any period of this relocation. We had to have scheduled unavailability, which was discussed and agreed with our various business areas at the beginning of the project. In stating that we remained at high availability, part of that was because we were able to negotiate with the business areas the times we would effectively relocate equipment. That buy-in and commitment from our business areas (and we were able to communicate that back out to people like Centrelink and the Job Network) was key to the success of the move.”
Teamwork was certainly a major factor in the success of the project and so was the use of a couple of first-rate communication tools. One was Telstra’s Meet Me conference phone system.
“We had key technical staff — around 10 core people — who were effectively empowered to make the decisions they needed to when they needed to. We set up regular meetings and had briefings daily at 9.00 am every morning where we discussed what the issues were and what needed to be achieved on that day, and we positioned ourselves for the next week, fortnight, few months out.
“What Meet Me allowed us to do was rather than drag everyone into an office, we set up a 9.00 am Meet Me call every day and as many people as could attend would dial into the number and we’d have an online meeting. It became a core tool. The people dialling into it changed as the work progressed and the number of people on the line extended or contracted depending on what the issues were.
“We also established a status line so on a weekend any interested party could ring in and get a two-hourly update on where we were in the process. That worked really effectively, giving key management representatives a recorded update during critical milestones throughout each moving weekend.”
Reyn says that a reason for the relatively smooth running of the project was a lack of interference from senior managers. “We were empowered beyond belief. We were demonstrating throughout the project that as key decisions were having to be made we were considering them, receipting them and making a valid decision. And so I think that sense of empowerment and the confidence to basically sit back and let us get on with the job was fantastic.”
When DEWR completely switched over to its two new centres there were a lot of crossed fingers — but it all switched on perfectly, smoothly and seamlessly and everything was completed on schedule.
“To be honest,” says Reyn, who can now have a bit of a laugh about it all, “I think we had quite a few dramas and we certainly had issues to manage on a daily basis. As an example, given the volume of cabling that needed to be installed, we had to make some quick management decisions to see if we would await a Certification of Occupancy, and we just couldn’t. We had to forgo that and move into what was effectively a building site. But at the end of the day it was worth taking the risk because we achieved the move in the time frame.
“I think overall our success was a reflection of the right people at the right time and I also think it was a reflection of the strong degree of commitment of all parties involved in the project. I think it was that high degree of trust, as well as the understanding that the decisions that had been made were going to be followed through within the time frame, that really made it work.”
DEWR on the Move
» Moved and fitted: 900 servers, four SANs (storage area networks), six silos, one mainframe, comms and firewall equipment, 200-plus racks, 5500 new data cables laid (that’s 35 kilometres of cable), 2000 power circuits, 39,000 kilograms of equipment, 96 optical fibre pairs connected » Relocation cost: $2.2 million » Fit-out cost: $2 million
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.