Time to get Agile
- 13 December, 2010 08:00
Michael Bromley had a big, big problem.
Bromley, who until recently was head of online at Telstra, was trying to get traction on an extensive integration project to create a portal supporting a major business initiative. But by mid-2009, 12 months of conflict, infighting, and lack of development progress had imperilled the chances of the project being ready for its February 2010 launch and project managers were suggesting they’d be ready to start building sometime in 2010.
Two successive project managers had failed to improve the situation by the time Bromley discovered Tony Christensen, who agreed to try speeding things up on two conditions: One, that nobody question his development methodology and two, that he be allowed to break the project down from its existing monolithic format into more palatable and manageable pieces. The requests seemed like heresy to an organisation where top-down, architecture and analyst-driven monolithic ‘Waterfall’ development was the entrenched standard. But Christensen’s appeal reflected the tenets of ‘Agile’ development’s ‘Scrum’ methodology — in which projects are built using a succession of one-to-four-week ‘sprints’ that each produces usable code designed to an ever-changing ‘backlog’, or list of business functional priorities that’s reviewed and adjusted with each iteration.
His proposal touched more than a few nerves in an environment where Waterfall methods were well-entrenched. “I walked into an organisation that was very structured, had a fairly rigorous governance model, and some time ago seemed to have lost the means to govern for its future success,” says Christensen, a self-declared “passionate, pragmatic Agilista” who earlier this year followed Bromley to national broadband architect, NBN Co, where they’re currently advocating the method as the company builds out its own development culture.Christensen faced resistance when promoting Agile in Telstra: “The architecture team couldn’t quite understand how it was that we would set about to build things before we had designed them,” he recalls. “The argument I put forward was that if we engaged the architectural team early enough, they could help share our thinking as we did the project — and we could arrive at the design sooner, and with more quality and accuracy, than by waiting to review it at the end.”
Read how CIOs are using Agile methods.
After a slow start, the team warmed to Agile, which involved business leaders at every stage of the process rather than having them set requirements at the beginning and stay hands-off until the end. Developers worked straight through the Christmas holidays, completing the final build with hours to spare. The final application had far less functionality than the original specification, but was more than enough to bring the business online in time. The Agile philosophy is gaining popularity as organisations recognise the strictures of traditional development models that pressure business owners and analysts to comprehensively scope final projects early on, even though business requirements, resources and priorities would naturally change as the project progressed. By continually demonstrating progress, reviewing requirements and identifying issues early — and involving business leaders in the process — the Agile approach allows teams to shift priorities more easily.
“Agility meant we were taking onboard the fact that business stakeholders often had no idea what they really wanted,” says Bromley. “This is common and acceptable, but they had never before been given the opportunity to say ‘I’m going to change my mind’. Before, this always meant change requests, weeks of delays, and impact on other projects. However, by doing fortnightly showcases of what we had done and telling them to tell us what they really needed, it brought them together with us to work as a team.”
It’s an epiphany for some developers and heresy to others, but Agile has quickly gained currency in the corporate world as companies seize on its potential to shorten development cycles, improve developer efficiency and head off costly, problematic and massive projects before they trudge along for months or years to produce something that only partly meets business requirements. Perhaps not surprisingly, surveys of Agile proponents confirm that satisfaction with the method is higher than traditional development: Ambysoft’s July 2010 State of the Union survey of 233 respondents, for example, found a success rate of 83 per cent for companies using small Agile groups (those with 10 or fewer developers), compared with 69 per cent for small groups using traditional methods. And they’re providing great value to stakeholders, with another Ambysoft survey of 293 respondents finding that 88 per cent considered the value as providing working software, 74 per cent for having regular stakeholder discussions, and 71 per cent in identifying stakeholders and their goals. Agile has become a significant tool for CIOs eager to foster a new level of business-IT understanding as traditional barriers are disposed of in the name of non-stop collaboration and a shared sense of purpose. This makes it appealing for CIOs facing unprecedented pressure to deliver due to integration challenges, customer service mandates and tight budgets.
Traditional approaches, says Suncorp CIO Jeff Smith, are often unable to cope. “We’re the only industry in the world where economies of scale don’t work,” says Smith, one of several IT executives sharing their Agile experiences at the recent sold-out Agile Australia 2010 conference. “You put more people on a project and productivity goes down. The fact is that, in development, the fewer people and the more focused those people are, the better.”
Other Ambysoft survey results support his contention: Larger development teams were correlated in every instance with lower success rates. Large teams (those with 26 or more developers) were successful just 55 per cent of the time using Agile methodologies, 50 per cent with traditional methods, and just 40 per cent of the time using ad-hoc development. In small teams — those with 10 or fewer people — the success rate jumped to 83 per cent using Agile and 69 per cent using traditional methods.
A three-year advocate of Agile, Smith has used the approach to drive a dramatic change in the way projects are managed within the company. The philosophy started within the IT division, but has expanded across the change-focused organisation under an umbrella program called ‘Living Agile’ that is empowering staff to think differently about their roles.
“The key is to have consistency in the things that bring you the greatest value,” Smith says. “It’s better to run a few things and get them done quicker than to run 10 things and not complete them. Our iterations and releases are going so quickly that they’re not trying to throw everything in right now; the biggest projects they’re doing are four to five months, and once they get that done they tick it off and prioritise it again.”
This sort of flexibility has become desperately important in the financial services industry, which is still working to shake off the effects of the punishment it felt during the global financial crisis. Faced with an underlying sense that massive transformation projects no longer meet requirements to act more nimbly, financial services CIOs are using Agile methodologies to improve their project flexibility and make the most of their resources. Many times, the desire for change comes not from simple experimentation, but a pressing business need: Bankwest, which has been dealing with the implications and changes inherent in its acquisition by the Commonwealth Bank of Australia, adopted Agile as it faced the need to improve its processes without traditional monolithic processes.
“Our ability to sweat our assets, and continue to serve our business, became really important,” says CIO, Andy Weir. “We were very good at Waterfall development — but being able to move nimbly and quickly, and add value to the business without making really long-term, multi-year investments, is going to become more and more difficult for us. Strategically, it’s really important for us to be able to become more nimble, increase our speed to market, and continue to add value for our business customers.”
Mounting evidence has confirmed the value of Agile within organisations of all kinds, but the extent of change necessary to make it happen is nothing to sneeze at. The mechanics of Agile may be readily accessible, but it involves a significant element of people change that CIOs and their teams may find both challenging and invigorating.
For example, business analysts and project managers, who have long been relied upon to scope, plan, construct, evaluate, and revise project plans, often end up marginalised or eliminated from the process altogether. Because Agile involves constant back-and-forth between business leaders and the people building for them, intermediaries are no longer required.
“Architects are very clever people and they will spend lots of time analysing and coming up with something elegant, but they don’t know whether it can be built,” says John Sullivan, general manager of online directory and search with Sensis, who has spent the past three years transforming the company’s Directory division into an Agile-based development organisation.
“The only time they know whether it can be built is by writing the code,” he continues. “Architects are not accountable for projects, and it’s the business owner and the guy writing the code who will be caned if the project is late. Telling people how to do things is not unleashing the your skill set; you need to push architecture responsibilities to the team, set a direction and a common understanding of where they’re going, and trust them to deliver really good things.”
Sullivans recommendations echo the results of another Ambysoft development survey, in which architects punched above their weight when respondents were asked why projects fail. The findings suggested that ‘ivory tower’ enterprise architects are often too inflexible, posed impediments to project success, and hindered development teams from getting on with things.
The issues were most effectively addressed when architects were active participants on business teams, trusted advisors of the business, and more flexible; in Agile thinking, these results are all available when CIOs flatten the reporting infrastructure by shifting architectural responsibilities to developers.
Yet developers need to stay focused, too: Sullivan warns it’s important they don’t get carried away, building broad frameworks and extensive code that is likely to go unused because it offers far more than the business needs.
“Some people just build stuff, but you can’t go building features expecting the business to then come up with uses. Who cares whether your code is the best or not? Only build what’s good enough, and does the job.”
Project managers, too, need to step away from their comfort zones, focusing less on being prescriptive and organisational, and more on removing roadblocks that stop developers from doing their best.
These roadblocks — be they environmental conditions, opponents of change, unnecessary complexity, or even traditional governance requirements that restrict change — may be resolved through careful politicking or drastically, by reallocating or even dismissing recalcitrant developers.
“We’re asking project managers to be much broader in their outlook, and we’re broadening their role within the organisation,” says Josh Melville, transformation manager with life reinsurer RGA Australia, where a five-month business intelligence project paved the way for an Agile-led organisational transformation.
“Project managers are really coming to a new space in terms of the value it creates in the business, the influence they have on the business, and therefore the way we need to invest in their capabilities.” Some CIOs find that the best way to go Agile is to not only change existing project management roles, but throw them out the window.
It became important for Suncorp’s Smith, for one, when incoming Suncorp chief executive, Patrick Snowball, saw the results Agile had delivered and started planning to use it across other Suncorp divisions, to hasten the company’s functional transformation.
“He wanted a single claims system rolled out across all 13 Suncorp brands, a single underwriting and pricing system, and one view of customers, and so on,” Smith laughs. “I figured this would take three to four years, but he said ‘you’ve got one year’.”
With political infighting strong and business leaders fighting to run the project, Smith ultimately decided the best way to drive change was to step out of the way completely. Line-of-business executives were instructed to choose four thought leaders from within their businesses, and those leaders empowered to build Agile teams from Suncorp’s base of about 2000 staff.
The approach let the company “pick the four smartest people we have in the company and let them organise their own programs”, Smith explains. “We stepped completely out to act as coaches and mentors, and within two weeks we had moved 50 per cent of our total delivery capacity onto this program under the auspices of four people that had never led anyone before. It’s been a phenomenal exercise, and a phenomenal success.”
Next: Building teams and governance
Team building for the future
The push towards Agile business forces reconsideration of project roles, but it also pushes CIOs to become better at facilitating interaction between team members. Some developers have, for example, pushed back against paired programming, an Agile technique for increasing visibility and speeding problem-solving.
Others have struggled with regular stand-ups and ongoing business engagement, which mean their work is held up to the light far more frequently and intensely than in the past. “Developers often aren’t used to dealing with the business face-to-face,” says Bankwest’s Weir.
Yet people from the business also struggle as they’re pushed from being a passive customer to an integral part of the Agile delivery effort. “These are people that have been completely isolated from the very reality of what it takes to deliver capabilities in an organisational context,” says RGA’s Melville. “Now we’re pulling them onto projects and it’s quite confrontational: Not only are we pulling them in, we’re foisting on them a whole new level of accountability and expectation.”
Weir has experienced similar issues: “By bringing the developers and testers directly face to face with the business,” he says, “they often ask questions — and if the product owner can’t answer those questions unequivocally and quickly, it’s pointless putting them all together.”
The key to resolving this dynamic is making sure product owners have the attitude and authority to make decisions, Weir says. But such issues are only the beginning; in order to facilitate Agile development CIOs must also address what may seem like some quite pedestrian issues, such as ensuring team members get enough ‘face time’ and dealing with the logistics when teams are split between sites. “Distributed Agile is hard to do,” says Suncorp’s Smith. Energy distributor, Jemena, recently faced this issue in working with integrator DiUS Computing on a year-long Agile redevelopment of its core energy management systems that was split up into four discrete projects managed using ‘sprints’. Up to 15 DiUS workers, along with a handful of staff from other organisations, conducted group stand-ups via videoconferencing, instant messenger and collaborative whiteboard applications, and the team embedded developers from Sydney to Melbourne, and vice-versa, to minimise isolation between sites.
Subcontractors add another element to Agile projects. Their programming skills may be valuable, but they can also lack the organisation’s enthusiasm for Agile and could potentially become an obstacle if they’re not playing by the same rules. Subcontractors can often learn to play along pretty quickly, however, when provided with short, sharp tasks and clearly defined goals.
CIOs must also consider the effect of Agile on conventional governance models. Agile adopters seem to value function over form, with one survey finding the most important success criteria are, in order, functionality, quality, time/schedule, and money.
“Our issue [with the business] isn’t money; they generally want to give you money,” says Suncorp’s Smith. “Our issue is more about constraining their thinking about how much throughput they can take to their business change, because a lot of things hit the same sets of people. Chemistry and cadence are really important.”
The priorities reflect Agile’s relatively flexible approach to scheduling and its focus on delivery rather than budgets. But they also have implications for the business, such as project documentation. Developers and business leaders often have widely varying perceptions of the importance of documentation, but Agile’s focus on delivery often pushes developers to compromise documentation standards.
It is a deliberate compromise that also reflects the tendency of Agile projects to change in mid-stream. “We’re all technologists and prefer to build software than document it,” says Paula Ngov, senior consultant with DiUS Computing, who led Jemena’s development work. “Rather than completing a full-function specification, we documented them in the form of a behavioural-driven development, in plain-English business language.”
RGA’s Melville has waged a similar war on documentation, eschewing project managers’ traditional Gantt charts for more direct communication and collaboration.
Sensis’ Sullivan is more direct: “Documentation delivers nothing to the business,” he says. “You can’t sell it; it doesn’t return more value; and it doesn’t help customers. What brings money in is actually the code that meets the needs of the business.”
If Agilists seem to be taking a relatively mercenary approach to project development, that’s because it’s in their blood. In the end, Agile is about removing traditional roadblocks to development and encouraging a flat organisational structure that nurtures collaboration, exchange, reflection, adjustment, and mutual progress.
Fostering successful Agile development, however, can even involve changing working spaces — known as ‘bullpens’ in Agile speak — as Bankwest’s Weir found out when a consultant from development partner ThoughtWorks came by for a spot check six weeks into the project.
“She asked why we had put the team in a broom closet,” Weir recalls. “I realised we had set up this strategic objective that said Agile is very important, and that we’re going to throw everything behind this, and then we hadn’t even changed the tiny room where our Agile team was meeting.
“I wasn’t fully behind it because I wasn’t providing the right tools and the right environment to let that thrive. If you’re going to get behind it, you’ve got to really get behind it; it isn’t just about a development methodology; it’s about every single component coming together to make it successful. We’ve invested a fair bit of money creating formal Agile spaces, and the team absolutely love it.”