Code Of Practice
- 10 May, 2004 10:11
Developers have been refactoring for years and will continue to do so, and in this regard, it is nothing new. But with the advent of new "agile" processes like extreme programming (XP) refactoring really rules.
Not so long ago, Robert Watkins, a senior Java architect for Suncorp-Metway, was working on a large and complex system development requiring loads of integration when it became clear that some underlying assumptions were proving to have been just plain wrong.
A reasonably common experience whenever there are unknowns prior to development, the discovery was nevertheless a real inconvenience: It meant the company would have to make significant change to a large part of the underlying application if its performance was ever to prove satisfactory. In many organizations that would have meant either undertaking "major surgery" or living with the problem because the cost of fixing it was seen as prohibitive.
At Suncorp-Metway they know just how to make those improvements without taking the system offline and without dropping all other programming work.
"What we did over a period of time was to introduce several 'refactorings' to isolate the performance problem and let us put in a functionally similar piece of code that happened to run a lot better," Watkins says. "It took about two months to do, because we had to keep the system working and we couldn't drop everything we were doing, but at the end the system was actually functional because the speed was acceptable. If we hadn't solved this we wouldn't have got the system into production."
Watkins, an application systems specialist in Suncorp-Metway's J2EE Centre of Excellence charged with promoting industry best practices in development, says whether or not his organization actually refactors varies from project to project. Officially the group has been behind refactoring for about nine months.
"Refactoring is all about changing code without changing behaviour," says agile programming expert Dr Neil Roodyn. "You don't expect to get extra features from doing it; you do it to keep the code maintainable. Refactoring is like keeping the workshop tidy; it enables you to work in a clean and tidy environment."
But refactoring can also be a low risk, incremental development methodology that can save businesses money by helping them build on existing infrastructure and experience.
ThoughtWorks chief scientist and software development guru Martin Fowler describes refactoring as a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behaviour. At its heart, he says, is a series of small behaviour-preserving transformations (refactorings), each of which does little on its own but which put together in a sequence, can produce a significant restructuring. "Since each refactoring is small, it's less likely to go wrong," Fowler says. "The system is also kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring."
Most organizations struggle with "worn code": Code that has been modified, extended or truncated to suit changing business requirements, until the integrity of the original design is lost and any documentation rendered useless. Refactoring the code can be a useful and relatively painless way to address the problem. In fact some development technologies, such as extreme programming, depend on it (see "Sabre Takes Extreme Measures", page 118). And refactoring comes with significant business benefits. Organizations that refactor existing software can better leverage IT investments and reduce the risk inherent in further additions to software in the future.
"While the structure and code has been modified, the user should not see any difference in the system," says Colin Garlick, senior trainer at Wellington, New Zealand-based Software Education. "As such, any tests, whether automated or manual, should be able to verify that the functionality of the system has not been changed." He points out that software developers have been doing refactoring for years wherever an existing system needs to be modified in response to new or changing requirements but the structure of the existing system does not facilitate the required modifications.
Refactoring consists of a number of patterns that have formalized what software developers "just did" in the past, according to Garlick. Formalizing these refactoring practices gives software developers a clear set of steps to follow in making changes, ensuring it is harder to overlook a required step and further minimizing the chances of introducing a bug. Having clear steps to follow also makes it easy for development tools to provide automated support for refactoring, allowing the software developer to say "I want this change made" confident he or she has the tools needed to manage the detail.
"A common tendency for software developers is to change the structure and make the new modifications all at the same time," Garlick says. "Such major changes are difficult to make without introducing bugs ('regressions'), and because of the scale of those changes, locating the source of any bug is also difficult. By applying refactoring first (before making the new modifications), any existing tests (in particular unit tests and automated test scripts) can verify that the system has not been broken. If any bugs are introduced, they are easy to locate and fix because of the smaller scale of the changes made."
But if the techniques themselves are old, and small refactorings have been done as long as people have been writing programs, the term itself has also been around for some time. It arose out of the Smalltalk community - the pioneers of many object-oriented techniques - in the 80s. The recent rise in the popularity of the term has to do with the simultaneous increase in interest in agile methodologies, and particularly extreme programming.
While reworking code has been common practice since software's earliest days, refactoring takes the practice to a different level altogether.
Eric Raymond, author of The Cathedral and the Bazaar - the collection of essays that has been credited with persuading Netscape to release its browser as open source, and putting Linus Torvalds on the cover of Forbes magazine and Microsoft on the defensive - sees refactoring as part of the process that is turning software engineering from art to science, much the way alchemy eventually led to chemistry.
"I'm beginning to think that from the wreckage of the software industry big dumb management made, I can see the outline of a mature, humane discipline of software engineering emerging - and that it will be in large part a fusion of the responsiveness and customer focus of the agile movement with the wisdom and groundedness of the Unix tradition, expressed in open source," Raymond has said.
Aberdeen Group predicts that despite the hype of recent years, it is "refactoring", not "reusability", that will truly lead the way to improved programmer productivity (http://www.aberdeen.com/ab_abstracts/2003/01/01030010.htm). After reviewing the evidence, Aberdeen concludes encouraging reuse in today's toolsets is having little positive effect on productivity - and sometimes even a negative effect - because development tools suppliers and users alike have mistaken a possible means (reuse) for a goal (faster, repeated delivery of software value-add to the customer).
Aberdeen now calls the over-focus on reusability what it is: A harmful waste of IT buyers' and developers' time. By contrast, it concludes that while refactoring has a slightly negative effect on project time in initial application development, from that time forward it can dramatically affect programmer productivity.
"Moreover, refactoring can help users avoid 'software sclerosis', in which legacy applications become more and more difficult to upgrade to new technologies and hence more and more costly to maintain," Aberdeen says, although it notes a paucity of true refactoring tools to date.
Dr Peter J Kootsookos, a lecturer at University of Queensland School of Information Technology and Electrical Engineering, says the practice was more common in Ireland, - one of the top three software exporters in the world - when he worked there than it is here in Australia. He says refactoring should become one of the tools of every company doing software development, to be used as and where appropriate.
Few organizations have a formal refactoring program in place, and Suncorp-Metway is no exception. Rather, Watkins says, refactoring is viewed as a normal part of the development process. But he says for refactoring to be successful, the organization must foster an environment where the practice is encouraged.
"A lot of places will have a focus where they put in a piece of functionality, and then they say, okay, I'll just move on to the next piece," he says. "The problem with that is that over time, the code becomes rather inflexible. By using refactoring as one of the steps to keep code clean and flexible, you improve your overall productivity even though each individual step looks like it has taken longer because you are taking this time to clean up."
Doing so, he says, also helps overcome problems with understanding code. Where code is hard to understand, Watkins says, whether because it was poorly written or the documentation related to it is confusing, then refactoring makes it easier to understand. The emphasis is on a series of safe transformations. "That's one of the key differences between refactoring and other forms of redesign - that it is meant to be safe," he says. "You're not meant to be changing anything, you're not meant to be breaking anything, it's really just rearranging the code to make it easier to read."
To keep the process safe, organizations should refactor using a series of "test harnesses" and test code - major conditions necessary to prevent human error. Without the solid test harness, organizations may find themselves introducing bugs into a clean system, he says.
"Refactoring gives us the benefit of continuous improvement of our code base," says Brad Long, development manager at Oracle's Australian Development Centre. "However, to ensure code still meets its requirements since making changes to working code can be risky we couple refactoring with a comprehensive unit and system test suite. Our development team uses a continuous integration environment (continuous code compilation) and has in excess of 4000 unit tests which run daily against our product."
Long says such unit tests are critical to code quality and allow developers to use refactoring with confidence. Without the unit tests, he would not recommend anything but carefully planned refactoring or reworking of projects. "In a nutshell, refactoring gives us the benefit of continuous improvement - not necessarily driven by external requirements - but you need the testing infrastructure to allow refactoring with confidence," he says.
Organizations should also encourage refactoring for "migration projects" when systems reach the end of the lifespan on an existing platform and have to move to a new one. William Scheer, chief operations officer of TurnAround Solutions, a systems developer with headquarters in Sydney, says his company has done a number of migration projects after systems had reached the end of their life on a particular platform or system. "What that requires is looking at the existing code bases, and either rewriting based off of that code, or refactoring that code and redeploying," he says.
Refactoring is also helping the company with a major project involving the taking over of an existing system from a supplier and redeploying and improving its maintainability and supportability. Scheer says that is a major business benefit of the practice, which should generally be seen more as an operational efficiency driver than a growth driver. On the other hand, where refactoring is used to refresh system requirements it does give the organization a chance to look more critically at the way a system has been structured, and if changes are needed, a way of using those changes as a path to access new markets.
When to Refactor
Kootsookos says organizations should never refactor arbitrarily, but only when there is a real business reason to do so, although that reason might be as basic as the fact that the code has been written in such a cryptic way that new developers cannot understand it. And refactoring is best done during quiet periods; it is not a practice to undertake when deadlines are looming. "In general, refactoring requires a deeper understanding of the code and the way the code base is organized, and for that you need probably to be in a bit more contemplative frame of mind," Kootsookos says.
When to refactor can be a matter of personal preference, Watkins says. His own practice is to examine the code for each piece of functionality upon its completion to see whether it has been written cleanly. "If it doesn't express what I was trying to say cleanly," he says, "then I will apply refactoring at that point, to improve the communication process, to make it easier to read, and easier for someone else to come in and understand."
Watkins also routinely uses refactoring before adding new functionality to existing code, if it appears the code needs to be made more flexible to accommodate that functionality. "So I do it both before I start work, and afterwards," he says. "If you think of writing code as cooking, then refactoring would be like cleaning up the kitchen. You would do it beforehand if the kitchen was messy, and do it afterwards to make sure the kitchen is clean again."
Some organizations might do small refactorings every day, but also do larger refactorings that take several months to roll out for systems with serious problems. Such an approach often involves "a round trip to the development life cycle", Watkins says, which means going back to the whiteboard, documenting the new solution then putting it in place and finding out whether it works.
But refactorings are primarily intended to be safe. The primary lesson is to be sure to proceed using small steps, Watkins says, no matter how large the refactoring itself is. You should also ensure each step is verified as it goes, because it can be very easy in large refactorings to head down the wrong path.
"If deadlines are looming it's usually not a good idea to refactor," Kootsookos says. "You have to have a reason. It's the old 'if it ain't broke, don't fix it'. You should refactor when you add functions, when you need to fix a bug, as you do a code review, or when code 'smells bad'," he says. You should avoid refactoring like the plague when it changes a published interface or when deadlines loom.
"One mistake teams often make is to attempt to do refactoring in big sections," Roodyn says. "This can be extremely painful for the developers, the managers and the customers as from the outside it appears that no work is being done."
Organizations should also position themselves to be ready to take advantage of any unexpected business benefits generated by the refactoring.
Scheer says one refactoring TurnAround Solutions did for a telco customer started essentially as a rehosting - moving a system off a mainframe-oriented environment onto a Unix-based server system - and has now become a project about delivering better service to customers. As the project migrated from a focus on improving performance and refactoring in terms of architecting technical architecture, to the services provided to customers, different stakeholders, including marketing, became involved.
This presents a challenge in itself that the IT group must manage, Scheer says, and the best way to do so is to ensure strong project governance and a solid alignment between IT and the business sponsors.
Scheer says the main difficulty with refactoring code is that the documentation is often not up to date, and business/technical knowledge about the inside of the system to be refactored has often "decayed". That means before you refactor you should undergo a period of upfront analysis, re-engineering of the design information and matching the system to the original requirements.
Then, as long as you make sure the refactoring will provide some business benefits, it can be one more valuable tool in the armoury used to ensure IT really does work to meet business needs.
SIDEBAR: Learn More . . .
Previous Articles in CIO
- Jack Be Nimble, Jack be Quick, CIO, October 2002
- Secrets to Software Success, CIO, August 2001
- Refactoring: Improving the Design of Existing Code, Martin Fowler, Kent Beck, John Brant, William Opdyke, Don Roberts, Addison-Wesley, 1999.
- Refactoring Workbook, William C Wake, Addison-Wesley, 2004.
- Agile Software Development Principles, Patterns and Practices, Robert C Martin, Prentice Hall, 2002.
- A Practical Guide to Extreme Programming, David Astels, Granville Miller, Miroslav Novak, Prentice Hall PTR, 2002.
- Data Sharing Using a Common Data Architecture, Michael H Brackett, John Wiley & Sons, 1994.
- Extreme Programming and Agile Methods: Xp/Agile Universe 2002: Second Xp Universe and First Agile Universe Conference, Chicago, Il, USA, August 4-7, 2002: Proceedings (Lecture Notes in Computer Science, 2418), Laurie Williams, Don Wells, Friedrich Schiller, Springer Verlag, 2002.
- Extreme Programming Examined, Giancarlo Succi, Michele Marchesi, Addison-Wesley, 2001.
- Extreme Programming Explained: Embrace Change, Kent Beck, Addison-Wesley, 1999.
- Extreme Programming Explored, William C Wake, Addison-Wesley, 2001.
- Extreme Programming Installed, Ron Jeffries, Ann Anderson, Chet Hendrickson, Addison-Wesley, 2000.
- Extreme Programming in Practice, James Newkirk, Robert Martin, Addison-Wesley, 2001.
- Planning Extreme Programming, Kent Beck, Martin Fowler, Addison-Wesley, 2000.
- Test-Driven Development: By Example, Kent Beck, Addison-Wesley, 2002.
- The Capability Maturity Model: Guidelines for Improving the Software Process, Software Engineering Institute, Carnegie Mellon University, Addison-Wesley, 1995.
- The Pragmatic Programmer: From Journeyman to Master, Andrew Hunt David Thomas, Addison-Wesley, 1999.
A six-part Insight series by Wayne Kernochan on the challenges and opportunities facing today's software development strategists. The following are titles and links:
- Developing the Enterprise's Software Portfolio: A Long-Term Strategy to Avoid "Software Sclerosis"www.aberdeen.com avoid software sclerosis
- Agile Programming, Extreme Programming, and Refactoring: Not Just Another Development Fad www.aberdeen.com agile
- Existing Application Upgrade Is Key to Future E-Business Strategies www.aberdeen.com upgrade
- Programmer Productivity Reconsidered: Reusability Considered Harmful - Refactoring Not www.aberdeen.com programmer productivity
- Software Quality: Test, Automate, Satisfice www.aberdeen.com software quality
- Design-Driven Development: Good Only If Done Wellwww.aberdeen.com design driven development
Web Resources for Extreme Programming
- Extreme Programming A Gentle Introduction: Provides an introduction and overview of extreme programming (XP). www.extremeprogramming.org
- XProgramming.com An Extreme Programming Resource: Offers downloads, articles and more. www.xprogramming.com
- PairProgramming.com An Extreme Programming Practice: PairProgramming.com houses articles and links to other sites discussing XP. www.pairprogramming.com
- Ward's Wiki Pages About Refactoring www.ward'swiki refactoring
SIDEBAR: Extreme Programming's 12 Core Practices
- Customers define application features with user stories.
- XP teams put small code releases into production early.
- XP teams use a common system of names and descriptions.
- Teams emphasize simply-written, object-oriented code that meets requirements.
- Designers write automated unit tests upfront and run them throughout the project.
- XP teams frequently revise and edit the overall code design, a process called refactoring.
- Programmers work side by side in pairs, continually seeing and discussing each other's code.
- All programmers have collective ownership of the code and the ability to change it.
- XP teams integrate code and release it to a repository every few hours and in no case hold on to it longer than a day.
- Programmers work only 40 hours per week; there's no overtime.
- A customer representative remains on-site throughout the development project.
- Programmers must follow a common coding standard so all the code in the system looks as if it was written by a single individual.
SIDEBAR: Sabre Takes Extreme Measures
Using extreme programming practices, Sabre Airline Solutions has reduced bugs and development times for its software products
By Gary Anthes
Saabre Airling Solutions had many years of experience with its AirFlite Profit Manager, a big modelling and forecasting package that many airlines use to wring more income out of flight schedules. Even so, Release 8 of the software was four months late in 2000 after final system testing turned up 300 bugs. The first customer found 26 more bugs in the first three days of its acceptance testing, and subsequent joint testing by Sabre and the customer uncovered an additional 200 defects.
"Our Sabre person on-site was as upset with the quality as the customer was," recalls Vinit Karandikar, a principal in Sabre's Airline Products Development group.
Sabre's experience with Release 8, which had almost 500,000 lines of code, isn't unusual. Software quality guru Watts Humphrey, a fellow at the Software Engineering Institute in Pittsburgh, estimates that commercial software typically ships with one to eight defects per 1000 lines of code.
But a lot has changed in the development labs of Sabre Airline Solutions, part of Sabre Holdings Corporation, a Texas-based, $US2 billion air-travel systems company. The software development firm has embraced extreme programming (XP), a controversial development framework in which testing precedes coding and programmers work in pairs. Sabre Airline Solutions has 62 software products with 13 million lines of code, and it claims that XP has dramatically boosted both the productivity of its 300 developers and the quality of their work.
For example, fewer than 10 bugs surfaced in Release 10 of its Profit Manager software in the two months after it began shipping to airlines in December 2002. Now, 16 months later, just 100 defects have been found. Release 10 was written in Java, while Release 8 was written in C and C++. But Sabre says it was XP, not Java, that produced the dramatic quality improvements.
And the initial benefits from developing better code may be eclipsed by long-term cost savings, Karandikar says. For Release 10, Sabre assigned just three developers to support 13 customers, while Release 8 required 13 people to support 12 customers.
The evidence is anecdotal so far, but there are enough anecdotes at Sabre to make a compelling case for XP. In another project, the company converted the user interface of its AirServ airline cabin provisioning optimization system from C++ to Java for the Web, a two-year effort that required rewriting about 100 graphical user interface programs. Programmer productivity jumped 42 percent - as measured by the number of labour hours required for each screen - after the development team switched to XP halfway through the project.
In yet another project, the Host Access Tool, which provides a common application programming interface for accessing legacy host systems and has 15,000 lines of code, was written from scratch using XP practices and has remained bug free in the 20 months since it shipped. Similarly, Sabre's Peripheral Manager, which manages interaction between host systems and peripheral devices, has 28,000 lines of code and has had just four bugs show up in 15 months.
Establishing Best Practices
Sabre Airline Solutions adopted XP in 2001. With its new model, Sabre does iterative development in small, simple steps. The company uses two-week iterations, and customers see a new release every one to three months. Features, called "stories", are expressed in user terms and must be simple enough to be coded, tested and integrated in two weeks or less.
Automated unit tests (against the programmer's criteria) and broader acceptance tests (against customer requirements) must be passed at the end of each iteration before the next can begin. Unit and acceptance tests for each feature are written before the feature is coded. If a developer has trouble writing a test, he doesn't clearly understand the feature.
Actual coding is done in pairs by teams in open labs, promoting collective ownership of code, although individuals sometimes do the simplest tasks. Programmers are re-paired frequently, often every day or two. They sign up for the tasks they want to do and the person they want to pair with.
Every project team has an "XP coach" and an application subject-matter expert called the XP customer. The XP customer stays in or near the programming lab all or most of the time. He decides on and prioritizes product features, writes the stories for programmers and signs off on the results.
Refactoring code - rewriting it not to fix bugs or add features but to make it less redundant and more maintainable - is strongly encouraged. Sabre says the concept hardly existed at the company before XP because it was too difficult.
Finally, simplicity is paramount. The simplest things are done first, and code is never made more complicated in order to accommodate an anticipated future need that may never materialize.
Brad Jensen, senior vice president for airline product development at Sabre, says the quality improvements come largely from XP's continuous testing and integration. "Every two weeks what you've completed has got to be production-ready," he says. "You code as you test. You actually write an automated unit test before you code the unit, so if bugs do creep in, you find out about it right then."
Damon Hougland, director of airline products and services, says pair programming is hard for some to swallow at first because it suggests that programming costs will double. But the method actually reduces cost, he says, because the extra time it takes to write a line of code is more than offset by the reduced effort required for testing, fixing bugs and maintaining the code.
And, because at least two people know every chunk of software, and because Sabre reassigns and re-pairs people frequently, there is always a backup on hand. "Everyone on the team works on every part of the system," Hougland says. "You have the weaker people paired with the stronger people, and business knowledge and coding knowledge are transferred very quickly."
XP doesn't encompass all the practices that a software development organization should follow, Hougland says. "XP really focuses on what [programmers] do," he says. "It doesn't cover the traditional project management you have to do with customers, such as customer communications, and a lot of the testing we do is not covered in XP. A lot of people try XP and fail because they assume that XP will do everything for a development methodology."
Sabre doesn't yet follow XP 100 percent. XP doctrine says that both unit tests and acceptance tests should be automated. Unit tests at Sabre are conducted with the open-source JUnit testing tool, but many of the more complex acceptance tests are conducted with manual scripts. "We've struggled to automate them, and we have made some progress," Hougland says.
Jensen compares XP to the Capability Maturity Model for Software, saying XP covers many SW-CMM practices. The difference, he says, is that CMM tells organizations what they should do while XP is more oriented to saying how to do those things. "CMM is all about management processes; it's less about technical processes, the techniques of coding," says Jensen.
Jensen, who sponsored the adoption of XP at Sabre Airline Solutions, says he considered using other so-called agile programming methods, such as Agile RUP, a version of IBM's Rational Unified Process; Scrum, an iterative, user-oriented methodology; and Feature-Driven Development. "None of them are as fully developed as XP," he says.
Jensen says he especially likes XP metrics because they come in user terminology rather than the technical jargon of some methodologies. "With XP, we are checking off features and stories that I can understand like a business person," he says. "There's a lot of increased communication between the customer and the development team because everything that's being prioritized is in terms the user can understand."
Jensen acknowledges that his developers deviate somewhat from pure XP practices. Sometimes the XP customer has to travel and can't be in his lab full-time, as XP doctrine demands; refactoring old legacy code can be extremely difficult; testing isn't as fully automated as XP would have it; and acute deadline pressure from customers can lead to cutting corners.
But when asked if he has reservations about any of the 12 core XP practices, Jensen says: "They are all good practices. The only thing you could possibly argue is whether you really can afford to do all of them all the time. But if there's any compromising, it's not intentional."