Introduction

This article introduces the basic concepts in agile software development. Agile software development contrasts with traditional approaches in that it acknowledges software development as inherently unpredictable. Because of this philosophy, agile methods try to avoid large upfront investments in both requirements analysis and code infrastructure. Instead, agile methods favor short bursts of development activity, rigorous developer testing and refactoring, and close communication with an onsite customer.

Traditional methods of project management encourage teams to follow fairly strict phases of activity, each of which must be completed before the next begins. This means that requirements are gathered and approved first, followed by software specifications, design, coding, and finally testing. In theory, this ensures a smooth, predictable flow, where the output from each phase provides enough information to the next phase to provide accurate estimates, and predictable delivery. Unfortunately, by the time the entire process is completed, the business circumstances may have changed, the original stakeholders may no longer be involved, or other priorities may have emerged. A team can deliver perfectly to the original specifications, and still fail to meet the needs of the business at delivery time.

Agile methods recognize the uncertainty of the business climate, and assume that requirements and priorities will change. Most of the practices are geared towards embracing this change and minimizing the impact of it. As such, agile methods do away with strict phases, preferring to gather information just in time from the customer.

In this article, I’ll describe some of the problems that can occur using traditional methods, and how agile techniques can help to solve these problems. While a full-fledged agile approach may not be right for every project, many of the techniques I’ll describe can be applied in traditional development projects as well.

Author Dave Churchville has a strong background in Agile techniques – he is co-founder of Extreme Planner Software, which has developed a project management system for Agile-based teams. However, the techniques described in this article don’t require any software to implement.

The Problem

I’m going to use a hypothetical commercial software company called Old School Software as a case study throughout this article. Old School makes enterprise software solutions for large, powerful companies. They use a traditional product development cycle, with major releases every two years, and phases of development including requirements analysis, design, implementation, and testing. Recently, Old School has seen its sales decline, as more competitive products are taking away market share. The management team asks Steve, the VP of Sales and Marketing, to come up with a product strategy that can reverse this trend.

After some market research, and some sleepless nights, Steve decides to kick off a new initiative to put Old School back in the technology leadership position. It will involve substantial changes to the product, and if all goes well, allow Old School to leapfrog the competition. In order to present the plan to management, Steve needs to know how much it will cost to implement, and how quickly they can deliver the updated product.

Steve goes down the hall to Paula, the product manager, and asks her to prepare an estimate. Paula barely understands what Steve is asking for, but she proceeds to go to Darren, the development manager, and says that Steve needs an estimate by the end of the week to build the new features. This doesn’t really fit into the normal process at Old School, as there would typically be a detailed requirements analysis, followed by a written requirements document to be signed off. Darren is very uncomfortable with the request from Steve, and would prefer to stay on the two-year cycle to which they are accustomed.

Darren coordinates with the development team to estimate the work (which they believe will take about two years),and feeds that back to Paula, who delivers the message to Steve. Let’s listen in:

Paula: "Well, we talked about it and it will take two years to implement this new system, just like all of our releases."

Steve: "Unacceptable! We need to have this up and running by the end of the year to turn our finances around!"

Paula (hesitantly): "Well the developers usually pad their estimates, it’s probably more like a year and a half. If we hire more people, we could probably do it in a year."

Steve: "Great! I’ll tell the Board!"

Obviously Steve has made a timeframe commitment that the rest of the team hasn’t really bought into. But Steve has a legitimate business concern – he’s trying to turn the company around. Is the development team’s two year release cycle working for the business? Would Steve be pushing back so hard if he actually believed their estimates?

There are a few issues here. First, Darren and the team provided a questionable estimate in reaction to their uncertainty about the scope of the work. Paula, knowing that Darren traditionally pads his estimates, tries to be helpful by assuming that they can cut out six months automatically. Finally, Steve believes that if the development team just works harder and adds a few more people, it will take even less time to implement the system. It hasn’t even occurred to Steve to ask for less, because he doesn’t trust that the team will deliver everything he asks for anyway.

So the project is approved, and Darren and his team start with requirements analysis. They spend the first two months of the project trying to uncover the needs for the new features, working with Paula as needed. They finally produce a 30 page document for the requirements of the new system, and submit it to Steve for sign-off.

Steve attempts to read the document, but finds it somewhat complex and difficult to follow. He starts to nod off after the first 5 pages. Darren reminds Steve that without signoff, the team cannot begin work on the software specification, and the project will be delayed. Not wanting to be a bottleneck, Steve rushes through the rest of the requirements and signs the document, as it appears to capture the customer needs as far as he read.

Darren and company proceed to create a 100 page software specification that includes detailed use cases for the new features, complete with abstract diagrams of the major processes that the new features will support. There is no real signoff for this document, other than the QA team verifying that each use case traces to a requirement specified earlier. This document takes an additional 2 months to produce.

Now, Darren and the team of developers begin the implementation. Darren’s lead software architect decides the system should support a flexible message architecture that he has been reading about, and that they will need to build some frameworks to support this. The architect also recommends creating a Web services infrastructure, in case this becomes important to future customers. Darren agrees with his architect, and the team spends the next 2 months developing a robust messaging infrastructure. They run into a few problems, but produce a very good framework. They spend an additional month learning about Web services, and one more month building hooks into the system to support this activity.

Paula drops by to see how the work is progressing, and Darren tells her the progress to date. She’s a bit concerned that they haven’t actually started on any of the new features yet, with only four months remaining in the schedule. Darren assures her that the infrastructure work is important, and that they will need it for a future release anyway. Paula is still anxious, but has to take Darren’s word for it.

Two months later, the development team has put together a few screens, and started some of the database infrastructure work needed for the new features. Unfortunately, they realize that they will still miss the delivery date, even without any formal testing on the system. Darren and Paula relay the message to Steve, who predictably, gets very upset. Steve sets a new deadline for a month later, but he’s very unhappy, and wonders aloud if he should have outsourced this project.

A month later, things don’t look a lot better, but Darren whips the team down the home stretch. The final product is released with only a week of formal testing. Unfortunately, some of the features don’t actually work, and many are extremely bug-ridden. The team declares victory anyway, and collapses in a heap of empty cola cans, half-eaten Twinkies, and the remains of a two-week old pepperoni pizza.

Is Steve happy? Well, he’s enthusiastic for about a week, until the complaints start coming in from customers. Apparently, this is the worst quality release of any software product the company has ever delivered, and several of the customers are threatening to sue unless some of the core problems are fixed. In addition to the defects, the new features are implemented somewhat differently than Steve had envisioned, and don’t really address the needs of the customers. Steve is upset, and asks Darren why the development team built something so different than what he imagined. Darren argues that Steve had a chance to read and sign off on the requirements documents, and should have asked for any changes at that point. Neither Steve nor Darren is very happy.

Bonuses are eliminated, jobs are threatened, and the exhausted development team now has to try to understand and fix several hundred thousand lines of brittle, poorly tested, duplicated, and generally pasta-like source code. To make things more fun, the lead developer has just given notice so that he can go work for a more developer-friendly company.

This may be a somewhat extreme example, but in many of the projects I’ve been involved with over a 15-year period, I’ve seen exactly this type of scenario play out. The developers and the business stakeholders have an adversarial relationship instead of partnering. Management questions the competence and reliability of the team, and the team fears and distrusts management. Deadlines are missed, quality suffers, and morale is low. Not only is the project team unhappy, but their customers aren’t exactly singing them praises either.

There are many reasons why this project failed, some of which are more about the organization than about the development process. Old School has trust issues between the business stakeholders and the development team, which won’t be improved with this latest disaster. The development process is not setup to foster communication and understanding, focusing more on signoffs and contracts to avoid blame. I’ll talk about how some of these problems might have been avoided in the following sections.

.

Agile Software Development

So what exactly is agile software development? It is a collection of methods, principles, and practices that follow a common philosophy of rapid feedback, close communication, and respect for people. The Agile Manifesto (http://www.agilemanifesto.org), a sort of Declaration of Independence of the agile movement, describes some basic principles that agile methods follow. The following is a summary of the key elements:

• Deliver working software early and often


• Do the simplest thing that meets the needs of the business


• Embrace changes in requirements


• Maintain close communication between developers and business people


• Set a sustainable pace for developers, users, and sponsors


• Reflective improvement (look back at what worked, and what didn’t)

The most popular of these methods include Extreme Programming (http://www.extremeprogramming.org) and Scrum (http://www.controlchaos.com). Extreme Programming (XP) tends to focus on development practices, while Scrum offers more in terms of project management practices.

Elements of Agile Software Development

The essence of agile development methods is getting rapid feedback from customers by delivering working software frequently. They are said to be both iterative and incremental. While there are other iterative processes, such as Rational Unified Process (RUP), these tend to emphasize phases of development, just like traditional methods, only with several repetitions of those phases. An agile iterative process is characterized by a fixed time period to do the work in, without distinct phases. An incremental process delivers an end-to-end running, tested piece of functionality that is ready to deliver to customers.

Combining incremental delivery with an iterative process means that fully functional pieces of a project will be continuously delivered in well-defined iterations. To make this more useful, the project sponsor can choose which pieces are most important, so that they can be delivered first. It may turn out that other pieces become less important over time, or that new pieces emerge that weren’t obvious when the project started. A typical iteration is anywhere from two to four weeks in duration.

Using short iterations helps to minimize the complexity of the project. For example, traditional projects often track dependencies in great detail. Agile projects attempt to reduce coupling between features when user stories are written so that there are minimal dependencies that can be tracked informally by the development team. Also, the effort required to test the output of an iteration is substantially reduced, especially when combined with automated unit and acceptance tests.

A project feature is described in terms of a user story, which instead of a formal requirements specification, is a simple description of a piece of functionality from the customer’s perspective that has business value. As the development team needs clarification and detail on a user story, they have conversations with the customer to elicit more information. So initially, just enough detail is captured to allow the developers to estimate the effort required.

Agile methods use user stories and frequent conversations with a customer (XP recommends an on-site customer or proxy) in place of formal requirements documentation. Important details are instead captured with acceptance tests for each story. These acceptance tests are frequently automated so that the code doesn’t regress with changes.

To clarify this, let’s say you’re packing up your house to move to another one in a different city. Using traditional waterfall techniques, you would do an inventory of the house, weigh each item, contract a moving service, have them pack every room in every house, load everything on the truck, deliver it to your new house, and unpack it.

Let’s say that I want to move into the new house as soon as possible. The moving company I’m using tells me it will take two weeks to pack up the whole house, another two weeks before enough trucks become available to move the entire set of boxes, and several more days to unpack all of the trucks in my house. That’s nearly six weeks to do the job using traditional methods.

If I apply agile techniques to this problem, I’ll start by sitting down with the movers, and listing all of the rooms in the house. I’ll prioritize which rooms to pack and move first. I will probably start with the living room (it has a comfortable chair and big-screen television), then the bedroom (the chair is only comfortable for a few nights), followed by the bathroom and the kitchen (I don’t really cook anyway).

The movers then commit to a time period of a week during which they will be able to pack, transport, and deliver the living room. This is what agile methods call an iteration (or sprint in Scrum terminology), a fixed time period where specific, prioritized pieces of functionality will be implemented. At the end of the week, I’ll have a nice TV to watch and a chair to sit/sleep in. Not quite Nirvana, but it’s a start.

After the living room is installed, I meet with the movers again to decide what to move next. I originally thought the bedroom would be next, but I bought a new futon, and won’t really need the bed. So I tell the movers to bring the bathroom furnishings instead. I also remember that there’s an attic in the house that has my baseball card collection. I decide getting those safely moved is now a high priority. The movers tell me that they can do the bathroom and the attic for next week, but that will delay the kitchen. I’m fine with this, and we move forward.

In agile development, this kind of prioritization and negotiation – called iteration planning – is a key componentiteration planning. Notice that I’m in control of prioritizing what work gets done, and the movers are in control of estimating what they can do in the time frame of each iteration. If I want something additional, I can have it, but it probably means giving up something of lower priority. At no time will I tell the mover how to pack a box, what kind of tape or bubble-wrap to use, or the faster route to drive to the new house. That’s their specialty, not mine. Although this isn’t unique to agile methods, respect for people is an important value in agile.

In the end, the entire moving process may take a bit longer over all (depending on how much I change my mind), but this is offset by the flexibility and immediate benefits I get in terms of access to the new house. I may also decide not to move everything, which could actually save me money.

The situation is even better on a software project. Although there is some overhead for packaging, documenting, regression testing and deploying software to production, much of this can be automated. In fact, continuous integration is a common practice on agile teams (builds performed at least nightly, with automated tests verifying at least a baseline of code quality). These techniques are generally applicable for software development, but are critical to the implementation of agile methods in order to support frequent releases.

Also, in my house example, I’ve presumably already paid for everything in my current house, where on a new software project, there is no sunk cost yet. In other words, there are often requested features that are considered important early in a project that become irrelevant later. Traditional teams might spend three months working on features that aren’t needed, instead of being able to adjust to changing requirements by collecting more frequent feedback.

The Solution

Now let’s look at how the hypothetical project team I described earlier might apply agile techniques to their project. Let’s start by re-visiting each of the players and their goals.

Steve, the project sponsor, is driven by several factors: the desire to stay competitive in the marketplace, the desire to increase revenue, and of course, fear of failure. His history with the development team is that projects always take longer than promised. He also believes that he won’t get anything unless he sets a strict deadline and pressures the team.

Paula, the product manager, primarily is interested in enhancing the product to better serve the customer base. She would like to see a product that delights the customer. Naturally, she also is motivated to succeed in her job, which is measured by customer feedback, and ultimately, financial success in the marketplace. While she gets along well with the development team, she is sometimes frustrated by their fixation on technology issues.

Darren, the development manager, wants to protect his team from outside pressures. He also wants to deliver quality software in a timely manner. His history with the company tells him that unexpected requests often distract his team from delivering software, so he tends to inflate estimates to accommodate that.

The developers want to build high-quality software, using the latest technology if possible. They feel that they aren’t given enough time to do the job right, and this is demoralizing. They are also pretty far removed from the customer, so they don’t really understand how what they build benefits anyone.

In terms of agile software development, Steve is the project sponsor, and ultimately controls the funding for the project. Paula is a good proxy for the customer, since she frequently visits customer sites, and has a good understanding of the domain. Let’s see what happens when we inject a little agility into this team.

A Little Planning

To kick things off, Steve, Paula, and the entire development team meet to discuss the project. Steve outlines his strategic goals of staying competitive and his desire to make an impact within the next year. He turns things over to Paula to articulate the specific project goals.

Paula has written out a list of user stories that describe the required functionality in the new project. Darren asks her to rank the list of user stories by business value. For each user story, the team discusses what kind of effort it might require, and makes a note of the estimate. The estimates are given in terms of ideal days. An ideal day is the effort that a single full-time developer could give during a business day without any distractions, email, phone conversations or meetings. Depending on the team, this could translate into two or three calendar days per developer. For now, these estimates do not need to be completely accurate, they are just an educated guess so that prioritization can move forward. Once the team drills down to the detailed tasks, these estimates may be modified. The important thing is that Paula and Steve have information about relative effort to guide their prioritizing process.

The team agrees that a two-week iteration is probably enough time to produce something useful. With three developers, they determine that they can probably complete about 10 ideal days of work (10 calendar days for the iteration = 3.3 ideal days per developer, so that’s 3.3 ideal days x 3 developers = 10 ideal days).

Paula lists the first few stories in order of priority as follows:

• User Story 1: A user can add a reminder to his calendar


• User Story 2: A user can track his business contacts


• User Story 3: A user can see a weekly view of events from his calendar


• User Story 4: A user can see a daily view of events from his calendar

The team asks Paula for clarification on User Story 1. What information is needed for the reminder? How will it notify the user? Paula thinks for a minute, and explains that the user can specify how long before the event to send a reminder. She also explains that email will be the only notification mechanism. The team estimates five ideal days to complete the work.

They also notice that the work needed to do a daily view overlaps with the work to do a weekly view. So if they implement the daily view first, it will be easier to implement the weekly view. Paula agrees to the reordering.

They repeat this process for the remaining user stories, and come up with estimates as follows:

• User Story 1: 5 ideal days


• User Story 2: 10 ideal days


• User Story 3: 3 ideal days


• User Story 4: 2 ideal days

Since User Story 2 will take more effort than the developers can complete in the iteration, Paula instead picks User Stories 3 and 4, which will total five ideal days and round out the iteration. User Story 2 will be the first one implemented in the second iteration.

For now, the team decides to plan for just two iterations, as Paula hasn’t had a chance to consider all of the possible features yet. This is perfectly fine, since every two weeks, the team can re-prioritize as necessary. The iteration plan looks like this:

Iteration 1 (capacity: 10 ideal days)

• User Story 1: 5 ideal days


• User Story 3: 3 ideal days


• User Story 4: 2 ideal days

Iteration 2 (capacity: 10 ideal days)

• User Story 2: 10 ideal days

After the high-level planning meeting, Steve goes back to his office, and Paula and the development team break for lunch. When they come back, they decide to drill down on the first iteration.

For each story, the team tries to figure out how to approach the problem, and to break it into one or more tasks. For example, for User Story 1 (adding reminders to the calendar), the following tasks are discussed:

• Develop user interface for adding reminders


• Design a database schema for storing reminder information


• Implement scheduling engine that can poll for reminder notification dates


• Build e-mail notification component

The developers then sign up for tasks. The developer who signs up for a task can estimate the work for the task. Note that one of the potential challenges on an agile team is that only one person has a certain skill, and is therefore always needed for that kind of work (e.g. database development). In order to support the needs of the business, some agile teams use pair programming in order to transfer knowledge to other team members. Over time, a team will still have experts in certain areas, but everyone on the team should be familiar with a variety of areas. This allows anyone on the team to take on any task, perhaps needing occasional help from a specialist. It also mitigates the impact of any team member going on vacation or leaving the company.

At the end, Darren adds up the task estimates and compares the total to the initial story estimates. He also makes sure that no one has taken on more work than they can do during the iteration. Things look pretty good, and the team is ready to proceed.

A Little Implementation

The team starts working on the highest priority story, with two of the developers pairing on the database design part, and the third creating the reminder screen. They write unit tests as they go to catch any mistakes that they might introduce later. The team checks in code frequently, and their automated build system lets them know right away if the code builds and passes all of the unit tests. Unit testing is another practice that can be applied to any development effort. The main benefit of unit testing is to support the developers when they need to make major changes or refactorings. Since agile methods embrace change, but also require frequent releases, having a harness of unit tests makes it safe to create minimal implementations, secure in the knowledge that later changes won’t break the system without anyone knowing.

Continuous integration is another generally applicable development practice, but again, in an agile context, the frequent need to deliver working software requires that the team keep the system working at all times. The continuous build system notices code changes that are checked in, builds them in an independent environment, runs all of the unit tests, and reports any problems to the team immediately.

As the developers work on the user stories, they call Paula over to ask her for more clarification as they go. Paula decides to just move into a vacant office right next to the development team. She puts a bowl of candy on her desk to encourage the developers to come by and ask questions.

Every day the team meets for a quick status meeting known as a "daily standup meeting" (XP) or "daily scrum" (Scrum). Each team member answers the following questions:

• What did you do yesterday?


• What are you planning to do today?


• What obstacles are in your way?

Darren makes sure that the team follows these rules, but he doesn’t participate otherwise. Paula is invited to attend if she wants to, but is only also allowed to listen in. The idea of the meeting is to help the people doing the work communicate and have obstacles removed. Once an iteration has begun, the business is not allowed to change the contents or priorities, so comments from Paula are not really useful in the daily meeting.

At the end of the iteration, the team has several working, tested features completed. They call a brief meeting to walk through the features with Paula for feedback. Paula notices a few minor oddities with the user interface, which she adds to the feature backlog to discuss for the next iteration.

A Little Deployment

After a few iterations of these (about 3 months worth), Paula meets with Steve to discuss the progress so far, which has been steady and consistent. When Steve hears that the top 10 features have already been fully implemented and tested, he gets very excited.

Steve: "This is great! When can we announce the new release? I don’t want to wait until the end of the year if we’ve already got this much!"

Paula (gulping): "Well, I’ll talk to the team. They tell me that with this new agile approach, they can deploy the results of any iteration."

Steve: "Make it so!"

Paula tells Darren about Steve’s request. Darren would normally have started screaming at Paula, but he is confident that the team has developed complete, well-tested functionality, and they really can deploy it. The team agrees, and spends the next iteration (shortened to a week) on deployment instead of building new features.

The first customer, GigaBank, gets the new release, and is thrilled. GigaBank can’t believe that Old School has delivered something in just three months, and it solves their top production issue. GigaBank is so happy, it agrees to publish a case study in a major industry journal, and Old School’s sales skyrocket.

The development team gets a feeling of satisfaction to have actually delivered a complete, high-quality product in just 3 months, without working late nights and weekends. Developers are showered with praise, and Steve even takes them out to an expensive dinner to show his appreciation. Of course, he doesn’t share the huge bonus check he received after the successful board meeting.

Conclusion

Traditional software development processes are often poorly suited to competitive business environments. The long delivery times, poor communication, and lack of feedback can result in competitive disadvantage for the business. Certainly they can be made to work, but more often the result is similar to the hypothetical Old School project disaster I described earlier in this article.

Agile software development offers an approach that encourages rapid feedback through continuous delivery of business value. The use of short iterations and incremental development of working software gives businesses unprecedented ability to respond to changing conditions.

Development teams learn to work closely with the customer to work on the most important things first, and to make course corrections before investing months into a technological dead end. Instead of waiting for a year or more to get feedback from their customers, they can show constant progress. This has the effect of reducing the amount of pressure on the team, since customers are confident that they will eventually get everything they need, and that their most important issues are being worked on.

Agile software development is not a magic bullet. Software projects are planned, designed, implemented and tested by real people with all of their strengths and weaknesses. Agile practices are designed simply to help people do what they do best, while embracing the unpredictable nature of software projects.