The proper estimation of effort required on a project is a controversial topic that has received much attention in the industry.
Many useful and innovative methods have been developed and described by my colleagues. Each method has unique advantages, so I find it difficult to recommend one method over another. This discussion is beyond the scope of this book. A number of useful estimation tools are available.
I encourage any organization bidding on software projects to invest in acquiring one of these tools and in training on estimation methodologies as well. Despite these methods and tools, one of the major impediments to estimation is human behavior. Only rarely does any estimate turn out to be exactly correct.
The actual experience usually ends up producing higher or lower numbers than the estimated number. For many software projects, the sad fact is that actual experience ends up producing far higher numbers than estimates, for both schedule and budget.
I believe that the primary causes are as follows:
The software industry and project management
The software industry as a whole is still very immature compared to other industries. Each company typically has its own development process and personnel competency level. There are wide variations from company to company and project to project. There is no consistency of methods from one company to another. If an outsourcing organization receives many bids for the same project, this does not necessarily mean that the lowest bidder is better at building software systems and can build them successfully for a lower cost.
It could mean that the low bidder does not have a complete understanding of what is involved in building the project, or perhaps it cut corners to produce a lower bid. Likewise, a company that bids high is not necessarily less competent at building software. It could mean that it has a more complete understanding of the tasks and risks and adheres to a more disciplined process that produces a more predictable result, so therefore it costs more. In other words, software is not a commodity.
Software projects that are put out for competitive bid receive estimates that do not reflect reality. The estimates are produced in an artificial environment where the emphasis is on producing a winning bid. This means that there is significant pressure for contractors to produce estimates that are on the low side. The procurement models described in this chapter help alleviate the problems created by estimates produced under competitive pressure. The estimates can be produced collaboratively in a better environment where the contractor knows it has the work, provided it continues to perform well.
To judge an estimate’s viability, I often use the “50% rule.” A proper estimate has a 50% likelihood of being too high and a 50% likelihood of being too low. If the organization can honestly make this statement about an estimate, it’s probably a good estimate. For example, if someone estimates that it will take 10 months to complete a project, I often ask under what conditions they can finish in less than 10 months.
If the response is “None,” I know the estimate is not viable. Many estimates produced under competitive pressure suffer from this problem. No wonder software projects are so frequently late. The estimates assume a perfect world.
Selecting a Contractor Proficient in the RUP
Suppose you are a manager in an outsourcing organization, and you have read enough about the RUP to believe that it is a better process for developing software. How do you select a contractor that is fluent in the RUP? How can you be sure the contractor really follows the best practices? Consider these points when evaluating contractors:
Look for contractors that have established a partnership with IBM Rational. One way to do this is to attend the annual IBM Rational Software Development Conferences. Many partners of IBM Rational present papers regularly at these conferences on a variety of topics. Some also have booths at the exhibit hall where you can speak to representatives of the company.
Find out if the contractor certifies its engineers. IBM Rational partners can obtain many certifications for their engineers. Certainly, certification in the RUP is valuable, and also for the supporting tools.In addition, ask the contractor questions about previous projects on which it has applied the RUP. Some of the suggestions in the following list do not necessarily have right or wrong answers, but they can provide insight into the contractor’s philosophy regarding the RUP:
- How do you tailor the RUP for individual projects? What factors do you consider when tailoring the Relational Unified Process (RUP) for a given project?
- Give examples of projects in which major problems were averted as a result of uncovering risk early through iterative development.
- How long are typical iterations on a project?
How do you identify the appropriate architecture for a project? What factors do you consider, and how do you prove that the architecture is viable?• What process do you use to elicit and manage requirements? How are changes to the requirements managed? How do you manage changes in scope to the project requirements?
Lessons Learned for Outsourcing Organizations
Outsourcing organizations can follow a number of practices to help their projects run more smoothly:
When reviewing proposals from bidders, examine assumptions very closely, particularly for Firm Fixed Price projects contracts. Evaluate the assumptions and determine how they will affect the project and how they probably will come into play. Ask the contractor what could happen if the assumptions are incorrect or fail to occur.
Question estimates closely. Ask the contractor under what circumstances the estimate may be invalid. Ask the contractor how likely it is that it could favorably beat the estimated amounts.
On Firm Fixed Price contracts, be flexible and willing to negotiate functionality so that the contractor can meet budget and schedule constraints.
Stay involved with the project. If possible or practical, monitor trends in requirements, change requests, and so on.
Be aware that traditional contracting practices may be in conflict with modern iterative development processes such as the RUP. For example, specifying that all requirements, documentation, and so on must be completed before any coding begins is contrary to iterative lifecycles. Similarly, requiring all development to be completed before testing begins suggests that a Waterfall lifecycle be employed.
Consider having an RFP reviewed by someone fluent in the RUP before releasing it.
Encourage the contractor to demonstrate releases made at selected iterations during the project lifecycle. This is an opportunity to observe progress on the actual product instead of through status reports and Earned Value charts.
Consider applying one of the two progressive procurement models described in this chapter.
Lessons Learned for Contractors
Here, I discuss lessons learned relating to contracts and contracting:
When bidding on projects, try to involve selected individual developers and other technical personnel deployed on other project teams. They can supply useful lessons learned based on other relevant projects from your organization. In addition, they may be in a better position to determine whether the proposed cost and schedule are reasonable for the functionality requested in the Request for Proposal (RFP).
When producing estimates for the project, have at least two people estimate the time and resources for the project independently. If the estimates vary widely, determine why.
Consider providing a probability figure with the estimate. Applying the “50% rule” described earlier produces a valid estimate. But what if you want to bid a more aggressive estimate to win the business? In other words, some contractors bid low knowing that they will take a loss on one project so that they can win business with a new customer. In that case, it may be worthwhile to produce two estimates.
The first estimate follows the “50% rule,” and the contractor gives the customer the second estimate to win the bid. The difference between these two estimates is the estimated exposure the contractor will have to absorb if the more aggressive estimate doesn’t work out. The contractor’s business organization needs to know this information to prepare for a possible overrun.
Consider computing and tracking Earned Value to help determine if the project continues to focus on the right activities as defined by the contract. Be aware that this requires tracking staff hours at a lower level of granularity than the level to which you may be accustomed.
The past several years have seen great progress in software development methodologies and processes. The next wave of progress will come from contractual mechanisms that set the right environment in which to practice these modern processes.
The next chapter provides guidance to those who manage an outsourcing organization. If you are such a manager and you are managing your first outsourced project, the next chapter helps you define important roles for people in your organization. The people in these roles work with the contractor to help the project be successful.