Our thoughts on this subject come from practical experience. Companies who come to Scio with their projects often come with a multi-megabyte PDF, UML diagrams, and a list of specifications. “Give us a firm, fixed price for getting this project done by June 2nd at 2pm Eastern,” they say.

Their basic idea is:

• Software projects have a much less than enviable record of finishing over budget and way over the estimated completion date – we’ll set those so they can’t creep.
• Software outsourcing is risky so we’ll limit our risk by agreeing to a cost and timeframe we can live with and possibly tag onto some event. “Shoot for the June trade show so we have a shiny new product to sell,” Marketing begs.
• We don’t have the resources to do the project in house, but we don’t trust any outsourcing group – so we’ll rope them in with a fixed fee and time and put all the risk on them.
• We know perfectly well what our product needs to be. If we don’t nail this down, we won’t get what we need for the price we can afford.

The result of this thinking generally speaking is:

Flaming Disaster!

Why? Their basic instinct wasn’t wrong. Software projects do fail to meet their targets with astonishing regularity. They were just trying to limit their exposure. What is happening?

There are five intrinsically linked factors in estimating software product development projects:

• The Total Elapsed Time expected to produce the specified product.
• The Effort required to produce the product.
• The Cost the client expects to expend.
• The Resources required for the project – their skills and availability.
• The Specifications for the product; the features, functionality and user experience.

 

In general terms, what clients are trying to do is set a “target.” In project management, the general assumption is you can set any one of the five factors as a target for a project, but when you do, you need to let the other four float to where they need to go to reach the target. So if you set cost, you need to vary time, specifications, effort and/or resources to reach a mix that will achieve the project goals within the target cost.

Instead, clients set two or more factors in an attempt to “hold the line” on all the other factors. They spent a lot of time on those specifications. They need them all!

But in fact, setting more than one factor as fixed creates an almost impossible tension among the remaining factors that almost assures the project will fail to meet its goals. There are no levers left to control the project! It starts out with the best of intentions, but with two or more factors fixed, any change in circumstances during the project creates an imbalance that cannot be corrected with the remaining factors.

Why does this happen? Stepping away from our example of setting time and cost as the fixed factors – think about each of the factors individually and the impact they have on the project:

Time

The elapsed project time from start to finish is always different than the total effort applied. Time is measured by a calendar start to finish. Conversely, the effort is the sum of all the time expended on a project by the assigned resources. Total time is never equal to the total effort unless only one resource is assigned, full time.

Software development projects rarely finish on time. Unplanned specification changes, unexpected risks, and resource changes always build up over time and eventually result in a project that is both over budget and beyond the allocated time.

Time to completion can only be estimated and controlled well over short periods. As the time period considered in an estimate increases, the accuracy begins to degrade because of variations in the expected effort, the depth and complexity of the specifications involved, the skills and availability of the resources required and the limitations an assumed total cost puts on the project.

It should also be understood that time to project completion is rarely scoped as a direct result of estimating the effort required. More often, “artificial completion dates” evolve from a point in a product marketing plan, the current product position, and/or customer demands. When this happens, there is usually some consideration of project scope but is rarely enough to address the situation that arises from not first doing a straight-forward evaluation of the effort required to complete the specified product.

Effort

The accurate estimation of effort is key to successful software project costing and setting a realistic expected time to completion. In practice however, the amount of effort required to actually produce each bit of application functionality always varies from estimates. The more detailed and contingency bound the estimate becomes, the more likely it is to be wrong. Because of this, past experience and general effort assumptions are used across a project estimation, in the belief that in the final outcome everything will average out. Of course, the reverse is also true; averages can never address all risks in an individual project. So, while averages are a practical approach to project estimation, they cannot yield a project quote that can be fixed to a specific figure without risk.

In this situation, risk buffers for variations in specifications and resources are recommended for effort estimation, especially in Agile development methodologies where development iterations are “timeboxed.” Timeboxing iterations means variations in the effort will generally cause functionality to be pushed ahead to the next iteration and a “snowball effect” can be produced where the amount of effort required for each iteration increases incrementally beyond estimates over time. If buffers were used, more projects would reach their estimates, but in the drive to reach a more competitive price, they are rarely employed when using assumed effort to arrive at a fixed cost. This results in a very narrow margin for error in effort estimation.

In addition, the amount of time required to reach project completion is not directly related to the number of resources available concurrently. Determining effort depends on an experienced assessment of an efficient team size for the project and the methodologies used. Increasing the number of resources and concurrent tasks beyond a certain point increases coordination and communication overhead exponentially. Increased team size tends to increase errors, oversight, and testing cycles without a cost-effective increase in total output.

Estimates of effort required tend to be assessed from a straightforward analysis of specifications. During projects, the actual effort required frequently increases beyond estimates because of “fixes” required to bridge the gap between specifications and the product as realized in development. In addition, the elapsed time required for QA by the client team is often underestimated and can result in either idling development or moving ahead with incorrect assumptions and subsequent rework.

Cost

Software development projects almost never finish under their expected cost from the point of view of clients. A few finishes at the client’s target cost, but generally only at the expense of other project factors. As a result, when projects do cost what was originally expected, the product is often a failure from an end-user point of view.

For clients, the target project cost is generally a function of:

• Expected product price and the desired return on investment that could be produced by an estimated number of paying customers in a reasonable period of time. In other words, a string of dependencies that may have little basis in the final analysis.
• Available funds and cash flow limitations.
• Experience with “similar projects” – However, only rarely do estimates based this way actually works out to be similar in the effort required.

Target cost is never or only rarely based on:

• The steps and effort actually required scope and develop a product that is a successful market fit.
• Small, incremental steps that can be estimated with a reasonable chance of success.

Specifications

Specifications are almost always assumed to be a known and set factor in fixed cost projects. They are used as the basis for effort estimation and effort estimation ultimately determines the quoted cost. Clients generally have a basic expectation that their specifications do not need to be varied from substantially to produce the desired product at the specified cost. Clients often expend great amounts of time producing specifications for bid to assure they will be complete and assumed to be fixed. But in fact, not assuming specifications will need to be varied as over the course of a project to meet fixed cost results in a continuous tension between the effort required, the scope remaining and the time remaining on the project clock.

Most fixed cost projects have intentionally limited options to change scope. Instead, limiting scope change by not providing workable options increases the risk the project will not reach the desired goals when the actual product is assessed by end-users.

Software development requirements can never be complete enough or communicated well enough to ensure understanding and success. Errors in interpretation, over-broad and over-complex specifications result in many “fixes” that are not related to actual code errors by the development team. These fixes are actually elaborations or “clarifications” of project specifications, but in most projects, they are assumed to be “bug fixes” in the process of development, testing and QA. In practice, this means the actual functionality works as specified but the implementation is not as desired by the client. Fixes of this type generally add to effort and resource allocations without the assumption they should impact specifications, time or cost.

Software development project requirements are by their nature improved by the discovery on the part of both the development team and the client team during analysis and development.

In the course of normal work, discovery exposes:

• More depth than expected (scope creep).
• Different aims and approaches from the client and end-user feedback or unexpected insight from seeing the product as it develops.
• Technical limitations or alternative approaches that change requirements, the effort and time required.
• In most software development projects, there are no assumptions or procedures to handle specification discovery and subsequent changes. This results in many variations from project estimates and is a significant factor in project overruns.

Resources

Resource management is a function of having the right skills available when needed for a specific task in a project. With limited resources and funds, this is a difficult task for software development companies.

Both internally and externally, software development companies have an ongoing need to balance new projects against support, maintenance, and enhancement of existing applications. Companies need to decide the level of investment they will put into new technology. Using time from existing work to move to new technology skills is a difficult and expensive proposition. Recruiting for internal resources is a long, expensive process that often fails to yield dependable, trained resources in the long run.

These factors are the leading reasons clients consider outsourcing. But they are also a factor in outsourced projects themselves because, at some level, the client team becomes involved directly with the outsourced team and the results of team resource management. The management of new software development projects is difficult by itself. Because of the time and risks involved in recruiting resources with appropriate skills and knowledge, client project/product managers often don’t have a good understanding of the technology and limitations in the project they are managing.

In this situation, outsourcing software development often leads to a confrontational relationship where the client team feels they have lost control and don’t understand the choices the outsourced development team has made or what effort is being applied to produce deliverables. They don’t understand that the estimation for time to completion was figured against assumed effort but the accuracy of that assumption varies according to specification clarity, resource skills, and availability.

In Summary

Variations in the five factors during a software development project leads to:

• Defensive reactions to clarifications and changes between the client and the development team.
• Situations, where the actual effort in the given time varied depending on specification accuracy and resource skills and availability, lead to confrontations. When the time to completion is figured for a fixed cost, it is generally figured against the assumed effort. Without assumptions for what controls are available to deal with variation, the confrontation continues to simmer throughout the life of the project.
• Lost opportunities for a partnership-like relationship of shared risk and reward.

The solution could be as simple as not setting more than one factor as fixed, but in practice that is hard to do for many projects. What is really needed is a consultive framework for communication and decision making that is informed by real-time reporting during the project and the collaborative resolution of issues to reach the client’s goals. It’s easy to say, but it takes understanding, planning, and agreement to accomplish.

We’re constantly working on this paradigm every day – it’s challenging and rewarding. What’s your experience? How do you hold the line? What controls do you have realistically? Have you recognized the five factors in your project estimation process formally? I’d love to hear your thoughts…