The Art and Science of reliable software estimation

The relationship between I.T. and the enterprise is a difficult one. Currently, internal I.T. is in a transition phase from internal service provider to manager of I.T. services either delivered by suppliers or as commodities, usually online.

To put this in perspective, let us consider the common trend towards outsourcing throughout the enterprise. There was a point in the past where pretty much every service that a large business required was done in-house. Facilities management departments would retain large teams of skilled staff. Over time this has changed to the management of external suppliers by a small team that is more about procurement and project management than about understanding in-depth, the skills necessary to deliver those services.

This has been a gradual but inexorable process operating over the last thirty years or so, during which pretty much all of these services have been either outsourced or commoditised. This process, with which I was heavily involved in during the early part of my career, was also largely painless.

It is a cause of frustration and confusion at senior level, for managers that have been through this with other parts of the business, that the process around the outsourcing and commoditisation of I.T. is proving so difficult. The reason for this is the critical nature of the processes over which I.T. presides. There is nothing more important to a business than its business processes.

At the heart of the pain in the relationship between I.T. and the enterprise is the perception that I.T. doesn't deliver on its promises. If we look at the number of software projects that fail to deliver it is quite difficult to disagree with the validity of that perception.   http://spectrum.ieee.org/computing/software/why-software-fails

Inaccurate software estimation has fundamentally undermined the relationship between I.T. and the business and once trust has been lost there is a long road to recover it.

Why do software requirements, seemingly inevitably, overrun on the initial estimates?

There are two main reasons, one is technical and related to complex systems and the other is human.

We don't know what we don't know.
In any large estimation process there are a number of imponderables which can have a significant impact once dependencies between the various components of the project have been played out, but deep at the heart of failed software projects are those grim moments that consist of the dawning realisation that some piece of software, technology or process will simply not do its job or behave as expected.

During project initiation, questions are asked relating to the third-party and legacy components of the project and the answers that are received guide the project team in its architectural and estimation decisions.

When we ask questions of suppliers, owners of legacy code or the third-parties with whom we are to integrate, our level of assumption about likely outcomes is necessarily higher than with our own code. The level of complexity is usually so great that it is impossible to know about all of these components and accurately identify the risks related to coding against them. Drafting in expertise is not an answer as, should one retain an Expert, it is not unusual for the said Expert to respond to direct questions in an enigmatic manner leaving the questioner no wiser than before the question was asked. There are numerous good reasons for this; from a failure of the Expert to understand the business requirements to an understandable reluctance to commit to decisions about a system that is in design and of course, potentially a desire to be retained for a longer period.

In summary, the domain of expertise of the architect is unlikely to span all of the technologies involved in sufficient detail that he or she will be able to assess all of the risks fully. This will inevitably result in the existence of the unknown unknowns which will then be encountered during the project.

This doesn't just apply to the architects but also to the developers as they need to code on a day to day basis against the components, interfaces or legacy systems, the behaviour of which is obscure to them. Hence a two week task is as likely to overrun as a two year project.

The need to please
Most people spend most of their time in a process of gaining approval. This is how humanity operates. In a business, staff seek approval from each other and from their managers, managers seek approval from their managers. Everyone seeks approval from the CEO who is bound to the investor's wishes and so on ad infinitum. There are also other drivers for the underestimation syndrome such as insecurity contributing to a feeling that we might take longer than others.

What this means in practice is that if you ask someone in a competitive environment how long a task will take they are most likely to underestimate and choose the least possible time that they think could be taken to complete.

The Science
Any process that delivers accurate estimation must take into account the above two issues; optimistic estimates and the unknown unknowns. That would seem a pretty tall order on the face of it as these are clearly unpredictable but there is a methodology that will improve the accuracy of estimates which is called Evidence Based Scheduling.

Evidence based scheduling
The only way we can possibly capture the effects of skewed estimates and unknown unknowns is historically. There is simply no way of factoring in influences like these as we proceed. Evidence based scheduling operates by taking past estimates and comparing the estimated time with the actual time taken. This gives us a ratio between the two which, free from other influences, will result in a value which we can use to multiply an estimate in future to derive a more accurate estimate. In all, for the average team, this transition will take three or four months to deliver a consistent estimation process.

The prerequisites for evidence based scheduling are;
• A change management system that allows estimates and actual times to be captured for each task.
• A business process that ensures the estimates and actuals are entered for each task.
• A way of excluding exceptions to the process which includes but is not limited to; pilots, documentation tasks, prototypes, completion of functional specs and so-on.


How it feels working in an Evidence Based Scheduling process

From the perspective of the business user;
• Either brief a business analyst or prepare the requirements using a use case template.
• Present the requirements for estimation.
• Accept or reject the estimate.
• If unacceptable either review the requirements or move on.
• If acceptable, accept the schedule.
• If the schedule is unacceptable ask for the task to be re-prioritised.
• If the schedule is acceptable receive delivery of the requirements on the due date.


From the perspective of development;
• Receive requirements for estimation.
• If the requirements are unclear, request clarification or a graphic.
• If the requirements are clear and a pilot or prototype needs to be done, estimate the time necessary to perform the pilot or prototype. (This is known as a spike).
• If spike time is acceptable, complete spike.
• Deliver estimate.
• If acceptable, implement the changes as per the priorities on the schedule and deliver to Q.A.


The Art
Convincing development that evidence based scheduling is in their interests is not usually a challenge once the benefits of reliable estimates and consistent requirements have been articulated, though it is never possible to assume that just because something makes sense that it will be easily accepted.

A process must be in place within the business to support evidence based scheduling which is why it has failed to become a universally accepted practice. The supporting process needs to deliver consistent and manageable requirements. Variations in the quality of the requirements can undo any stability in the estimation process. Reliable estimates rely on reliable requirements.

To be consistent and manageable, requirements must be described in a consistent format, ideally a use case, and also with adequate detail. The proof of this is that there are no holes in the requirements and that they are comprehensible

To be manageable a requirement should not consist of any individual task that takes longer than four days to code. Longer requirements must be revised into a collection of tasks of under four days and prioritised. The reason for this is that the longer the task the more chance that some detail of the requirements has not been considered and there is a hole that will extend the time taken. Any large task can be split into smaller tasks. If that is impossible it can only be because the level of detail is unavailable and so it will not been thought through. In this case, the author of the requirements must go back to the drawing board.

Implementing this entire process is difficult because it describes the relationship between the business and development and that relationship usually consists of something other than finely grained, consistently presented requirements. Change will usually be necessary and as we all know, instituting change is the difficult part.

If I'm in marketing, my job is to devise and implement sales strategies. I may have a requirement which I need to be developed. I want to get this done but I don't want involvement in the technical details to limit my ability to do the thing for which I am employed, salaried and possibly also commissioned. If I am expected to spend hours in meetings discussing the nuts and bolts of what I need from development then I can't get on with my job which will frustrate me and possibly limit my effectiveness thereby impacting the business.

The results of me not contributing may however, play havoc with the schedule. What I must do in order to ensure that I get what I want when I need it from development, is to commit to the requirements process to ensure that an accurate estimate can be produced. In other words I need to buy in. I need to have the case described for evidence based scheduling and I need to be able to see how important it is for me, for development and the company that I engage in order to get what I need when I need it. I then need encouragement to continue and improve how I articulate my needs.

Delivering evidence based scheduling is a tall order that requires buy-in from the business users, development and senior management. It needs a champion of sufficient commitment, patience and gravitas to carry all of those parties through the process when things become difficult and who can enable the enterprise to realise the benefits of a new relationship built on a foundation of reliability and mutual trust.