The Art and Science of Software Estimation - Part 3: Yin & Yang

This is a four part article; I’ve divided this article into smaller ~4 min posts to make easier to read and follow up:

• Part 1: Anthropology of Time Estimates
• Part 2: The Nature of Software
• Part 3: Yin & Yang
• Part 4: Estimating Software

In Part 2: The Nature of Software we dove into the nature of software to identify the reasons behind the inherent difficulty in estimating software development.

In this post, we look into deadlines and projects, and the science behind both that could greatly improve our skills of predicting the future.

Deadlines: Yin & Yang

If we suck at it, why do we estimate software development at all? Well, one word; Deadlines!

Most software engineers hate that word -at least in my experience-, it’s usually associated with long nights and stressful weekends, but even though most would rather work on their state of the art piece of code without the pressure of time looming over their heads, Deadlines are a necessary evil.

A necessary evil

Deadlines have a very important role, they give us humans a sense of closure and purpose, they align teams and organisations, and they push us to make decisions even when we don’t want to.

You’ve probably worked on that one project left to our own devices, that internal project with no deadline where we just have the Time and Money to perfect everything, you know that project; its the one you ended up scrapping and never published.

As bad as they are, we cannot conduct business without deadlines, it would be chaos and if we’ve learned something from our own human nature, nothing would be otherwise get done.

Deadlines can be your friends

But what if I told you that Deadlines can exist without the long nights and stressful weekend? The one rule we need to follow is to properly plan.

Setting deadlines is easy, but sticking to them is hard; as we’ve seen in Part 1 we are flawed in measuring effort in relation to time, so simply slapping a deadline on that project is probably not going to work.

Instead, we should approach setting deadlines with a bit of analysis and planning, and this is where the art comes in, figuring out that balance of time, quality and cost that would yield the most success.

Zen

The Golden Triangle

The sought after balance is usually represented using the Golden Triangle; a very known figure in project management where each vertex of an equilateral triangle represents one of the three variables affecting the project:

You are only allowed to adjust one of the three vertices.

With the advent of the Agile methodology, the triangle has been revised to include scope as a variable and quality as an outcome of adjusting the variables:

Balancing those variables is what most seasoned project managers are after, its an art and a science, and it can make all the difference determining whether a deadline sucks or not.

Accuracy vs Precision

Another distinction we often fail to make, is the distinction between Accuracy and Precision.

Accuracy is the degree of closeness of a measurement to the true value of the quantity being measured, while Percision is the measure of reliability and consistency of the value being measured.

In layman’s terms, when we say a value is precise, we are saying that this value is 100% accurate, however, when we say a value is accurate, its expected that we provide a degree of distance from the precise true value.

Good deadlines are accurate, and rarely precise.

The Undeadline

No matter what you try, its definitely hard to convince people with ranged deadlines, they will either completely dismiss your arguments, or simply take your later date in the range as the deadline.

You might be thinking, what’s wrong with having the later date as the deadline? Well remember, we humans have no control over the passage of time, and we fail miserably at predicting that which we don’t know, so you’d be forgiven if you succumb to our optimistic nature.

As we time passes on, given we are hard at work trying to meet the deadline, our certainty increases (remember the four stages pyramid), and now we have a better prediction whether we’ll be early to our deadline, meet it, or miss it completely, giving us a shifting line instead of a static one, this line I call the Undeadline.

This phenomena can be seen and usually represented by the cone of uncertainty, a nice visualization showing how the variance of the probability that we are correct or mistaken in our assumptions decreases with the passage of time:

So early on in our projects, its a mistake to assume we have a Deadline, instead we have an Undeadline that is continuously shifting back and forth until the real deadline reveals it self.

In statistics, this process of discovering the real deadline is called regression analysis, and is a very useful tool to ensure that we align expectations as we progress through the project, it encourages transparency and thus trust.

Conclusion

We’ve observed the nature of deadlines and projects, and how our ability to set deadlines significantly improve as we invest more effort into the project and approach that initial deadline.

In Part 4: Estimating Software we talk about the process of estimation and how we can decrease that error variance by following a few simple rules.