The Art and Science of Software Estimation - Part 1: Anthropology of time estimates

This is a four part article; I’ve divided this article into smaller ~4 min posts to make easier to read and follow up:

• Part 1: Anthropology of Time Estimates
• Part 2: The Nature of Software
• Part 3: Yin & Yang
• Part 4: Estimating Software

Estimating software development is one of the most underrated skills in the Software Engineering industry; we are often more focused on technical competency that we fail to train our estimation muscle causing us to tip the scale to the side of gut feeling and intuition over logical analysis and sound scientific method.

Don’t get me wrong, there is a bit of art when it comes to estimating software development effort and time, however we often think the art is in time and rarely do we consider effort.

In this series, I jot down my thoughts on estimating software based on my experience of what works and what often fails.

Anthropology of time estimates

Science measures in relation

Its important to study how we humans interact with the universe around us; we exist in time and space, those are our perceivable dimensions and we define our existence and the existence of other matter in relation to those dimensions.

Think about it, our notion of time is relative to the Earth’s rotation around its axis, and its rotation around the sun, a metre is defined by the length of the path travelled by light in vacuum in a three hundredth of a second, and a kilogram is defined relative to Planck constant.

Every single measurement we have is defined in relation to a universal constant that we’re certain of and have no control over, yet, even with such knowledge we consistently make the mistake of measuring in absolutes, or better yet, predicting unknowns.

Our ability to measure required effort is flawed when it comes to doing something we have no previous experience doing before; we love giving absolute time as a measure of unknown effort, failing to account for the universal tendency towards entropy, the inherent uncertainty of time.

Inflated sense of competence

Another constraint to our abilities is competence; we humans suck at measuring our own competence and mastery! It has been proven again and again that we simply can’t fathom the sheer vastness of the universe, and even though we have empirical data to prove it, we rarely account for our incompetence.

One of the most popular observations of that limitation is the Dunning-Kruger effect; a cognitive bias where we size our abilities much greater than what they truly are.

The effect was first observed in a 1999 study and later confirmed through a subsequent series of studies, however, we’ve had other precursors for it long before it was scientifically studied and formalized.

Prior to the 1999 study by Kruger and Dunning, Martin M. Broadwell, a management trainer, described a model of competency in 1969 and called it “The four levels of teaching”, over the years many peer-reviewed papers have examined the model and incorporated it into their studies, and it’s now known as “The Four Stages of Competence”¹.

As our experience and knowledge increases, our ability to make intuitive decisions and assumptions increases, a fact further illustrated by another chart compiled out of the findings by Kruger and Dunning²:

Time has no king

One more anthropological observation is the myth of time control; we often make wrong assumptions about our ability to control the passage of time and how we perceive it.

Adding together our inflated confidence at competence and our tendency to measure effort in terms of absolute time, we tend to falsely predict output with an increasing error rate as the time frame of prediction increases.

We often assume that the further away the date/time, the more control we have over the output and outcome of our efforts, and thus the more optimistic we are about our predictions.

But just as we have no control over the vastness of the universe; we cannot create a new universe or stop its expansion, we also cannot create new time, or stop its passage.

Conclusion

We’ve examined how we humans measure values, assess our own competence and experience time. In Part 2: The Nature of Software we look at the nature of software and why its hard to estimate.