David Greenlees

Disclaimer – There is likely nothing new in the following post, and I’ll likely rehash SEVERAL points that have been made hundreds of times… but I need to get this out of my system.

So I was recently informed of a manager who LOVES stats (shocking, I know). I emphasise the LOVES, as I was told that there was a huge amount of passion that went along with this information. In fact, the LOVE was so great it even spread to spreadsheets… so even the home of said stats was included in the LOVE. Now that’s passion. I’m not going to question the passion… that is important. I am however going to question stats.

Now to give you some context, these are the common stats that get sought… namely number of test cases and percentage of those test cases that have been completed. There are valuable stats out there in the wild, but these are not part of that group.

I want to look at a simple example…

We have three testers:

Han

Luke

Chewy

And the following stats:

Test cases outstanding	50
Test cases complete	50
Total test cases	100

What information can you gleam from the above stats? I’ll let you in on a secret… the most you can gleam is that there are 50 test cases outstanding, 50 test cases completed, and there is a total of 100 test cases. And even then, what does it all mean? Who knows!

It would be a common mistake to say that as the stats show 50% complete, testing must be half way through. If you were to ask someone to plot a point in a timeline as to where the testing is at they would likely plot that point dead centre. So you’ll now have management telling even more management that testing has taken two weeks so far, and you’re halfway through, so there is two weeks left until you’re finished.

Let’s stick with the timeline theme and have a look as some of the things that happened in those first two weeks while 50 test cases we’re being completed:

Han was called away on a ‘cargo’ run and therefore was absent from the team for 3 days.
Luke did all that he could to make up for this by coming in on the weekends, but only managed 2 days instead of 3.
Chewy was there for all 10 working days, but spent most of the time moaning and groaning instead of working.
They all agreed that it would be a great idea to execute the quick and easy test cases first so that the manager would not only LOVE stats in general, but would also LOVE their stats.
To help with the above, and knowing that he had been absent for 3 days, Han decided to working a few extra hours each day – or just when he could be bothered.
Because Luke had worked so hard on the weekends he just couldn’t do a full day on the final working day; he was simply too exhausted.
They raised a bug at the beginning of week 2 which blocked 2 full features from being tested or 2 whole days.
Oh, and they also had a developer leave the team at the end of week 1 so had an extended lunch break to farewell her; Leia was her name. Lovely lass that was affectionately known as ‘Princess’.
Etc, etc, etc.

This is hardly a minute by minute blow of ALL that happened during the first two weeks of testing, but you get the idea.

So should the manager still think that we are half way through testing? It is reasonable to assume that all of the above will also happen in the final two weeks of testing? It sounds as though the test cases may be more complex and time consuming, the loss of ‘Princess’ Leia will no doubt have an impact on bug turn around times, and Chewy may even jump ship if he’s so disgruntled! There is a dev shop down the road that are looking for testers; they’re called Empire Development and he may be in with a shot given his extensive knowledge of his current employer, Rebellion.

If the manager has a crystal ball then they may be able to plot the testing against the timeline a little easier, but alas… there is no such thing. So what do the stats tell you… still nothing.

There could be ways to make these stats valuable to somebody, but how much work would be required? Tur manager could look at every test case and estimate how long each one is likely to take; however I dare say they would spend more time trying to build the value in the stats than it would take to just finish the testing.

Stats like these are dangerous and can back you into a very nasty corner. Don’t hide behind these stats unless you know exactly what they mean, and can tell their story… in great detail.

Ahhh, now that feels better. Thank you for allowing me to get that off my chest. I know it’s a very simple example… so perhaps you can understand why it’s so frustrating that people still don’t get it!