Recently Matt Heusser (that’s a new site BTW, so check it!) laid down a challenge of sorts. Disclaimer first – This is not a post to accept that challenge. I won’t be coming up with a new “good [control] metric.”
The reason I’m posting is due to a coincidence in timing. The day that I read Matt’s great post I received from news, the origins of which shall remain nameless. The news kinda went like this (numbers are not real but representative)…
Metric = Outstadning known defects at time of release.
- Two releases ago – 300+
- One release ago – 100+
- Last release – Less than 50
- Target for next release – ZERO
Wow, this is great news… or is it?
When I saw this news my instant reaction was to ask “Why?” Why are these numbers declining at such an impressive rate? I was pleased with that reaction as it highlighted to me just how much I’ve learnt over the past few years. Not too long ago I would have rejoiced along with the news reporter.
One of my first thoughts wasn’t that the development team are now fixing way more defects, or that their development quality has sky-rocketed (so less defects in the code)… no, one of my first thoughts was that the quality of the testing may have dropped (maybe a combination of both)! Is that bad of me? Why shouldn’t I jump on the good news bandwagon? Because I like to have my house in order before I do such things. Maybe, just maybe there are the same amount of defects in the code but the testing is no longer tracking them down.
But hang on, we have a target of ZERO outstanding defects at the time of the next release… that’s easy to make. Let’s just find less defects! That way we can take the pressure of the development team in terms of fixing, and we can all look ‘totally gnarly dude’ when we go live with ZERO defects! Sweet as bro! Ok, ok… I’m being synical. Not all testers would have that view, but if they were so inclined that target could induce such a view.
Do we stay hidden behind such numbers? Not I. I want to know why. I want to be certain it’s for a good reason. If it is, then lets all have a beer! If it isn’t, then let’s get to work!