The Evolution of Metrics Thinking

I had one of *those* moments recently… I was sitting at a dinner table with some great testers who were all part of OZWST. Somehow we got on to the subject of metrics in software testing (as is usually the case). I took a bit of a backseat for the discussion as I wanted to listen to the arguments being put forward by various people. While listening in, and trying to determine where each person was at in the conversation, it struck me…

The evolution of metrics thinking:

  • Metrics are good
  • Metrics are bad
  • Metrics are bad and here’s why
  • Metrics are good and bad, and here’s why

I’d like to clarify that this has been my evolution throughout the last 12 years of my software testing career. However, I do get the sense that many may also share this evolution.

Metrics are good:

This was my stance early on in my career. When I started in testing our teams used metrics (and pretty much only metrics) for all reporting. Of course they did; what other ways are there to report on progress, etc? At least that was how I used to think. While this frustrates me now, I can also relate with my former self and understand why I didn’t see anything wrong with it at the time. It was how it was always done, and it was how my *leaders* (or perhaps I should say managers) taught me to do it.

The innocence (and likely ignorance) of youth!

Metrics are bad:

These last three steps have happened relatively quickly when considering the first step lasted approximately 8 out of my 12 years!

Without delving too deep into how and why I started to align with the Context-Driven Testing (CDT) community, my evolution towards thinking metrics were bad began with the wonderful book; Lessons Learned in Software Testing. Certain lessons within this book helped me to question metrics in general, and the more I researched from that point on the more I convinced myself of this notion.

Metrics are bad and here’s why:

As any member of the CDT community would understand, you don’t get along too well if you make bold statements without backing them up. I was very happy to state to anyone that metrics were bad! I did this for a little while comfortable in the knowledge I had read it, and therefore it was true (to a certain extent). One day, someone asked me why metrics were bad… ah, well… I read it somewhere. BAM! A Batman right hook to the jaw.

Of course I need to understand why I was making such a bold statement.

More research, questioning, coaching, etc. They have all helped me to gain a better understanding of the dangers that are inherent in metrics.

Metrics are good and bad, and here’s why:

Not necessarily the final step, but the one that I’m currently at (or heading towards very quickly). After being involved in several conversations recently, and the wonderful metrics debunking series that we’re currently producing over at Hello Test World, I’m beginning to see *some* value in *certain* metrics in *particular* contexts. For me the biggest value comes from the identification of potential patterns. At the very least these can be used as indicators to prompt further investigation and/or analysis. That’s not to say the lack of patterns in metrics are an indication of anything good!

While this post is not in-depth, I do hope that it highlights an evolution of thinking that I have begun to notice more often in our industry (or perhaps more in the CDT community); one that is not limited to the subject of metrics. Sure, it could be stated that this evolution is simply the process of gaining knowledge, however I think it’s important for us stop and reflect upon, and therefore recognise, evolution’s such as these. By being able to explain such things I believe we stand a better chance of spreading the knowledge and helping others to relate, and hopefully learn.

One thought on “The Evolution of Metrics Thinking

  1. I think if more people were honest with themselves, they would agree with you. Traditional testing documentation and methodologies reigned for so long that so many people just accepted it as ‘the way’. It’s as if at some point, many people just quit thinking. However, I also find it interesting that people just jump to the other extreme only to work their way back toward traditional thought.

    Scripts are bad! –> Scripts are useful, but understand the script may not equal the actual execution
    Automation is bad! –> Tool assisted testing is not so bad
    Checking is bad, testing is good! –> Well, most tests include checks, so it’s ok
    Metrics don’t prove anything, they’re worthless! –> Well, metrics may be an indication of a potential issue

    I’m not a student of psychology, but I find that is often how I work through things. I take the opposing stance and force myself to prove the alternative. I typically find myself back in the middle somewhere.

    The main problem I have is that some folks, while working through this process, are belligerent toward one side or the other. I enjoyed your post and I think many folks would benefit from your learning process.

Got something to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s