Experimenting with Process Change – You’re Allowed to Fail

Over the past 12-18 months I’ve been working a lot with various process change. Transition to this, transition to that, and all the associated process change required as you learn and adapt along the way. If I had to call out the biggest challenge, or even blocker to success, it would be people’s unwillingness to fail.

*Fail: to fall short of success or achievement in something expected, attempted, desired, or approved:

The experiment failed because of poor planning.

*Dictionary.com (I underlined the word experiment)

From my experience people’s understanding and definition of the word fail is too negative. It can often prompt quitting… “I failed and that, so I’m going to quit”.

I’ve been thinking about how to change that, or perhaps even a new word to use…

• Flounder
• Fall
• Fizzle
• Flop

They just don’t work. 😉

The reason I underlined the word experiment in the example taken from Dictionary.com is that it aligns so nicely with where my thinking is currently at – treating process change as an experiment, or a series of experiments to be more precise. Attempting to soften the blow of the words fail and failure.

Taking from science, what happens when an experiment fails? Sure, we could simply quit thinking that our hypothesis is false. However, the science community teaches us to push on, revisit our variables and control them more tightly if required, take information gathered from the failure and revise the hypothesis, then experiment again. Rinse and repeat.

Remember, a failed experiment can yield just as much valuable information as a successful one, if not more!

Identify Potential Process Problem/Desired Change – At this point you’re thinking about the why, the goal of the change. Transitioning to a new way of working? Identified a timeliness problem via value stream mapping? This is the starting point where you identify a potential process problem, decide a change is required to drive efficiency, or perhaps you’re told to change by a higher power. Whatever the reason, you need/want to change.

Build Process Change Hypothesis – You’re now thinking about the how. Big bang? Small increments? What can you attempt in order to see success with the change and meet the desired goal.

Plan Experiment/Control Variables – The size of the plan and the amount of effort put into it will of course depend on the experiment and the amount of variables you’re potentially dealing with. When experimenting you need to be able to control and understand the variables, and with process change this can be extremely difficult due to the often large scale of human involvement. You may even need to allow for the mood of people involved, and who knows what that could be from one minute to the next. This is where understanding more than control is important. If you can at least understand the variable (as you cannot control a person’s mood for example) you can allow for it during the experiment and when drawing your conclusions.

Run Experiment – Execute your plan and run your experiment! Take notes, collect data/information, monitor your variables and continue to understand them if you cannot control them.

Draw Conclusions – This is the important part (well, it’s all important but this is where you can influence people’s view on failure). Did what you implement meet your goal? No. That’s fine, we’re simply experimenting remember. Take what you’ve learned from the experiment and circle back to the top!

This all seems very easy and logical in theory. As with most things, in practice it’s far more difficult. Some cautions:

• Don’t bite off more than you can chew. The size of the experiment can make a huge difference in the success of this change process and people’s willingness to fail. If you build and experiment that will take months to run then people’s acceptance and understanding of a failure will be much harder to come by. Not only that, with a larger process change experiment comes more variables. You need to limit the variables as much as possible, especially those you cannot control.
• Use the language of experiment often. The ‘general’ understanding of the word experiment (and the process of experimenting) brings a softer meaning to the words fail and failure. People understand that experiments fail, and that it doesn’t mean we should immediately give up.
• In line with the first caution above, think about the level of risk associated with your experiment. If your experiment does fail, what will the impact of that failure be? Can you recover quickly? If you assess the impact of the failure being too great then it could be a sign you need to breakdown the experiment into smaller ones, or remove potential high risk variables. You can’t sell a process of experimenting if your experiments keep killing the business!

The key with all of this is getting people to understand and accept that a certain level of failure is OK, as long as you’re learning from it.

The Invisible Gorilla…

I finally finished it… after a duration of more than a year! While I don’t entirely blame the book for such a long period of time being required (moving to NYC played a part) it was a difficult read. Not that it contained overly complex ideas, but I found it pretty tedious throughout. I would normally just stop reading a book like this, but I was very interested in the content it proposed, so I pushed myself.

Pretty sure most people reading this post would have heard of the short video titled ‘The Invisible Gorilla’. Count the passes of the basketball, get to the end, did you see the gorilla (sorry, spoiler)? It’s a sound experiment, and holds a great wealth of insight to EVERYBODY, not just software testers. It was that video that lead me to purchase the book. The video captures one illusion; the illusion of attention, and book further builds on that illusion including other common ones namely the illusions of memory, confidence, knowledge, cause, and potential.

Each chapter focuses on a different illusion in the above order. It’s a mix of example via real-life story and example via real-life experiment. I found the stories quite fascinating and likened them to those told within Freakonomics. It was fascinating to read about criminal witnesses mistaking what they sore or what they remembered, or how confidence can be the deciding factor in multiple situations, and so on. As with many elements of psychology, the explanations used to describe human behaviours in the book were broad, and you can never really be 100% certain of their accuracy, however it was enlightening for the most part.

When the examples moved into descriptions of experiments it all slowed down a bit for me. This is no doubt a personal preference on how a book can keep my attention, so don’t let be a deciding factor for you. Having the ‘data’ to back up the theories does help, albeit a little boring for me to read about. They weren’t all that way, but I think as you near the end of the book you can sense an element of repetition in the writing which causes things to drag on a bit.

For me personally the illusions of attention, memory, and cause were the most powerful.

The illusion of attention has been a big player in my software testing career and until I learned about inattentional blindness and James Bach’s focus/defocus heuristic (something that also helped me a lot with the Dice Game), I’m pretty sure I fell victim to it a lot more often. I have also witnessed many other testers move directly past obvious bugs because their focus was on another part of the product, or on something entirely different all together (having a bad day, etc.). I probably should clarify that the bugs were obvious to me, but using the word obvious generally is not entirely fair. Due to the illusion of attention the bugs were not at all obvious to the tester sitting in front of me, and they cannot be blamed for that… they’re human. Pair testing helps a great deal to counter this illusion. Even if you both try and focus on exactly the same thing, you won’t be. Each of us see things in a unique way, which helps us identify different bugs, even if only slightly. The trick is to be consciously aware that your attention is a limited resource. Don’t think that you’ll see it all, because you likely won’t (hell, it’s science baby!). Another thing that has helped me counter this is product tours with particular charters. If you spend a lot of time with the same product and you test for similar things, take a step back and chart a course for a usability tour, or a security tour, etc. Whatever type of tour you think may yield some important information that you may not have discovered otherwise.

Early failures with the Dice Game also alluded to the illusion of memory. I first played the Dice Game during my Rapid Software Testing training. I was lucky in that respect as we could work in groups attempting to find the pattern. During that game, and in some since, I very quickly forgot (well, actually misremembered) the previous patterns I had been working on. There were many occasions where I finally solved it thinking that I had already been down that path with no success. This continued until I began taking more significant notes throughout the game. With those notes I could constantly check on what theories I’d worked on previously and my problem of misremembering was gone. As a tester, how many times have you been asked why you didn’t identify a bug that was now present in production? How many of those times do you distinctly remember testing in that area of the product? How did you know? Did you just ‘remember’ testing it, or did you actually go back and look over your notes? The book does a fabulous job of showing the reader how human memory can deceive us, very easily. Reading about this illusion once again highlighted the importance of taking notes and gathering evidence while testing.

The illusion of cause has been an interest of mine since reading Freakonomics. Generally speaking the entire book is about this illusion and how people are quick to jump to a particular cause rather than seeing it merely as a correlation. Humans have evolved in a way that allows them to identify patterns, but not only this, they also seek them. Our understanding of time (always moving forward) drives many of the patterns we identify/seek. In a sequence of events it’s natural for humans to identify the first event as the cause of the remaining, when in reality they may not be related at all. In the software industry I think we can spend too much time seeking and then identifying the wrong cause for many of our failures. If the sequence of events lines up nicely we jump on the first event as the cause. This chapter was a reminder for me to look at events in a different way, and to question my own tendency to identify a particular pattern.

There are also many lessons to be found in relation to the illusions of confidence, knowledge (often a result of the former!), and potential. While it was a struggle at times, I would encourage people to read it.

The ultimate lesson I took away from this book? To live my life and do my work while being aware that these illusions exist. Sure I’ll forget from time to time (I’m only human after all), but hopefully I’ll be able to remind myself before I miss something really important.

Peace.

Seek, Identify, Gather… A model for talking about testing.

When I talk about testing I tend to talk about information. For me, testing is all about information. Information that helps decision makers make decisions, and information that informs risk. How we identify that information depends on the type of information we need, and to know that we can do several things… identify risk, ask questions, user research, analyse specifications, find out what’s important to those who matter; the list goes on.

Only recently have I actually thought about this as a model. So I thought I’d spend some time visualising it (something that helps me think deeper about a subject).

What testing means to me at a high-level:

If testing wasn’t all about information, then it would be all about risk (generally speaking). So first up I like to talk about risk and risk identification. Testers (for the most part) are good at that. We tend to come with varied backgrounds; business, tech, users, and so-on. So we can view a product with many different lenses which provides greater risk identification coverage.

Then I move to the information required in order to understand and potentially mitigate the risk. The information we seek depends on the risk we’re trying to understand, as does how we seek and identify it.

Let’s visualise that part:

I then like to talk about gathering and reporting the information. Test reporting can be very painful, or it can be a breeze. Which one it ends up being not only depends on your stakeholders, but also on how you gather the information while testing. If you don’t record your testing (screen captures, notes, etc.) then how are you going to gather the information you need to report? Pass and fail? Whatever… realistically that means nothing. You need to work out what your stakeholders need (or in the real world, what they want) to see, and then gather the information in a way that translates as easy as possible to it. You may have a project sponsor that totally gets what you’re doing and completely trusts you, and so doesn’t need to see anything… but lurking in the background is the internal audit team who need to see EVERYTHING! So I like to include that when I talk about testing as it can make your break your entire effort (rightly or wrongly).

Thoughts? Feedback is always welcome.

Peace…

Time to step back…

If you align yourself with the Context-Driven Testing (CDT) community, or you’ve been a part of it in the past, then you’ve likely seen the flame wars on Twitter recently. If you haven’t, lucky you… don’t bother looking. Instead, just worry about doing good testing… whatever that means to you and your organisation.

I’ve remained silent for a few reasons:

• In my opinion Twitter is absolutely awful for discussion/conversation, especially when there are more than two people involved.
• I think many of the Tweets carry hatred due to personal issues/personality conflicts.
• I think many of the Tweets are simply childish, like throwing the first punch and watching all your ‘mates’ deal with the aftermath.

I’ve seen awful words like:

• Bullies
• Asshats
• Cowards

While these were not directed at me personally, they were directed at members of the CDT community in a very general and broad way. I therefore have included myself within their general and broad target. Rightly or wrongly, this is what happens when you label a community in such a negative way… the self-identified members take it personally (shocker).

This is me now raising my voice because I’m sick of being labelled in such a generalised way.

There are members of both ‘sides’ (I feel dirty even writing that word in this context) that simply need to grow up. If I witnessed my 9 year old daughter partaking in such rubbish I would educate her on how to handle it in a far more mature way. In fact, I’d be surprised if my daughter was even that bad in the first place.

Drop the elementary/primary school stuff folks, you just look silly.

What follows is a very personal story/experience report. I would like it known up front that I may also use generalisations, and if I offend anyone while doing do, I apologise.

When I first started out in testing I worked for a large organisation for many years. If you were to label their approach to testing then you may call it ‘factory’, or perhaps ‘traditional’. Whatever you label it, it was pretty damn wasteful for the most part. I spent 8 years learning how to test with that organisation and toward the end I began discovering that the way they tested could have been so much better. The turning point for me was reading Lessons Learned in Software Testing. This was my introduction to the CDT approach to testing (not the community – an entirely different beast).

After reading the book I then began to research more. I found more books, blogs, Twitter, LinkedIn, etc. I was then beginning to get introduced to the CDT community. After 8 years of ‘factory’ testing and no interaction outside of my organisation it was very refreshing. I met people (most virtually) that were making the same discoveries I was, other people that had called themselves CDT for years, and others that were ‘kind of’ in the CDT community and ‘kind of’ not. At the time I couldn’t have cared less what the people called themselves or what community they thought they were in… and guess what? I still don’t. If someone is willing to interact with me and help me make my testing better, I’ll interact with them. It’s true that most of the people I could find at the time were in the CDT community. Perhaps this was because there was no ‘factory’ community, or perhaps it’s because of where I was looking. I don’t care… and yes, I still don’t.

I naturally moved towards the CDT community and eventually self-identified as a member. This is an important point… I self-identified. I wasn’t initiated, welcomed, tested, approved, etc. I simply referred to myself as a member of the CDT community. If James Bach (yes, I’m using his name… like grown-ups do) had told me that I wasn’t a member of his CDT community, then I would have been a member of my own CDT community, with other like-minded people. You don’t need James’ approval. Sure, you can attribute a lot of what people understand CDT to be to James, but there are also MANY others who constantly contribute to it and move it forward… whether James agrees or not (and he’s the first to welcome that). Before I move on from James (because this is bigger than that)… yes, he can be abrupt, loud, intimidating, rude, and so-on… he once left me hanging on a Skype chat with “I thought you were better than that.” Ouch. However, I cannot dispute his testing intellect, passion, drive, work-ethic, and at times what appears to be pure genius. He’s not for everyone (from an interaction perspective), but you’d be silly not to at least research his work… even if you have to do so from arm’s length.

In more recent years I’ve spoken and written about the need for labels (i.e. CDT), and how I would LOVE them to not be required. I think this is partly because I can see the tension within the different ‘schools’ of testing, and the absolute waste of energy is causes – for the most part. Perhaps gradually I’ve been moving away from identifying myself or the way I test with a particular label. I just want to do good testing, or help to get good testing done. An unfortunate part of testing is the need for labels. If I want to differentiate the testing I do to the testing that someone else does, how do I do that without showing it (preferred option which is not always possible), or having a label for it as a conversation starter? Perhaps ‘good’ can be my new label. Then I can talk about what ‘good’ looks like/is in a particular context.

I’ve never been an advocate for “Can’t we all just get along?” I don’t think that is helpful when trying to move our industry forward. We need debate, different approaches, and different personalities who focus on different aspects of testing. How else would we make our ideas better? Self-critique can work, but other’s critiques are often far more powerful (never proof read your own work, etc.). However, maybe I’ve been looking at “Can’t we all just get along?” in the wrong way. We can all get along while debating and disagreeing on different approaches to testing, can’t we? Then again, we’re human. If getting along was that easy when we disagreed on something… well, news headlines would be a lot different.

After spending 8 years as a ‘factory’ tester, and almost as many in CDT, I think the time is right to move on from labels. From now on I’m going to be a tester, no… a good tester (or test manager, etc.). I’m sick and tired of the ‘politics’. I know they will never go away, and it would be ignorant of me to think I can avoid them in the future, but for me personally I think I can do way better if I put my energy towards good testing instead.

For those of you that will jump at the chance to declare me another victim being ‘pushed’ or ‘forced’ out of the CDT community, it couldn’t be further from the truth. No-one has done that, and the only person you can ‘blame’ (because some of you have to it seems) is me. Besides, while I won’t label myself, I’ll continue to interact with members of the CDT community (because most of them are awesome), and I really hope that I also interact with those that aren’t as well (because some of them seem awesome too). 🙂

I’m going to focus on good work and not the bullshit that often surrounds it. I’ll deal with bullshit when I have to, but no more excess energy will be spent on it. I’d like to continue helping others like Katrina has written about. For example, I started OZWST after my crew from NZ introduced me to KSWT… and I’ll continue to do cool things like that, but I will not make it about labels.

Peace…

Wonderful Words from Jerry Weinberg

Today I received some wonderful words from a man who has inspired many parts of my software testing career. Those words were in relation to my book, Software Testing as a Martial Art…

“What a gift David Greenlees’ book is!

The day I received my copy, I spent all morning scanning, digging, smiling, nodding appreciatively. It’s a wise book, and I’m recommending it to all the testers (and others) I know.

But prepare yourself. This insightful book shows the reader that testing is like a martial art in a number of ways—ways essential for testers to understand. A caution though, about a way this book itself is like a martial art:

Beginners may simply not understand much of what Software Testing as a Martial Art has to say until they have practiced. Very few people begin their testing careers by applying principles they’ve read about, no more than I did when I began testing (or a martial art). So, though Software Testing as a Martial Art is a book testers should read at the beginning of their testing careers, it’s also a book they should re-read every year as their experience grows. I know I plan to re-read it every year on my birthday.

I also plan to continue my study of Aikido, my preferred martial art. Thanks to Software Testing as a Martial Art, I now understand that every Aikido practice will also improve my testing art.” – Jerry Weinberg

Thank you Jerry, I couldn’t be happier.