Too smart for checklists, and a consultants’ war?

My post last week “Why do we think we’re different” “Why do we think we’re different” (about goal displacement, trained incapacity and functional stupidity) attracted an interesting comment that deserved a considered response. In the end that merited a new post, rather than a reply in the comments section. Here is the comment, followed by my response.

I don’t disagree with your basic premise nor your conclusion. Your arguments are logical and elegant, but I wonder if people hear the anti-ISO 29119 group as saying “Don’t bother having process,” and “I’m too smart to need a checklist” (ala Gawande’s “Checklist Manifesto”).

Worse yet, the people arguing against the standard the loudest are not the common tester but the already heavily active in the community of testers. I described it as a “consultant’s war” because consultants seem the most active in the community are often consultants.

As a people problem, it seems both sides have a great deal of apathy outside of perhaps 1/100th of the testing community. How do we change that? The best argument in the world will have no affect if most people are apathetic. This is the problem I struggle with and not just around the standards but around testing as a career. I would love to see any insight you have around the problem, assuming you see it as a problem.

– JCD

JCD raises some interesting points and I am grateful for that.

I do need to keep thinking about the danger that people hear the Stop29119 campaigners as saying “don’t bother having process”. That is not the message I want to get across. I would phrase it as “beware of the dangers of prescriptive processes”, i.e. processes that spell out in detail each step. These may be required in some contexts, but not in software development or testing, where their value has been hugely overstated.

However, processes are required. The tricky problem is finding the sweet spot between providing sufficient guidance and ensuring appropriate standardisation on the one hand, and, on the other hand, going into excessive detail and preventing practitioners from using their judgement and initiative in a complex setting.

Similarly, checklists do have their place. They are important, but of limited value. They take you only so far. I certainly don’t decry them, but I do deplore excessive dependence on them at the expense, again, of judgement and initiative. I don’t disagree with the crux of Gawande’s book, though I do have reservations about the emphasis, or maybe it’s just with the way the book has been promoted. I’m also doubtful about his defintion of complexity.

I agree with what Gawande is saying here in The Checklist Manifesto.

It is common to misconceive how checklists function in complex lines of work. They are not comprehensive how-to guides, whether for building a skyscraper or getting a plan out of trouble. They are quick and simple tools aimed to buttress the skills of expert professionals. And by remaining swift and usable and resolutely modest, they are saving thousands upon thousands of lives.

I would quibble about building skyscrapers being a complex activity. I would call it highly complicated. I’d prefer to stick to the Cynefin definition of complexity, which is reserved for situations where there is no obvious causality; cause and effect are obvious only in hindsight. Nevertheless, I do like Gawande’s phrase that checklists are “quick and simple tools to buttress the skills of expert professionals”.

You’re right about this being a consultants’ war, but I don’t see how it could be anything else. It’s hard to get new ideas through to people who are doing the real work, grafting away day to day. Some testers are certainly apathetic, but the bigger issues are time and priorities. Campaigning against ISO 29119 is important, but it’s unlikely to be urgent for most people.

When I was a permanent test manager I didn’t have the time to lift my head and take part in these debates. Working as a contract test manager isn’t much better, though at least it’s possible, in theory, to control time between contracts.

I think that places some responsibility on those who can campaign to do so. If enough people do if for long enough then those testers who are less publicly visible, and lay people, are more likely to stumble across arguments that will make them question their assumptions. The lack of public defence of the standard from ISO has meant that anyone searching for information about ISO 29119 is now very likely to come across serious arguments against the standard. That might tilt the balance against general acceptance of the standard as The Standard. A few more people might join the campaign. Some organisations that might have adopted the standard may think better. It could all have a cumulative effect.

I’m working on the idea of a counter to ISO 29119. It wouldn’t be an attempted rival to ISO 29119. Nor would it be an anti-standard. Many organisations will adopt the standard because they want to seem responsible. That would be doing the wrong thing for the right reason.

What I am thinking of doing is documenting the links between the requirements of the Institute of Internal Auditors, and the Information Systems Audit & Control Association (and maybe other regulatory sources) with a credible alternative to the traditional, document heavy approach, showing how it is possible to be entirely responsible and accountable without going down the ISO 29119 dead end.

It’s a matter of explaining that there are viable and valid choices, something that is in danger of being hidden from public view by the arrival of an ISO standard intended to cover the whole of software testing. Checklists could well play a role in this.

I smiled at the suggestion that opponents of ISO 29119 might believe they’re too smart to need a checklist. I can see why some people might think that. However, I, like most ISO 29119 opponents I suspect, am acutely aware of the limits of my knowledge and competence. Excessive reliance on checklists and standards create the illusion, the delusion, that testers are more competent, professional and effective than they really are. I prefer a spot of realistic humility. Buttressing skill is something I can appreciate; supplanting skill is another matter altogether, and ISO 29119 goes too far in that direction.

Advertisements

4 thoughts on “Too smart for checklists, and a consultants’ war?

  1. Selective hearing is amazing.

    I’m not too smart for a checklist. I use checklists all the time. (Here’s one example: http://www.satisfice.com/tools/htsm.pdf) However, I’m constantly struggling to make sure that my use of a checklist does not become standardized or calcified template-following.

    The 29119 standard is not a checklist. At several hundred pages in total, it’s too unwieldy to be a checklist. It doesn’t, by my reading, offer pointers to any reasonably useful checklists.

    One of my long-standing complaints about our craft is the problem of reification: turning an abstract concept or a construct into a thing, an object. The process model is not the process; the map is not the territory.

    We always a have a process; process is what happens. We support both process (how could we not?) and process models that are helpful. The opponents to Stop29119 confuse “don’t bother having an unwieldy, overstructured, oversimplified, or unhelpful process model that ignores judgement and skill” with “don’t bother having a process”.

    Here’s a way to avoid apathy, in my view: stop putting standards and process models and best practices and bullshit certifications at the centre of testing, and start putting testers and conversation about skilled work at the centre instead. Things like “judgement” and “initiative”, as you (James) suggest above. That’s what the context-driven community has been trying to do for the last decade and more.

    —Michael B.

  2. Thank you! I deeply appreciate your reasoned response.

    Michael, I did not mean to imply that ISO 29119 was a checklist, but it is the sort of response that sounds reasonable as it has been argued that the standard is a map that you can map your own process to. Basically a check list to ensure your process has the documents you need. I am aware the document is too long to be a checklist, but looking at Dr. Reid (http://www.softwaretestingstandard.org/29119petitionresponse.php ) has said, it sounds like you can basically use it as a guide and be standards compliant:

    “”The standards require users to create too much documentation.” Unlike the IEEE 829 standard, which it replaces, the test documentation standard, ISO/IEC/IEEE 29119-3, does not require documentation to follow specific naming conventions nor does it require a specific set of documents – so users can decide how many documents to create and what to call them. The standard is information-based and simply requires the necessary test information to be recorded somewhere (e.g. on a Wiki). As stated above, it is fully aligned with agile development approaches and so users taking a lean approach to documentation can be fully compliant with the standard.”

    It very much sounded like he was describing a sort of checklist, and as such it felt like a logical argument those who are pro-standards might give.

    Speaking only for myself, I am trying to see how the other side sees themselves and see what they are thinking. Even if they are wrong in my opinion, I don’t want to be stuck in a purely attack position and end up unempathetic to their viewpoint. With most people not having much time to consider these matters, if we look like we are just bashing people’s heads and saying they are wrong with no constructive alternative we become the bad guys. I just want to ensure we are not only addressing why this is a problem, but what alternatives we believe are helpful and why they are better alternatives. You end up creating a fight or flight response rather than a reasoned consideration (https://medium.com/mother-jones/the-science-of-why-we-dont-believe-science-adfa0d026a7e ).

    I like your idea James! I would like to see a nice alternative to ‘the standard’. One of my problems with CDT is we don’t have a very good set of *organized* routes for learning or developing. I talked about it a while ago (http://about98percentdone.blogspot.com/2013/09/devils-advocate-where-context-driven.html ). That isn’t to say we don’t have data, just their isn’t something even akin to the degree with structure that a college gives. There isn’t a path or even a set of possible paths like a choose your own adventure book. I know RST, LST, BBST and other classes exist, but they feel like only a starting off point and how do you know which one to choose? It gets into the problem of choice in which less is more (http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More_Is_Less ; http://en.wikipedia.org/wiki/Overchoice ). That is a powerful draw to using a standard. One choice and you are done. Again considering managers don’t have much time, which are you going to choose? The people with “The Answer” or the people with 15 different answers and no clear path to advancement? This is where I fear apathetic testers will just take the standard and move forward. In fact having more choices tends to make it harder to choose and the CDT side of the fence has lots of choices.

    I say this as more of a challenge to us to become better as a community, not to say give up and go with the standard. My co-author of my blog actually took up the challenge and came out with with an ad-hoc list (http://about98percentdone.blogspot.com/2013/09/where-cdt-fails-rebuttal.html ) of ideas for learning within CDT. I think it is brilliant, but it is only one small part of what needs to be a bigger framework for new testers.

    – JCD

    • Stuart Reid has stated explicitly, in the response to Stop 29119, that ISO 29119 serves as a checklist. This looks like casting around for any argument to try and defend the standard even if the cumulative effect is an incoherent case. ISO standards and checklists are different things, or they should be. Anything that tries to be both will struggle to achieve credibility, and rightly so.

      “One of the most common uses I make of the testing standards is to use them as checklists – I then have more confidence that I haven’t accidentally ignored an important aspect of the testing.”

      I would hesitate to say my idea of a counter standard wold serve as an educational tool, or even a route map. I hadn’t considered that. I think the work of Messrs Bach and Bolton stands for itself. The most I’d claim is that my idea may have value as an adjunct. It might make RST, and other valid alternatives to ISO 29119, more accessible and attractive to those who feel the need to adopt a rigorous, credible and accountable approach. Such people are the natural target market for ISO 29119. However, if they are open minded they could be amenable to more effective alternatives, provided that we can explain that these would work for them.

      The article you linked (and thanks for that) “The Science of Why We Don’t Believe Science” concludes that if we want to change minds we should “lead with the values”. There is a good lesson there. Appealing to the values that might motivate people to make the wrong choice, and explaining how they can satisfy those values with a better choice could be the way ahead.

  3. It very much sounded like he was describing a sort of checklist, and as such it felt like a logical argument those who are pro-standards might give.

    Typically a logical argument is one that the presenter doesn’t contradict as soon as it suits his purposes.

    if we look like we are just bashing people’s heads and saying they are wrong with no constructive alternative we become the bad guys.

    I don’t understand the “no constructive alternative” argument. There is a constructive alternative: the free market of ideas and practices unencumbered by an attempt to gain control or credibility by appeal to authority. There are dozens of constructive alternatives, from TMap to RST to the QAI classes Rob Sabourin’s JIT to, even, Lord help us, the ISTQB—at least the one of the past, the one that didn’t hitch its wagon to ISO.

    When you say CDT doesn’t have an organized route for obtaining education, what you mean is that CDT doesn’t have a mandated route. We believe that people should set up their own (self-organized) route, as they see fit, which is something that responsible adults are perfectly capable of doing. A bogus standard is a powerful draw to people who wish to abdicate responsibility for designing their own educations. In other words: learning how to choose your own path through education is an important part of learning.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s