ISO 29119 and “best practice”; a confused mess

In yesterday’s post about the Cynefin Framework, testing and auditing I wrote that best practice belongs only in situations that are simple and predictable, with clear causes and effects. That aligned closely with what I’ve seen in practice and helped me to make sense of my distrust of the notion of best practice in software development. I’ve discussed that before here.

In stark contrast to the Cynefin Framework, the ISO 29119 lobby misunderstands the concept of best practice and its relevance to software development and testing.

Wikipedia defines a best practice as,

…a method or technique that has consistently shown results superior to those achieved with other means, and that is used as a benchmark.

Proponents of best practice argue that it is generally applicable. It is hard to get to grips with them in argument because they constantly shift their stance and reinterpret the meaning of words to help them evade difficult questions.

If one challenges them with examples of cases when “best practice” might not be useful they will respond that “best” does not necessarily mean best. There might indeed be situations when it doesn’t apply; there may be better choices than “best”. However, “best practice” is a valid term, they argue, even if they admit it really only means “generally the right thing to do”.

Sadly some people really do believe that when defenders of best practice say a practice is “best” they actually mean it, and write contracts or take legal action based on that naïve assumption.

This evasion and confusion is reflected in the promotion of ISO 29119. Frankly, I don’t really know whether the standard is supposed to promote best practice, because its producers don’t seem to know either. They don’t even seem clear about what best practice is.

Is ISO 29119 built on “best practice”?

These two quotes are extracts from “ISO/IEC/IEEE 29119 The New International Software Testing Standards”, written by Stuart Reid, the convener of the ISO 29119 working group. The current version of the document is dated 14th July 2014, but it dates back to last year at least.

Parts of ISO/IEC/IEEE 29119 have already been released in draft form for review (and subsequently been updated based on many thousands of comments) and are already being used within a number of multi-‐national organizations. These organizations are already seeing the benefits of reusing the well-defined processes and documentation provided by standards reflecting current industry best practices.

Imagine an industry where qualifications are based on accepted standards, required services are specified in contracts that reference these same standards, and best industry practices are based on the foundation of an agreed body of knowledge – this could easily be the testing industry of the near future.

This next quote comes from the ISO 29119 website itself.

A risk-based approach to testing is used throughout the standard. Risk-based testing is a best-practice approach to strategizing and managing testing…

That all seems clear enough. Standards reflect best practices. However, Stuart Reid has this to say in the YouTube video currently promoting ISO 29119 (5 minutes 25 seconds in).

A common misconception with standards is that they define best practice. This is obviously not a sensible approach as only one organisation or person can ever be best at any one time and there is only one gold medal for any Olympic event. We all know there is no one best approach to testing. It should be context driven.

If standards defined best practice then most of us would never even bother trying to achieve this unattainable goal knowing it was probably too far away from our current situation. Instead standards should define good practice with the aim of providing an achievable goal for those many organisations whose current practices are deficient in many ways.

Well, I think the world can be forgiven for the misconception that ISO 29119 is intended to define best practice since that is what the convener of the ISO 29119 working group has clearly said.

Stuart Reid repeated that last quote in Madrid in May this year, and the accompanying slide provides plenty to mull over, especially in the light of the assertion that standards are not about “best practice”.

quality & standards

The slide assumes quality improves as one moves from current practice, to good practice and on to best practice. This depiction of neat linear progression implied by the rising quality arrow is flawed. It seems to be based on the assumptions that quality can be tested into products, that potential users of the standard (and every organisation is considered a potential user) are currently not even doing good things, that best practice is the preserve of an elite and that it is by definition better than good practice.

Not only do these assumptions look indefensible to me, they are not even consistent with the words Stuart Reid uses. He says that there is no “best approach” and that testing should be “context driven”, yet uses a slide that clearly implies a standards driven progression from good to best. This is not the meaning that people usually ascribe to “best practice”, defined above. Best practice does not mean doing the best possible job in the current context regardless of what others are doing. If that were its generally accepted meaning then it would not be a controversial concept.

Muddled language, muddled thinking, or both?

It’s hard to say whether all these inconsistencies are just muddled thinking or whether ISO 29119’s defenders are starting to give way on some weak points in an attempt to evade challenge on the core message, that standards represent an accepted body of knowledge and we should buy into them. Is “best practice” now acknowledged to be an inherently unjustifiable concept in testing? Has it therefore been thrown overboard to stop it discrediting the rest of the package, and the various ISO 29119 materials have not all been brought into line yet?

The tone I infer from the various documents is a yearning to be seen as relevant and flexible. Thus we see “best practice” being suddenly disowned because it is becoming an embarrassment. The nod towards “context driven” is deeply unimpressive. It is clearly a meaningless gesture rather than evidence of any interest in real context driven testing. Trotting out the phrase glibly does not make a tester context driven and nor does it make the speaker seem any more relevant, flexible or credible.

If a standard is to mean anything than it must be clear. ISO 29119 seems hopelessly confused about best practice and how it relates to testing. Comparing the vague, marketing froth on which ISO 29119 is based with the clear perspective offered by the Cynefin Framework reveals ISO 29119 as intellectually incoherent and irrelevant to the realities of development and testing.

Ironically the framework that helps us see software development as a messy, confusing affair is intellectually clear and consistent. The framework that assumes development to be a neat and orderly process is a fog of confusion. How can we take a “standard” seriously when its defenders don’t even put up a coherent defence?

3 thoughts on “ISO 29119 and “best practice”; a confused mess

  1. It’s clear that the concept of “best” in the expression “best practice” is vulnerable to even the most shallow analysis. I’d also like to see us, as a community, start to point out the vulnerability of “practice”, too. It seems to the concept of “practice” poorly framed, and practices are consistently under-specified and under-described.

    • “Best practice” certainly is a vulnerable concept. However, Stuart has started to abandon the concept in such haste that he’s not shifted to stronger ground. It might have been better to stand firm till he’d worked out a credible plan B. Still, I guess I’m in a lousy position to offer advice on the best route to coherence. As the old Irish joke goes, “if I were I you I wouldn’t be starting from here”.

      You’re right about “practice”. I read Donald Schön’s “The Reflective Practitioner” last year and it helped my thinking. Writers like him have a much clearer and more sophisticated idea of “practice” than the people who bandy around “best practice”. In that context “practice” means little more than “something”. Schön’s book influenced
      this post on my blog.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s