ISO 29119 the new software testing standard – what about it?

This is the same piece that I posted at the Software Testing Club. I picked that site because I hoped to see a good debate about the issue. If you want discuss this it would probably be best to go there, but feel free to add a comment below.

“One definitive standard”

“The aim of ISO/IEC 29119 Software Testing is to provide one definitive standard for software testing that defines vocabulary, processes, documentation, techniques and a process assessment model for software testing that can be used within any software development life cycle”.

Oh dear! That’s the introduction to “ISO/IEC 29119 Software Testing – the new international software testing standard”.

I’m afraid my hackles rise when I see phrases like “one definitive standard” and “used within any software development life cycle”. It immediately triggers an adverse emotional reaction as I remember this rhyme from Lord of the Rings, about the One Ring that would give the holder power over all.

“One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie”.

More pertinently, I also think of this piece of ancient software development wisdom, from McCracken and Jackson in 1982. I’m afraid I don’t know of a free source.

“Any form of life cycle is a project management structure imposed on system development. To contend that a life cycle scheme, even with variations, can be applied to all system development is either to fly in the face of reality or to assume a life cycle so rudimentary as to be vacuous”.

What’s the problem with standards for testing?

The people who are putting ISO 29119 together are experienced, well intentioned professionals, and I’m sure that they are well aware of many of the potential pitfalls that they face.

However, I’m far from convinced that the exercise will ultimately prove worthwhile. Is there a happy middle ground between an excessively prescriptive standard that will be an irrelevant nuisance in many situations, and a high level standard “so rudimentary as to be vacuous”?

Perhaps the framers of the standard will get the balance right. However, even if they do, I worry about what will happen when the standard is released. Even if the standard’s creators envisage a set of useful guidelines, from which people can select according to their need, the reality is that other people will interpret them as mandatory statements of “best practice”.

Lawyers and large consultancies will see a business opportunity, and write contracts that require the production of all the documentation examples offered by the standard. Regulators and legislators will pounce on the standard as evidence of professionalism, with failure to comply being interpreted as deviation.

I believe that guidelines are extremely valuable, but allowing them to mutate into rigid standards is dangerous. I’ve written about this here, and I won’t repeat my arguments in any detail.

I have three basic concerns about standards in software development.

1 – They inhibit the growth of novices and constrain experts.

2 – They are consistent with, and in practice encourage, a fundamental misunderstanding about the nature of software development, which is an iterative process of design and discovery, not a linear production process.

3 – Standards, by their very name, encourage non-experts to insist that they should be mandatory, without any understanding of whether they are relevant in a particular context. In my experience goal displacement poses a major threat. The focus of projects can become production of mandatory documentation and adherence to standards, rather than the real work.

Michael Bolton has produced some compelling arguments against standards. I’d urge you to familarise yourself with them. See Michael’s 2011 presentation (PDF, opens in new window), and his blog.

The question, as posed in a tweet by Michael Bolton the other day, is “What can we do, then, to stop ISO 29119? A worldwide movement is called for”.

A debate about testing standards?

Forums like the Software Testing Club are great, but often we use them to preach to the converted. Testing conferences may reach a more diverse audience, but the real problem is neither the testers who want to introduce standards to our profession, nor the standards themselves. The problem lies with the way that they are applied. We therefore have to look to those outsiders who will turn the standard from guidelines to rules.

So what do we do?

I think we should be reaching out to these outsiders, networking, cultivating contacts and influencing those who can help us, our who might make our lives difficult if they don’t understand what we are doing.

I used to work as a computer auditor, so I believe strongly that good auditors can be a great ally. They are often, wrongly, blamed for imposing mandatory standards. Managers often demand slavish adherence to standards, especially detailed documentation, “because the auditors insist”. Only bad auditors insist that software development standards should be applied without exception, in every case, regardless of context. Likewise, the US Sarbanes Oxley is often wrongly blamed for demanding excessive documentation. That’s something I expand on in the last part of my article (referred to above) and also in this blog about the similarities between testing and auditing.

What do you think? Are you bothered? Are you happy enough to work with whatever standards are produced? If so, why?

If you are worried, what are you going to do?

I hope that this is something that people will want to discuss. A new testing standard will affect the way that huge numbers of testers will have to work, whether they like it or not.

Edit. A petition has been set up in August 2014 to calling for ISO to withdraw ISO 29119 on the ground that it lacks the consensus that its own rules requires. Consensus is defined in ISO/IEC Guide 2:2004 as follows.

“Consensus: General agreement, characterized by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments.”

The petition argues, correctly, that there is no consensus. Further, the process did not seek to take into account the views of all parties concerned. The standard reflects one particular view of how testing should be conducted and marginalises those who disagree. If governments and companies insist that ISO 29119 should be adopted, and that suppliers should comply, this will have a dramatic, and damaging, effect on testing and our careers.

I urge all testers to sign the petition.