This is the same piece that I posted at the Software Testing Club. I picked that site because I hoped to see a good debate about the issue. If you want discuss this it would probably be best to go there, but feel free to add a comment below.
“One definitive standard”
“The aim of ISO/IEC 29119 Software Testing is to provide one definitive standard for software testing that defines vocabulary, processes, documentation, techniques and a process assessment model for software testing that can be used within any software development life cycle”.
Oh dear! That’s the introduction to “ISO/IEC 29119 Software Testing – the new international software testing standard”.
I’m afraid my hackles rise when I see phrases like “one definitive standard” and “used within any software development life cycle”. It immediately triggers an adverse emotional reaction as I remember this rhyme from Lord of the Rings, about the One Ring that would give the holder power over all.
“One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie”.
More pertinently, I also think of this piece of ancient software development wisdom, from McCracken and Jackson in 1982. I’m afraid I don’t know of a free source.
“Any form of life cycle is a project management structure imposed on system development. To contend that a life cycle scheme, even with variations, can be applied to all system development is either to fly in the face of reality or to assume a life cycle so rudimentary as to be vacuous”.
What’s the problem with standards for testing?
The people who are putting ISO 29119 together are experienced, well intentioned professionals, and I’m sure that they are well aware of many of the potential pitfalls that they face.
However, I’m far from convinced that the exercise will ultimately prove worthwhile. Is there a happy middle ground between an excessively prescriptive standard that will be an irrelevant nuisance in many situations, and a high level standard “so rudimentary as to be vacuous”?
Perhaps the framers of the standard will get the balance right. However, even if they do, I worry about what will happen when the standard is released. Even if the standard’s creators envisage a set of useful guidelines, from which people can select according to their need, the reality is that other people will interpret them as mandatory statements of “best practice”.
Lawyers and large consultancies will see a business opportunity, and write contracts that require the production of all the documentation examples offered by the standard. Regulators and legislators will pounce on the standard as evidence of professionalism, with failure to comply being interpreted as deviation.
I believe that guidelines are extremely valuable, but allowing them to mutate into rigid standards is dangerous. I’ve written about this here, and I won’t repeat my arguments in any detail.
I have three basic concerns about standards in software development.
1 – They inhibit the growth of novices and constrain experts.
2 – They are consistent with, and in practice encourage, a fundamental misunderstanding about the nature of software development, which is an iterative process of design and discovery, not a linear production process.
3 – Standards, by their very name, encourage non-experts to insist that they should be mandatory, without any understanding of whether they are relevant in a particular context. In my experience goal displacement poses a major threat. The focus of projects can become production of mandatory documentation and adherence to standards, rather than the real work.
Michael Bolton has produced some compelling arguments against standards. I’d urge you to familarise yourself with them. See Michael’s 2011 presentation (PDF, opens in new window), and his blog.
The question, as posed in a tweet by Michael Bolton the other day, is “What can we do, then, to stop ISO 29119? A worldwide movement is called for”.
A debate about testing standards?
Forums like the Software Testing Club are great, but often we use them to preach to the converted. Testing conferences may reach a more diverse audience, but the real problem is neither the testers who want to introduce standards to our profession, nor the standards themselves. The problem lies with the way that they are applied. We therefore have to look to those outsiders who will turn the standard from guidelines to rules.
So what do we do?
I think we should be reaching out to these outsiders, networking, cultivating contacts and influencing those who can help us, our who might make our lives difficult if they don’t understand what we are doing.
I used to work as a computer auditor, so I believe strongly that good auditors can be a great ally. They are often, wrongly, blamed for imposing mandatory standards. Managers often demand slavish adherence to standards, especially detailed documentation, “because the auditors insist”. Only bad auditors insist that software development standards should be applied without exception, in every case, regardless of context. Likewise, the US Sarbanes Oxley is often wrongly blamed for demanding excessive documentation. That’s something I expand on in the last part of my article (referred to above) and also in this blog about the similarities between testing and auditing.
What do you think? Are you bothered? Are you happy enough to work with whatever standards are produced? If so, why?
If you are worried, what are you going to do?
I hope that this is something that people will want to discuss. A new testing standard will affect the way that huge numbers of testers will have to work, whether they like it or not.
Edit. A petition has been set up in August 2014 to calling for ISO to withdraw ISO 29119 on the ground that it lacks the consensus that its own rules requires. Consensus is defined in ISO/IEC Guide 2:2004 as follows.
“Consensus: General agreement, characterized by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments.”
The petition argues, correctly, that there is no consensus. Further, the process did not seek to take into account the views of all parties concerned. The standard reflects one particular view of how testing should be conducted and marginalises those who disagree. If governments and companies insist that ISO 29119 should be adopted, and that suppliers should comply, this will have a dramatic, and damaging, effect on testing and our careers.
I urge all testers to sign the petition.
James Christie's Blog 
To be clear: I’m not opposed to standards. Standards can make life much easier, and enable interoperability. Standards can establish protocols whereby information can be exchanged between applications. Standards afford the ability to make informed choices about whom we hire. Standards allow us to connect things together. Imagine a world ruled by Sony, where every new gadget seems to come with a new hardware interface!
It’s not standards that I object to; it’s standardization of things that shouldn’t be standards, and standards built around overstructured, oversimplified models that misconstrue what actually goes on in the world. The proposed ISO 29119 standard has all of these problems.
Look, for example, on this page, at the ISO/IEC 29199 Test Planning Process, around the middle of the page. Notice that there is a set of specific steps in a specific order. Note that there are no feedback loops; that none of the learning from subsequent stages feeds back into prior stages (despite the lame disclaimer, only recently added, that says “some steps may need to be revisited”). Note that all of this happens exclusive of, rather than in parallel with, development activity. Note how closely all this resembles the process model described in this paper, a model that the author explicity states is risky and invites failure, saying “Unfortunately, for the process illustrated, the design iterations are never confined to successive steps.” So here, in 2012, we’re seeing the Watefall model being applied to testing, a full 42 years after its first rubbishing. The Empire Strikes Back.
Now scroll down a bit farther for one more example, to “ISO/IEC 29119 Test Environment Setup and Maintenance Process”. This is interesting. It contains two steps: (1) Establish Test Environment and (2) Maintain Test Environment. How insightful! How clever! Wish I’d thought of that! Only one small issue: that’s not a process; it’s a process model. All the messy, complex, human stuff; all the competition for resources; all of the discovery and investigation and exploration and learning is left out of the model. But bet your boots: you will be required to prepare documents to show that you’re complying with the standard. Development and maintenance of those documents—or gaming the system—will require time and effort that might otherwise have been spent on actual testing.
Now go to this page to find out how you might oppose this travesty. “The most significant way people can get involved is to join the software testing working group (WG26) that is responsible for the development of the standard. This requires the most amount of commitment. To do this, you first need to become a member of your national standards body and then become a member of ISO/IEC JTC1/SC7 WG26 (Software Testing). This requires attendance to 6 days of meetings, every 6 months, at various locations across the world. ISO/IEC do not provide funding to members of working groups to attend meetings. Thus, each individual or their company needs to fund the trips and the time it takes them to develop materials for the standard.” See the setup here? The individuals who can participate are those who are a) interested AND b) capable of funding their participation independently. There aren’t many of those. I predict that the makeup of the working group will be strongly biased towards those companies that have an interest in making the standard obscure, hard to explain, and hard to follow—the kind of standard that will require armies of consultants for big consultancies to explain and support it. Don’t believe me? Look at the list of participants in the working group. Look at the companies and businesses that they’re in, the services they offer. Good thing there are no ethical dilemmas here.
The most exasperating part of all is that in order even to try to get the standard stopped, we must all contact our national standards bodies, read through the standard, provide critique, and then (likely as not) be ignored.
People: software development isn’t an assembly line; it’s a slinky. It might make sense to have one universal standard for testing if there were one universal standard for software development AND one universal standard for bugs. One might as well create a standard for editing in the absence of a standard for writing. But what kind of editing—and what kind of writing—would we get out of standardizing those activities? It’s back to 1970—and 1984.
—Michael B.
Thanks for this Michael. That’s all good stuff. In fact, it’s too good to hide away on my blog. Could you maybe post it on the Software Testing Club? I’m hoping there will be some serious debate there.
I should maybe have held back from putting it on my own blog for a couple of weeks.
I share your concerns about the make up of the standards group. It would be very hard for independent consultants to get involved. Inevitably the group will largely consist of people from a corporate environment that is comfortable with rigid standards.
Great Information! While developing software or building a company that works with technology or engineering-related projects, it might be worthwhile for you to apply for ISO Certification.
Sorry, I don’t think that’s a route I want to go down!
Head, meet desk.
—Michael B.
I copy here what I wrote in comment in “Let’s Talk Testing” Blog http://blog.johanjonasson.com/?cat=10
—————
As this standard is not coming into an industry where no other standard was available before, I guess the value or lack of it should be evaluated based on comparison to the number of existing software testing standards it comes to replace:
IEEE 829 Test Documentation
IEEE 1008 Unit Testing
BS 7925-1 Vocabulary of Terms in Software Testing
BS 7925-2 Software Component Testing Standard
It may ease the contracts which are now based on those, or worsen the terms.
If any of you know of a short comparison list we can base this discussion on (so that we will not waste our time already on reading the whole thing to figure it out, or just run empty discussions based on our previous standing without even reading it) – that will be great.
—————-
From all the tweets about that topic I feel many just “discuss” this without even trying to go into the details, and taking it as another one of those anti certification type of activity.
I’m afraid to say, that its quite possible that many of those who will sign the petition will not even bother reading the content of this document – just since people they follow said its not good… – as a Tester I resent such actions – I prefer to verify information before taking sides over it.
@halperinko – Kobi Halperin
@Kobi…
1) Does it bother you that you are not allowed to review the documents without paying (so far) $600 for the first three and heaven knows how much for the next two?
2) Here is a fairly detailed description of the standard. http://www.mstb.org/Downloadfile/Wonil%20Kwon%20-%20Software%20Testing%20ISO%20Standard%2029119.pdf (If that doesn’t work, try Googling “ISO 29119 Kwon” and you should find it right away.) Does this description reassure you that there will be a good and useful model of testing in the documents you paid for in (1)?
Hi Kobi – thanks for commenting, and thanks for chipping in Michael. As Michael says it’s very difficult to have a debate about the content of ISO 29119. It is extremely expensive, and anyone who buys it cannot share or copy the material.
I have been able to see copies. They are not encouraging. The paper that deals with test documentation is 138 pages long. It will only encourage an obsession with documentation and templates. Incidentally the only IEEE 829 test documentation standard was 132 pages.
However, there is a very strong argument against the relevance of standards to testing, regardless of the content. This article, which I wrote for Testing Experience, makes the argument.
Pingback: Stop ISO 29119 | Where Worlds Collide
Reflections on Iso29119 standards.
I am not a religious person. But I do act in my professional activities (as a tester) like one. I do follow some skills developed and educated in my accumulated previous experience – sorry no prayers.
One can have any of the monotheist belief (Cristian, Muslim, Jew or any other) it does not necessarily means he is the only righteous one. For each of us the truth lies in different culture, location, skills, and standards. Any attempt to enforce one religion upon us, sounds and smells like opposing freedom and may lead to unnecessary bloodshed. In addition to being obviously not fitted to a specific needs.
In my opinion, having one specific standard imposing on the testing profession reflect the same. For me personally – I do need to establish criteria and rules in my testing activities, the ruling I adapt will be fitted (hopefully) to my own specific professional environment. I wish all of us could relate to the new standards as a possible recommendation and not as new religion (we already have too many extremists in this world).
Dani…
I could certainly relate to the standards as a possible recommendation. I’d be happy to do that. However, let’s look at what the standards promoters actually say:
“The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed set of standards for software testing that can be used by any organization when performing any form of software testing.” That’s a pretty bold claim, the very first paragraph of the standard itself. And it gets bolder. Stuart Reid has said, “If testing is your career, this will be your standard – whatever type of testing you do, it will affect you.” (http://archive.newsweaver.com/qualtech/newsweaver.ie/qualtech/e_article00121840164e4.html?x=b11,0,w, accessed September 7, 2014). This sounds a lot like fundamentalism to me.
Meanwhile, Stuart also quotes the (he doesn’t provide a specific citation) saying that standards are “guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognized industrial, professional, trade associations or governmental bodies.” He goes on, presumably in his own words, “Guidelines documents as they are not compulsory unless mandated by an individual or an organization” (sic) and “Agreements because they should reflect a certain level of consensus”. Yet the claim of “non-compulsory” is vulnerable, as the National Institutes of Science and Technology is well aware: “Still another classification scheme distinguishes between voluntary standards, which by themselves impose no obligations regarding use, and mandatory standards. A mandatory standard is generally published as part of a code, rule or regulation by a regulatory government body and imposes an obligation on specified parties to conform to it. However, the distinction between these two categories may be lost when voluntary consensus standards are referenced in government regulations, effectively making them mandatory” standards.” (http://www.nist.gov/standardsgov/definestandards.cfm).
So, on the one hand, the standard is intended to be universally applicable and it is intended to be the standard that you are bound to follow; on the other hand, it’s just a guideline, not mandatory. In other words, “you must follow our prescription, but you don’t have to”. This is tantamount to wanting to suck and blow at the same time. But I’m hearing mostly a sucking sound coming from over the horizon.
So, Dani: when you say I wish all of us could relate to the new standards as a possible recommendation and not as new religion (we already have too many extremists in this world), I agree. But it’s not the context-driven community that is promoting a fundamentalist agenda here: it’s the standards promoters that are doing that.
—Michael B.
Pingback: ISO 29119 – A standard in name only | Brian Osman
Thanks for sharing details. Nice Informative Post.