Periodically I am asked to write for a magazine or blog, and more often than not I agree. Two years ago I was asked by an online magazine to write about my opposition to the ISO 29119 testing standard. I agreed, but they didn’t use my article. I’ve just come across it and decided to post it on my blog. A warning! There’s nothing new here – but the arguments are still strong, relevant, and ISO have neither countered them nor attempted to do so. Clearly they hope to win the debate by default, by lying low and hoping that opponents will give up and be forgotten.
In August 2014 I gave a talk in New York at CAST, conference of the Association for Software Testing. I criticized software testing standards, and the new ISO 29119 in particular.
I thought I would talk for about 40 minutes, then we’d then have an interesting discussion and that might be the end of it. Well, that happened, but it was only the start. My talk caught the mood of concern about the standard amongst context driven testers, and so the Stop 29119 campaign kicked off.
Life hasn’t been quite the same since I acquired a reputation for being anti-standards. My opposition to ISO 29119 has defined my public image. I can’t complain, but I’m slightly uncomfortable with that. I’d rather be seen as positive than negative.
I want to make it clear that I do approve of standards; not all standards, but ones that have been well designed for their particular context. Good standards pool collective wisdom and ensure that everyone has the same understanding of what engineering products and activities should do. They make the economy work better by providing information and confidence to consumers, and protecting responsible companies from unscrupulous competitors. Standards also increase professional discipline and responsibility, and this is where the International Standards Organization has gone wrong with ISO 29119.
The standard defines in great detail the process and the documents for testing, but fails to clarify the purpose of testing, the outcomes that stakeholders expect. To put it bluntly, ISO 29119 is vague about the ends towards which we are working, but tries to be precise about the means of getting there. That is an absurd combination. Obviously stakeholders hope for good news from testers, but what they really need is the unvarnished truth, however brutal that might be.
Remember, it’s not the job of testers to test quality into the product. It’s our job to shine a light on what is there so that the right people can take decisions about what to do next. The outcome of testing isn’t necessarily a high-quality product; there may be valid reasons for releasing a product that looks buggy to us, or it might even make sense to scrap the development. I once saw an 80 person year project scrapped after testing. It’s not our call. The point is that the outcome of our testing must be the best information we can provide to stakeholders. ISO 29119 makes no mention of that. Instead it focuses in minute detail on the process and documentation.
Strict copyright protection means I can’t share the content of ISO 29119, but I can say that the sample Test Completion Reports in the standard epitomise what is wrong. They summarise the testing process with a collection of metrics that say nothing about the quality of the product. A persistent danger of standards and templates is that people simply copy the examples and fill in templates without thinking deeply enough about what is needed on their project.
It would be simple to comply with the ISO 29119 Test Completion Process, and produce a report that provided no worthwhile information at all.
The Institute of Internal Auditors offers a worthwhile alternative approach with their mandatory performance standards, which in striking contrast to ISO 29119 are available to the public for scrutiny and discussion. The section covering audit reports says nothing about the process of reporting, or what an audit report should look like. But it stipulates brief, clear and very demanding requirements about the quality of the information in the report.
The difference between ISO 29119 and internal audit standards is that you can’t produce a worthless audit report that complies with the standard. The outcome of the audit has to be useful information. Why couldn’t testing standards focus on such a simple outcome? Do testers want to be zombies, blindly complying with a standard and failing to think about what our stakeholders need? Or do we want to offer a valuable service?