This is an email interview I gave to Jeff Bounds of Service Virtualization about the Stop 29119 campaign in October 2014. It appeared in two parts, ”James Christie explains resistance to ISO standards” and “ISO 29119 is damaging, so ignore it, advises James Christie”.
The full interview in the original format follows. Jeff’s questions are in red.
What is ISO 29119?
And why is it important to the software testing field?
ISO 29119 is described by the International Organization for Standardization (ISO) as “an internationally agreed set of standards for software testing that can be used within any software development life cycle or organization” .
When ISO are promoting a standard that is intended to cover everything that testers do then that is a big deal for all testers. We cannot afford to ignore it.
What’s wrong with ISO 29119?
Why do you oppose ISO29119?
I think the question is framed the wrong way round. A standard requires consensus and it has to be relevant. It is up to the promoters of the standard to justify it. They’ve not made any serious, credible attempt to do so. Their interpretation of agreement and consensus is restricted to insiders, to those already in the working group developing the standard. Those testers who don’t believe that formal, generic standards are the way ahead have been ignored.
Even before I found out about ISO 29119 I was opposed to it in principle. Standards in general are a a good thing, but software testing is an intellectual activity that doesn’t lend itself to standardization.
There is a wide range of evidence from pyschology, sociology and management studies to back up the argument that document driven standards like ISO 29119 are counter-productive. They just don’t fit with the way that people think and work in organizations. On the other hand the defenders to standards have never bothered to refute these arguments, or even address them. They simply assert, without evidence, that standardization is good for testing. Typically they make spurious arguments that because standards are a good thing in many contexts then they must be a good thing for testing. It’s logical nonsense.
This is all without considering the detailed content of the standard. It is dated, excessively bureaucratic, prescriptive and badly written.
Sure, the standard does say that it is possible to apply parts of the standard selectively and claim “tailored conformance”. However, the standard requires agreement with stakeholders for each departure from the standard. For any significant project that means documented agreement with many people on all sorts of detailed points.
Dr Stuart Reid has claimed that he wants to see companies and governments mandating the use of ISO 29119 in contracts. Lawyers and procurement managers don’t understand testing, as Dr Reid concedes. He sees that as being a case for providing them with a standard they can mandate.
My perspective is that such people, precisely because they don’t understand testing, will require full compliance. In their eyes, full compliance will seem responsible and professional while tailored compliance will look like cutting corners. That’s the way that people react. It’s no good shrugging that off by saying people don’t have to act that way.
All the evidence supports the opponents because they know how people behave. There is no evidence to support the standards lobby.
Isn’t standardisation good?
An argument in favor of ISO 29119 is that it would bring standardization to a software testing process that historically has seen people using a variety of techniques and methods, rather than one set way of doing things. What’s wrong with that?
Everything. Testing has to fit the problem. It seems crazy to think that everyone should be expected to do the same things. Again, why should that be the case? If everyone is doing the same then most people will be doing the wrong thing.
Are opponents trying to save their jobs?
Some proponents of ISO 29119 could also argue that opponent of the standard are simply trying to save their jobs, when automation and simulation represent a better, faster and cheaper way of doing testing. What are your thoughts about that?
Even if it were the case that opponents were simply concerned about their jobs it would still be a compelling argument against ISO 29119. As the ISO working group has conceded many opponents are more expert than the average tester. Why should they have to change the way they operate, for the worse, or pass up opportunities for work?
I could actually earn more money by collaborating with the standard and cleaning up the mess it will create. There will be a good market for test consultants to do that. However, I am not interested in that sort of work.
Anyway, opponents are unhappy about the standard, not automation and simulation, which are extremely important and valuable at the right time. The standard isn’t based on the assumption of an automated approach and the test process paper (Part 2) doesn’t even mention simulation. The discussion about automation is a quite separate matter from the debate about ISO 29119.
Can ISO 29119 provide a baseline?
Dr. Stuart Reid recently argued that ISO 29119 would, among other things, help define good practices in testing, along with providing a baseline to compare different test design techniques. What are your thoughts about that? (His full argument is here).
I don’t think the standards deals with good testing practices. It advocates what it sees as good practices in test management, specifically documentation. It is really a documentation standard rather than a testing standard. It is a classic case of confusing the process with the real work. The difference is crucial. It is like the difference between the map and the territory. The map is a guide to the territory, but it is not the real thing.
Dr Reid hasn’t argued the point about a baseline. He has merely asserted it without evidence or explanation. I’m afraid that is typical of ISO’s approach. Even if it is so, I don’t think testers should have to tie themselves in knots for the benefit of others.
What is the alternative?
If you believe ISO 29119 isn’t the solution, then what is the best standard for software testing, and why?
As I’ve said above I don’t think a generic standard is appropriate for testing. A good alternative to doing wasteful and damaging things is to ignore them. There are many sound alternatives to ISO 29119. I don’t think it is up to the opponents of the standard to justify these. They are being applied and they work. Where is the evidence that ISO 29119 works?