This is the full text of the email interview I gave to Infoworld.com, which appeared on August 25th. They used only part of it, which was fine by me. I was delighted they approached me and was happy to get any part of my message out.
What don’t you like about ISO 29119? Will this “standard” have much impact anyway?
ISO 29119 puts too much emphasis on process and documentation, rather than the real testing. Of course that is not its purpose, or the intention of the people who have developed it. However, I have seen in practice how people react when they are dealing with a messy, complex problem, and there are detailed, prescriptive standards and processes on hand. They focus on complying with the standard, and lose sight of the real goal. This sort of goal displacement is a familiar problem in many situations. It is frustrating that so many influential people in IT have failed to acknowledge the problem.
This would not matter so much if testing standards were closely aligned with real, valuable testing. However, the emphasis on documentation encourages heavy up-front commitment to producing paper work, at a time when the nature of the problem isn’t yet understood. I’ve worked on too many projects where most of my test management effort went into detailed documents that were of disappointingly little value when it came to test execution.
There is a danger that lawyers, procurement managers and project managers will focus on the standard and the documentation, and insist that it is always produced. They assume that the standard defines best practice and that failure to comply is evidence of a lack of professionalism. ISO 29119 does allow for “tailored compliance”, where testers might opt out of parts of the standard and document their justification. But the promoters of the standard are pushing the line that failure to comply with the standard will leave testers in a difficult position if there are problems. Stuart Reid has gone on the record saying;
“imagine something goes noticeably wrong. How easy will you find it to explain that your testing doesn’t comply with international testing standards? So, can you afford not to use them?”
Faced with that many testers will feel pressured, or will simply be told, that they have to comply with the full standard to cover their asses.
If ISO 29119 is taken up by governments and big companies, and they insist that suppliers have to comply then it will force large parts of the testing profession to work in this document driven way. It will restrict the opportunities for good thoughtful testers, and so I believe it is anti-competitive.
That’s a strong accusation, but ISO knows its standards must be credible, and that they must be able to demonstrate that they are not anti-competitive. That is why they have rules about the consensus that standards require. Consensus is defined by ISO as “general agreement, characterized by the absence of sustained opposition… by any important part of the concerned interests”.
That is why I called for action at CAST 2014. Testers should speak out so that is clear that there isn’t general agreement, that there is sustained opposition, and that thoughtful, professional testers are very much a concerned interest.
Are there any improvements you could suggest?
No, not really. I think the idea of a standardised approach to software testing is misconceived. It is far more useful to talk about practices and guidelines that are useful in particular contexts. That doesn’t require a standard. If guidelines are abstracted to the level that they are always applicable then they are so vague that they become vacuous and don’t offer helpful and practical advice. If you make them practical then they won’t always be applicable.
However, one improvement I would certainly like to see is not in the standard itself. It’s in the way it’s marketed. I really disapprove of the way that it’s being sold as being more responsible than alternatives, and that it can provide a more certain outcome. There’s a strong message coming from suppliers who use ISO 29119, and who contributed to its creation, that compliance with the standard makes them better than their competitors and guarantees a better service.
What is your particular interest in this specification?
I have worked as an IT auditor as well as a tester. When I look at how ISO 29119 is being sold I see an appeal to fear, a message that signing up will protect people if things go wrong. The implied message is that compliance will allow testers to face audits with confidence. I know from my experience that such a standard will encourage too much emphasis on the wrong thing.
Auditors want to see evidence of good and appropriate testing, not necessarily good planning and documentation. They need to see an appropriate process that works for the company, not a generic standard. A project plan is poor evidence that a project was well managed. It’s the results that matter. Likewise heavy advance documentation isn’t evidence that the testing was good. As an ex-auditor the approach of ISO 29119 concerned me.
I am also keen to help stop poor auditors fixing onto ISO 29119 as something they can audit against to make their job easier. Neither testing nor auditing are meant to be easy. They’re meant to be valuable, and document driven standards encourage the illusion that testing can be easier, and performed by less skilled staff, than is really the case.