On August 12th I gave a talk at CAST 2014, the conference of the Association for Software Testing (AST) in New York, “Standards; promoting quality or restricting competition.” It was mainly about the new ISO 29119 software testing standard, though I also wove in arguments about ISTQB certification.
I was staggered at the response. Iain McCowatt asked what we should do in response. Karen Johnson proposed a petitition, which subsequently became two. Iain set up a petition through the International Society for Software Testing (ISST), directly targetted at ISO.
Karen’s petition is a more general manifesto for all professional testers to sign if they agree with its stance on certification and standards.
I strongly commend both the petition and the manifesto to all testers.
Eric Proegler also set up an AST special interest group to monitor and review the issue.
This action was not confined to the conference. In the last three weeks there has been a blizzard of activity and discussion on social media and throughout the testing community. Many people have blogged and spoken out. I gave a brief interview to Infoworld.com, and wrote an article for the uTest blog.
Huib Schoots has drawn together many of the relevant articles on his blog. It’s a valuable resource.
However, my own blog has been embarrassingly silent. I’ve not had time, till now, to write anything here. I’ve seen a big rise in traffic as people have hunted down articles I have previously written. August was the busiest month since I started the blog in 2010.
Significant and sustained opposition
Finally I have had a chance to set some thoughts down. I never expected my talk to get such a huge reaction. The response has not been because of what I said. It has happened because it came at the right time. I caught the mood. Many influential people, whom I respect, realised that it was time to speak out, and to do it in unison.
There has been significant and sustained opposition to ISO 29119 as it has developed over the years. However, it has been piecemeal. Individuals have argued passionately, authoritatively and persuasively against the standard, and they have been ignored.
The most interesting response since my talk, however, has been from the dog that didn’t bark, ISO. Neither ISO nor the 29119 working group has come out with a defence of the standard or the way that it was developed.
They have been accused of failing to provide a credible justification for the standard, of ignoring and excluding opponents, and of engaging in rent-seeking by promoting a factional interest as a generic standard.
And their response? Silence. In three weeks we have heard nothing. In their silence ISO have effectively conceded all the points that the Stop 29119 campaigners have been arguing.
There have been some sporadic and entirely unconvincing attempted defences of the standard, mixed up the with odd spot of risibly offensive ad hominem attacks. Collectively the weakness of these defences exposes ISO 29119 for the sham that it is.
Defence #1 – “It’s a standard”
This is possibly the weakest and most wrong-headed argument. ISO are trading on their brand, their image as a disinterested promoter of quality. It is disappointing how many people accept anything that ISO does at face value.
It is important that promoters of any standard justify it, demonstrate that it is relevant to those who will have to use it and that they enjoy a consensus amongst that community. In practice, ISO can spin the argument around.
Once ISO anoints a document with the magic word “standard” too many people suspend their own judgement. They look for opponents to justify their opposition, with reference to the detail they object to in the standard.
In the absence of such detailed arguments against the standard’s content people feel content with saying “it’s a standard, therefore it is a good thing”. That argument might impress those who know nothing about testing or ISO 29119. It lacks any credibility amongst those who know what they are talking about.
Defence #2 – “It’s better than nothing”
Whoops. Sorry. I have to backtrack rapidly. This argument is even worse than the last one. Something that is lousy is emphatically not better than nothing. Just doing something because it’s better than doing nothing? Oh dear. It’s like being lost in a strange city, without a map, and putting a blindfold over your eyes. Well, why not? You’re lost. You don’t know what you’re doing. Blindfolding yourself has to be better than nothing. Right? Oh dear.
This is the politicians’ fallacy, as explained in the classic comedy “Yes Minister”. Thanks to Scott Nickell for reminding me about it (see comments below).
If an organisation’s testing is so disorganised and immature that ISO 29119 might look like the answer then it is ripe ground for a far better approach. Incompetence is not a justification for wheeling in ISO 29119. It is a justification for doing a better job. Next argument please.
Defence #3 – “ISO 29119 doesn’t have to be damaging. It’s possible to work around it.”
The arguments in favour of ISO 29119 aren’t really getting a whole lot more convincing. Sure, you might be able to mitigate some of the worst effects by tailoring your compliance, or ignoring parts that seem particularly unhelpful. However, that is putting effort into ensuring you aren’t worse off than you are if you do nothing.
Also, if standards and rules are so detailed and impractical that they cannot be followed in full then it exposes people to arbitrary enforcement. Once users start tailoring the standard they will leave themselves open to the accusation that they did not comply in full. If things go well there will be no thanks for tailored compliance.
If there are problems, and there are always problems, then any post mortem will reveal that the standard wasn’t followed in full. The implication will be that the problems were caused by deviation, not that the deviation was the only reason anything worthwhile was achieved. Testers and developers will rightly fear that retribution will be arbitrary and quite unrelated to the level of care and professionalism they brought to their work.
Further, ISO 29119 is being marketed with an appeal to fear, as a way of protecting individuals’ backsides if things go wrong. If managers buy into the standard on that basis, are they really likely to take a chance with tailoring the standard?
ISO 29119 is also being pushed to buyers who don’t understand what good practice in testing means. Are they not likely to play safe by insisting in contracts and internal standards that ISO 29119 is complied with?
No, the possibility that we can “work around it” is not a credible argument in favour of ISO 29119.
Defence #4 – “The dissenters are just self-interested”
The standard is apparently a threat to our “craft” and to our businesses. Well, there’s more to it than that, but even if that were the only objection it would still be an entirely valid one. We have argued strenuously that the standard is anti-competitive. A riposte that we fear it will hit us financially is merely to concede the argument in a graceless way.
Anyway, if I were chasing the money I could have made a lot more, an awful lot more by taking lucrative test management contracts to do poor quality work managing the testing process on dysfunctional projects.
I can do great documentation. I am literate, organised and have the knack of getting tuned into corporate politics. My ability to churn out high quality, professional looking documents that would get the money in from clients was a great asset to my last employer.
Testing? Sorry, I forgot about that. I thought we were talking about documentation and money.
Defence #5 – “They’re whingers who wouldn’t get involved.”
This point ignores the argument that the ISO process for developing standards is carefully designed to exclude those who disagree. Again, the “whingers” line spins the argument around. It is not our responsibility to justify our failure to get involved with rent seekers and try to persuade their committees to desist.
I have seen the ISO 29119 working group’s schedule for meetings worldwide over the last few years. In the unlikely event that I would have been allowed to join, my expenses would have wiped out a huge slice of my income. It would certainly have cost me far more than we’ve spent on family holidays in that period. And for what? I’d have sacrificed all that time and money in order to be ignored. Those gallant souls who have tried to fight from the inside have been ground down and spat out by the system. That’s how the system works, how it is intended to work.
No, I didn’t get involved. I had better things to do with my time. In any case, it seems that my campaigning from the outside has not been ignored!
So what now?
We keep going. All this activity has just been the start. ISO is not going to withdraw the standard. However, the blogs, articles and petititions lay down a marker. It shows that the standard was not developed according to any plausible definition of consensus, and that it lacks credibility.
The opposition will strengthen the resolve of testers who wish to stop their organisations buying into the standard. It will provide ammunition to those who want to persuade lawyers and purchasing managers from requiring compliance to ISO 29119.
This one is going to run. We are not going away, and if ISO continue to ignore us then they will simply make themselves look foolish. Do they care? Or are they confident they have enough power and can just sail on regardless?