This article originally appeared on the uTest blog on February 23rd 2015.
In August last year the Stop 29119 campaign and petition kicked off at the CAST conference in New York.
In September I wrote on the uTest blog about why the new ISO/IEEE 29119 software testing standards are a danger to good testing and the whole testing profession.
I was amazed at the commotion that Stop 29119 caused. It was the biggest talking point in testing in 2014. Six months on it’s time to look back. What has actually happened?
The remarkable answer is – very little. The Stop 29119 campaigners haven’t given up. There has been a steady stream of blogs and articles. However, there has been no real debate; the discussion has been almost entirely one sided.
There has been only one response from ISO. In September Dr Stuart Reid, the convenor of the working goup that produced the standard, issued a statement attempting to rebut the arguments of Stop 29119. That was it. ISO then retreated into its bunker and ignored invitations to debate.
Dr Reid’s response was interesting, both in its content and the way it engaged with the arguments of Stop 29119. The Stop 29119 petition was initiated by the board of the International Society for Software Testing. ISST’s website had a link to the petition, and a long list of blogs and articles from highly credible testing experts criticising ISO 29119. It is a basic rule of debate that one always tackles an opponent’s strongest points. However, Dr Reid ignored these authoritative arguments and responded to a series of points that he quoted from the comments on the petition site.
To be more accurate, Dr Reid paraphrased a selection of the comments and criticisms from elsewhere, framing them in a way that made it easier to refute them. Some of these points were no more than strawmen.
So Cem Kaner argued that IEEE adopts a “software engineering standards process that I see as a closed vehicle that serves the interests of a relatively small portion of the software engineering community… The imposition of a standard that imposes practices and views on a community that would not otherwise agree to them, is a political power play”..
Dr Reid presented such arguments as “no-one outside the Working Group is allowed to participate” and “the standards ‘movement’ is politicized and driven by big business to the exclusion of others”.
These arguments were then dismissed by stating that anyone can join the Working Group, which consists of people from all parts of the industry. Dr Reid also emphasized that “consensus” applies only to those within the ISO process, failing to address the criticism that this excludes those who believe, with compelling evidence, that ISO-style standardization is inappropriate for testing.
These criticisms had been made forcefully for many years, in articles and at conferences, yet Dr Reid blithely presented the strawman that “no-one knew about the standards and the Working Group worked in isolation”. He then effortlessly demolished the argument that no-one was making.
What of the content? There were concerns about how ISO 29119 deals with Agile and Exploratory Testing. Eg, Rikard Edgren offered a critique arguing that the standards tried but failed to deal with Agile. Similarly, Huib Schoots argued that a close reading of the standards revealed that the writers didn’t understand exploratory testing at all.
These are serious arguments that defenders of the standard must deal with if they are to appear credible. What was the ISO response?
Dr Reid reduced such concerns to bland and inaccurate statements that “the standards represent an old-fashioned view and do not address testing on agile projects” and ”the Testing Standards do not allow exploratory testing to be used”. Again these were strawmen that he could dismiss easily.
I could go on to highlight in detail other flaws in the ISO response; the failure to address the criticism that the standards weren’t based on research or experience that demonstrates the validity of that approach; the failure to answer the concern that the standards will lead to compulsion by the back door; the failure to address the charge from the founders of Context Driven Testing that the standards are the antithesis of CDT; the evasion of the documented links between certification and standards.
In the case of research Dr Reid told us of the distinctly underwhelming claims from a Finnish PhD thesis (PDF, opens in a new tab) that the standards represent “a feasible process model for a practical organisation with some limitations”. These limitations are pretty serious; “too detailed” and “the standard model is top heavy”. It’s interesting to note that the PhD study was produced before ISO 29119 part 3 was issued; the study does not mention part 3 in the references. The study can therefore offer no support for the heavyweight documentation approach that ISO 29119 embodies.
So instead of standards based on credible research we see a search for any research offering even lukewarm support for standards that have already been developed. That is not the way to advance knowledge and practice.
These are all huge concerns, and the testing community has received no satisfactory answers. As I said, we should always confront our opponents’ strongest arguments in a debate. In this case I’ve run through the only arguments that ISO have presented. Is it any wonder that the Stop 29119 campaigners don’t believe we have been given any credible answers at all?
What will ISO do? Does it wish to avoid public discussion in the hope that the ISO brand and the magic word “standards” will help them embed the standards in the profession? That might have worked in the past. Now, in the era of social media and blogging there is no hiding place. Anyone searching for information about ISO 29119 will have no difficulty finding persuasive arguments against it. They will not find equally strong arguments in favour of the standards. That seems to be ISO’s choice.