Testing standards? Can we do better?

At EuroSTAR 2013 I had a brief disagreement about software testing standards with Stuart Reid. To be more accurate, I was one of a group of sceptics pressing Stuart, who was putting up a battling defence of standards. He has been working on the new standard ISO 29119 and made a very interesting and revealing point. He insisted that the critics of standards don’t understand their true nature; they are not compulsory.

The introduction to standards makes it clear that their use is optional. They become mandatory only if someone insists that they must be applied in a certain context, often by writing them into a contract or a set of in-house development standards. Then, and only then, is it appropriate to talk about compulsion. That compulsion comes not from the standard itself, but from the contract or the managerial directive.

I found that argument unconvincing. Indeed I thought it effectively conceded the argument and amounted to no more than a plea in mitigation rather than a plausible defence.

Even a cursory analysis of this defence reveals that it is entirely specious, merely a statement of the obvious. Of course it is a choice made by people to make standards mandatory, but that choice is heavily influenced by the quite inappropriate status of IEEE 829 and, in all likelihood ISO 29119, as standards. Calling them standards gives them a prestige and authority that would be missing if they were called guidelines. The defenders of standards usually want it both ways. They refer to standards when they are making an implicit appeal to authority. They refer to the standards as guidelines when they are on the defensive. That doesn’t wash. Standards and guidelines are not synonymous.

Stuart’s defence struck me as very interesting because it was entirely consistent with what I have long believed; the rationale behind standards, and their implicit attraction, is that they can be given mandatory status by organisations and lawyers with a poor grasp of software testing.

The standards become justified by the mandatory status assigned to them by non-testers. The justification does not come from any true intrinsic value or any wisdom that they might impart to practitioners. It comes from the aura of the word “standard” and the creators of standards know that this gives them a competitive advantage.

Creating standards is a commercial activity

Standards are not produced on a disinterested “take it or leave it” basis. They do not merely offer another option to the profession. Standards are created by people from the companies who will benefit from their existence, the companies who will sell the services to implement the new standard. In my experience heavyweight, document-driven processes require large numbers of expensive consultants (though not necessarily highly skilled consultants). Creating standards is a commercial activity. The producers of standards are quite consciously creating a market for the standards.

If the creators of standards were merely expanding the market to create a profitable niche for themselves that might not be a big deal. However, the benefit that accrues to them comes at the the expense of everyone else.

It comes at the expense of the testers who are frequently committed to following inappropriate and demoralising practices.

It comes at the expense of their employers who are incurring greater and unnecessary costs for results that are poorer than they need be.

It comes at the expense of the whole testing profession. The standards encourage a dangerous illusion. They feed the hunger to believe, against all the evidence, that testing, and software development in general, are neat, essentially linear activities that can be be rendered orderly and controllable with sufficient advance documentation. Standards feed the illusion that testing can be easier than it really is, and performed by people less skilled than are really needed.

As I said in my EuroSTAR tutorial last week, testing is not meant to be easy, it’s meant to be valuable.

Good contracts or bad contracts?

It is understandable that the contract lawyers find standards attractive. Not only do standards offer the lawyers the illusion that they promote high quality and define the correct way for professionals to work, they also offer the lawyers something they can get their teeth into. A standard makes it easier to structure a contract if you don’t know about the subject area. The standard doesn’t actually have to be useful. The point is that it helps generate deliverables along the way, and it requires the testers to work in a way that is easy to monitor.

Contracts are most useful when they specify the end, or the required value; not when they dictate how teams should reach the destination. Prescriptive contracts can turn unwarranted assumptions about the means into contractually mandatory ends.

I once faced what looked like a horrendously difficult challenge. I had to set up a security management process for a large client, who wanted assurance that the process would work effectively from the very start. This had been interpreted by my employer as meaning that the client required a full-scale, realistic test, with simulated security breaches to establish whether they would be detected and how we would respond. This would have been very difficult to arrange, and extremely expensive to carry out. Failure to deliver on the due date would have resulted in heavy weekly penalties until we could comply. However, the requirement was written into the contract so I was told we would have to do it.

I was sceptical, and went back to the client to discuss their needs in detail. It turned out that they simply needed to be reassured that the process would work, smoothly and quickly. Bringing together the right people from the client and supplier for a morning to walk through the process in detail would do just as well, at a tiny fraction of the cost. Once I had secured the client’s agreement it was straightforward to have the contract changed so that it reflected where they really wanted to end up, rather than stipulating a poorly understood route to that destination.

On many other occasions I have been stuck with a contract that could not be changed and where it was mandatory for testers to comply with milestones and deliverables that had minimal relevance to the real problem, but which required such obsessive attention that they detracted from the real work.

Software testing standards encourage that sort of goal displacement; management attention is directed not at the work, but at a dubious abstract representation of the work. Their attention is directed to the map, and they lose sight of the territory.

We can do better

Sure, no-one has to be a sucker. No-one has to buy the snake oil of standards, but caveat emptor (let the buyer beware) is the legal fallback of the huckster. It is hardly a motto to inspire. Testers can do better than that.

What is the answer? Unfortunately blogs like this preach largely to the converted. The argument against standards is accepted within the Context Driven School. The challenge is to take that argument out into the corporations who are instinctively more comfortable, or complacent, with standards than with a more flexible and thoughtful approach.

I tried to challenge that complacency in my EuroSTAR tutorial, “Questioning auditors questioning testing”. I demonstrated exactly why and how software testing standards are largely irrelevant to the needs of the worldwide Institute of Internal Auditors and also the Information Systems Audit and Control Association. I also explained how more thoughtful and effective testing, as promoted by the Context Driven School, can be consistent with the level of professionalism, accountability and evidence that auditors require.

If we can spread the message that testing can be better and cheaper then corporations might start to discourage the lawyers from writing damaging contracts. They might shy away from the consultancies offering standards driven processes.

Perhaps that will require more than blogs, articles and impassioned conference speeches. Do we need a counterpart to testing standards, an anti-standard perhaps? That would entail a clearly documented explanation of the links between good testing practices and governance models.

An alternative would have to demonstrate how good testing can be accountable from the perspective of auditors, rather than merely asserting it. It would also be directed not just at testers, but also at auditors to persuade them that testing is an area where they should be proactively involved, trying to force improvements. The testers who work for consultancies that profit from standards will never come on board. The auditors might.

But whatever form such an initiative might take it must not be called a standard, anything but that!

Edit. A petition has been set up in August 2014 to calling for ISO to withdraw ISO 29119 on the ground that it lacks the consensus that its own rules requires. Consensus is defined in ISO/IEC Guide 2:2004 as follows.

“Consensus: General agreement, characterized by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments.”

The petition argues, correctly, that there is no consensus. Further, the process did not seek to take into account the views of all parties concerned. The standard reflects one particular view of how testing should be conducted and marginalises those who disagree. If governments and companies insist that ISO 29119 should be adopted, and that suppliers should comply, this will have a dramatic, and damaging, effect on testing and our careers.

I urge all testers to sign the petition.

14 thoughts on “Testing standards? Can we do better?

  1. James–

    Terrific post. A few offerings:

    1) Apropos of the very slipper “voluntary” nature of standards, let’s see what the NIST has to say: http://nist.gov/standardsgov/standards-in-regulations.cfm.

    2) It might be important to note when we support standards that are genuinely helpful. For example, there are lots of hardware standards. A protocol specification is a standard. HTML 5 is a standard. As someone who has driven in Canada and Ireland, I like that whole worldwide-brake-pedal-on-the-left thing in the rental cars. Good manners are standards, and it’s easy to believe that many ISO standards are indeed helpful, especially for physical interfaces between products or when we know something must be done in a specific way or to verify specific facts.

    3) Standards for physical, tangible, material things are relatively easier to interpret and to measure. Standards for more generalized tasks or behaviours or polimorphic actions are subject to a far greater degree of interpretation, as are the actions themselves. In software development, those actions are responding to highly variable events, politics, emotions, calculations of cost and value, explicit desires, implicit requirements, tacit knowledge.

    —Michael B.

    • Thanks Michael. That NIST link is certainly interesting.

      I agree that standards are a good thing. That’s one of the reasons I object to testing standards, which do not qualify as standards by any useful definition. If they were touted as guidelines they would have little commercial value.

  2. Excellent analysis James and having standards for a discipline like software testing is not easy. I started my life as a Mechanical Engineer and then became software tester and can say with certainty that standards work better in the physical engineering world (as Micheal also mentioned). I was once in love with ISO 9000 and also implemented it for a software organization. The result was that it was never helpful.

    That is why in software testing (and software development world in general), standards are not doing much good and frameworks like SCRUM or others are doing good.

  3. Hi James,

    1. I noticed an inconsistency in your post.
    – First you proclaim that you belong to “Context Driven School of Testing”
    – Then you provide context insensitive analysis of testing standards.
    Apparently in your context, testing standards have no value.
    However quite a few people out there believe that such standards have value to them.
    Hint: It’s a “Value to some person”. 🙂

    2. It doesn’t mean that software testing standards do not have value in case neither you nor Stewart Reid can explain it.

    Yury Makedonov

    • Thanks for commenting Yury. It is good to have dissenting opinions here. I want people who disagree with me to read my blog, not just those who already agree.

      I don’t claim to belong to the Context Driven School. I am certainly a supporter and advocate of that school, however.

      You make a fair point about standards having possible value in some contexts. I need to clarify my position.

      My main objection to IEEE 829 and ISO 29119 is their status as standards, for reasons that I explained above, and which I cover in greater depth here, in another blog post, and here in considerable detail.

      Even if the content were uniformly excellent I would still object to their being described as standards. In the vanishingly unlikely event that Michael Bolton and James Bach were ever to turn Rapid Test Management into a standard I would still object, even though I agreed with the content.

      IEEE 829, and probably ISO 29119, have some elements that will be useful in certain contexts. I fully accept that. If you want to follow the sort of linear, mechanical and document driven approach that these standards imply then there will be good content in there. However, I don’t think that that approach is a good one, and certainly not a universally valuable one.

      Standards certainly have value to the people who produce them. I don’t dispute that. However, I do think it is important to point out that the value is commercial, and not because the standards point the way to better testing.

  4. Hi James,

    I apologize for not being clear enough.
    I believe that in addition to people who make their living by selling and implementing standards, these standards also have value for people who actually use them.
    You need to be a real genius to sell something nobody needs.

    Apparently these standards are valuable for many “Standards Driven” organizations and this situation supports flourishing standards offering.

    It seems that the same organization value metrics, best practices and certifications.
    This demand leads to quite a few people working full time to deliver metrics, best practices and certifications.

    It’s a free market with balance demand and supply. 🙂


    • I don’t think you need to be a genus to sell something nobody needs. There are are examples all over the place that prove that not to be true.

      Lots of sales and marketing create the need, usually by convincing their target audience that it will do something to improve there live, or make it easier, or make them have to think less, or even do nothing at all.

      This created market is ultimately because we as humans are weak and easly manipulated 🙂

      P.S. James great post thank you.

      • Thanks Darren. I agree with you that one doesn’t have to be a genius to sell something that nobody really needs. The customer has to want it, to believe that they need it, but that’s not the same thing.

        I agree with Yury that there is a demand for standards, metrics and certifications. That demand exists regardless of the value of what is offered. Yury uses a very interesting phrase; “standards driven organisations”. I think such organisations do exist, and they tend to be large. Therefore what they do is important, because there is money, prestige and influence at stake. However, such organisations are in danger of losing sight of what they should be trying to do. The standards are not an end, they are merely the means. I think the danger of goal displacement is very real. It can be better to do bad work that complies with the standard rather than good work in a non-standard way.

  5. Pingback: Five Blogs – 15 November 2013 | 5blogs

  6. Pingback: Report from EuroSTAR 2013 | Let's Talk Testing

  7. Hi,

    very good post! Standards in technical context are very good to know how it should work and what are the requirements. I have really trouble with standards for “work-processes” (didn’t find a better word).

    An example: In food industry you have many standards WHAT you need to fulfill to get things produced, tested and also into the market. They HOW is most times not defined because your INDIVIDUAL production process can’t be a standard and quality checking is in that industry a “standard” but everybody has his own process definition for it – because every company is different, every product could be different etc. They only have the same metric at the end.

    Testing is the same: It is different and I am not a fan of blocking intentions from testers with standards… sometimes they have a better understanding about the product and who would helps it if we fulfilled a standard but you can’t use the product?


  8. Pingback: Solving the real problem | Let's Talk Testing

  9. Pingback: Testing Bits – 11/10/13 – 11/16/13 | Testing Curator Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.