What does the audit profession have to say about testing standards?

Yesterday I attended a half day meeting of the Scottish Testing Group in Edinburgh. As ever it was an enjoyable and interesting event that allowed me to catch up with people, meet new folk and listen to interesting talks.

The first talk was by Steve Green. He started with a brisk critique of the ISO 29119 software testing standard, then followed up with his vision of how testing should be performed, based on an exploratory approach.

Steve’s objections to ISO 29119 were as follows.

1. It is based on a flawed assumption that testing is a mature profession.
2. The unjustified claim by ISO that consensus has been achieved.
3. The illusion that burying people in documentation produces “thoroughness” in testing.
4. It is a documentation standard, not a testing standard.
5. It will be mandated by organisations that do not understand testing, indeed it is targetted at such organisations.
6. Exploratory testing is misunderstood and misrepresented.
7. It reinforces the factory model of testing. It recommends the separation of test design from test execution. There is no feedback loop.
8. ”It emphasises ass coverage over test coverage” (Michael Bolton).

After this section of the talk Steve took questions, and I joined him in answering a few. The question that intrigued me during Steve’s talk followed my observation that when you plough through the detailed guidance from ISACA (Information Systems Audit & Control Association) and the IIA (Global Institute of Internal Auditors) there is nothing whatsoever to support the use of ISO 29119, IEEE 829 or testing standards in general.

One tester asked if there was anywhere she could go that authoritatively demonstrated the lack of a link. Is there anything that maps between the various documents demonstrating that they don’t align? In effect she was asking for something that says ISO 29119 requires A, B & C, but ISACA & IIA do not require them, and say instead X, Y & Z.

There isn’t such a reference point, and that’s hardly surprising. Neither ISACA nor IIA have anything to say about the relevance of astrology to testing either. They simply ignore it. The only way to establish the lack of relevance is to plough through all the documentation and see that there is nothing to require, or even justify, the use of ISO 29119.

I’ve done that for all the ISACA and IIA material as preparation for my tutorial on auditors and testing at Eurostar last year, and I’ve just checked again prior to repeating the tutorial at Starwest next week.

However, it’s one thing sifting through all the guidance to satisfy myself and be able to state confidently that testing standards are irrelevant to auditors. It’s another matter to demonstrate that objectively to other people. It’s harder work to prove a negative than it would be to prove that there were links. It might be worthwhile taking the work I’ve already done a bit further so that testers like the one who asked yesterday’s question could say to managers “ISO 29119 is irrelevant to the auditors, and here’s the proof”. I intend to do that.

ISACA’s guidance material is particularly important because it’s all tied up in the COBIT 5 framework, which is aligned to the requirements of Sarbanes-Oxley (SOX). So if you comply with COBIT 5 then you can be extremely confident that you comply with SOX.

Maybe it’s because I was an auditor and I have an inflated sense of the importance of audit, but I think it was extremely ill-advised for ISO to develop a software testing standard while totally ignoring what ISACA were doing with COBIT 5. The ISO working group might argue that ISO 29119 is new and that the auditors have not caught up. However, both the high level principles of COBIT 5 and the IIA guidance, and the detailed recommendations within COBIT 5 are all inconsistent with the approach that ISO 29119 takes.

To quote ISACA;

“COBIT … concentrates on what should be achieved rather than how to achieve effective governance, management and control….

Implementation of good practices should be consistent with the enterprise’s governance and control framework, appropriate for the organisation, and integrated with other methods and practices that are being used. Standards and good practices are not a panacea. Their effectiveness depends on how they have been implemented and kept up to date. They are most useful when applied as a set of principles and as a starting point for tailoring specific procedures. To avoid practices becoming shelfware, management and staff should understand what to do, how to do it and why it is important.”

That quote comes from the introduction to COBIT 4.1, but the current version moves clearly in the direction signposted here.

You could comply with ISO 29119 and be compliant with COBIT 5, but why would you? You can achieve compliance with COBIT 5 without the bureaucracy and documentation implied by the standard, and astute auditors might well point out that excessive and misplaced effort was going into a process that produced little value.

It’s not that either ISACA or IIA ignore ISO standards. They refer to many of them in varying contexts. They just don’t refer to testing standards. Why? The simple answer is that ISO have taken years to produce a brand new standard that is now utterly irrelevant, to testers, developers, stakeholders, and auditors.