This post was prompted by an article from 2013 by Mukesh Sharma that Sticky Minds recycled this week. I disagree with much of the article, about exploratory and scripted testing and about the nature of checklists. However, I’m going to restrict myself to Mukesh Sharma’s comments about regulatory compliance, specifically Sarbanes Oxley.
“In such (regulatory) scenarios the reliance on scripted testing is heavy, with almost no room for exploratory testing. Other examples include testing for Sarbanes-Oxley… and other such laws and acts, which are highly regulated and require strict adherence to defined guidelines.”
Let’s be clear. The Sarbanes-Oxley legislation does not mention software testing, never mind prescribe how it should be performed. It does mention testing, but this is the testing that auditors perform. Standards and quality control also feature, but these relate to the work of accountants and auditors.
Nevertheless, compliance with Sarbanes-Oxley does require “strict adherence to defined guidelines” but this is a requirement that is inferred from the legislation and not the law itself. The guidelines with which software testers must comply are locally defined testing policies and processes. Each compliant organisation must be able to point to a document that says “this is how we test here”. The legislation does have plenty to say about guidelines, but these are guidelines for sentencing miscreants. I suppose the serious consequences of non-compliance go a long way to explaining the over-reaction to Sarbanes-Oxley.
I suspect the pattern was that companies and consultants looked at how they could comply by following their existing approach to development and testing, then reinforced that. Having demonstrated that this would produce compliance they claimed that this was the way to comply. Big consultancies have always been happy to sell document heavy, process driven solutions because this gives them plenty of opportunity to wheel out inexperienced, young graduates to do the grunt work tailoring the boiler plate templates and documents.
I used to detest Sarbanes-Oxley, but that was because I saw it as reinforcing damaging practices. I’m still hardly a fan, but I eventually came to take a more considered approach because it doesn’t have to be that way. If you go to look at what the auditors have to say about Sarbanes-Oxley you get a very different perspective. ISACA (the professional body for IT auditors) provides a guide to SOX compliance (free to members only) and it doesn’t mention scripts at all. Appropriate test environments is a far bigger concern.
ISACA’s COBIT 5 model for IT governance (the full model is free to members only) doesn’t refer to manual test scripts. It does require testers to “consider the appropriate balance between automated scripted tests and interactive user testing”. For manual testing COBIT 5 prefers the phrase “clearly defined test instructions” rather than “scripts”. The requirement is for testers to be clear about what will be done, not to document traditional test scripts in great detail in advance. COBIT 5 is far more insistent on the need to plan your testing carefully, have proper test environments and retain the evidence. You have do all that properly, it’s non-negotiable.
COBIT 5 matters because if you comply with that then you will comply with Sarbanes-Oxley. Consultancies who claim that you have to follow their heavyweight, document driven processes in order to comply are being misleading. You can do it that way, just like you could drive from New York to Miami via Chicago. You get there in the end, but there are better ways!
Exploratory testing, Context Driven Testing and Bach & Bolton’s Rapid Test Management are all consistent with the demands of Sarbanes-Oxley compliance provided you know what you’re doing and take the problem seriously, caveats that apply to any testing approach. If anyone tells you that Sarbanes-Oxley requires you to test in a particular way challenge them to quote the relevant piece of legislation or an appropriate auditor’s interpretation. You can be sure that it’s a veiled sales pitch – or they don’t know what they are talking about. Or both perhaps!