I am giving a half day tutorial at EuroSTAR this year, so not surprisingly that has forced me to think around the subject, “questioning auditors questioning testing”.
Over the last few weeks I have been struck by the number of times that I have come across one very interesting word – binary.
It’s an important concept, and it is hugely important in both professions. However, I have become increasingly aware that testing and auditing are taking very different approaches to the concept.
Testing and checking
Discussion of binary results in testing is usually tied in with the debate about the distinction between testing and checking. James Bach and Michael Bolton set out the argument clearly here.
The distinction is fundamentally important, but frustratingly the debate hasn’t really got through to the whole of the testing profession.
There are still regiments of testers oblivious to to the distinction, beavering away with detailed test scripts, checking the results. The testing establishment from which these traditional testers take their lead, directly or indirectly, have not engaged with the debate. They have given the unfortunate, and probably accurate, impression that they regard checking and testing as being effectively synonymous in practice.
Reality isn’t binary
Rikard Edgren gave a very good talk on the specific idea of binary opinions at Øredev 2011 and Let’s Test 2012. Here is the Øredev talk.
The slides for Let’s Test are here (opens in new tab). Rikard also wrote a blog on the subject. The key phrase I picked up from Rikard was;
Reality isn’t binary, we can communicate noteworthy information – we don’t know everything in advance.
I’m not going to get further into that debate here. I just want to illustrate the contrast with auditing where Rikard’s comment resonates strongly.
Two types of binary opinion (naturally!)
Firstly, I’d better explain that the type of binary opinions vary depending on whether one is talking about internal or external auditing. In internal auditing they would take the form of pass/fail checking of controls. Are they present? Are they complied with?
Binary opinions in external auditing have historically been largely about the truth and fairness of the company accounts, or about whether the company is a going concern. That has been the core of the external audit report. In recent years there has been the added requirement imposed by the Sarbanes-Oxley Act for US companies to express an opinion on whether the framework of internal controls is effective.
Binary opinions in internal audit – a relic of yesteryear
There isn’t a great deal of debate in internal auditing circles about binary opinions. Traditional internal auditing focused on internal controls. The debate has been held, and the overwhelming consensus, at least in informed circles, is that any audit that offers only binary opinions is hopelessly limited, blinkered and hopelessly outdated.
I like the definition of internal controls from Anthony Catenach.
Internal controls are how management makes sure the company’s business model is operating correctly.
If you view internal controls in that broader perspective then you should be able to see how simple binary opinions are unhelpful. Auditors need to set their findings in context and explain why they are significant and what danger they pose. Simply saying that certain controls are missing, or have not been applied, is unhelpful. That’s not to say that such simplistic audits have vanished.
A couple of weeks ago I was speaking to a friend who works as a developer for a multinational company. He told me that the internal auditors work from a checklist, using questions that require yes/no answers. People are very way of the auditors and answer only direct questions without offering anything more. It horrifies me that auditors should ever accept the answer “yes” or “no” without following up with “why?”.
That sort of auditing is ineffective and unprofessional. I can’t stress strongly enough that audit checklists have a place, but they are not the audit! They are merely the starting point for a conversation.
Previously to illustrate how an audit interview is conducted I have used the analogy of an advocate (barrister or attorney) questioning a witness in court. The advocate cannot know what answer the witness will give and has to vary the follow up questions accordingly, rather than ploughing on with a prepared script. Conducting an audit by checklist is very much like sticking to the script regardless of the answers.
This is now orthodox modern opinion. The opinion formers, the leading lights of the profession know that binary opinions are dated and the debate has moved on to risk; how can auditors inform stakeholders about the risks that matter, the risks that keep them awake at night? How can auditors help management to understand the risks that they are facing and to take decisions that are better informed about the risks?
Regulators and binary opinions in external audit
The debate about binary opinions in internal audit may be largely over but it is still very much alive in external audit. The regulators in the UK and the USA are pushing hard for auditors to provide more useful opinions in their reports rather than relying on simple, and frequently misleading, binary opinions.
The response from the Big 4 audit firms has been cool, but telling the regulators to take a hike is politically tricky! They have to engage with the debate. It’s not good enough for them to defend current practices. The problems with these are glaringly obvious, so they have to respond constructively.
The position is slightly confused by Sarbanes-Oxley’s requirement that external auditors state whether they believe the framework of internal controls is effective. That takes them into internal audit territory, and raises concerns about whether such a judgement can be accurate or helpful. Certainly the experience of recent years isn’t encouraging.
There are countless examples of companies whose accounts have been passed by their external auditors, only to collapse from problems that existed before the audit was conducted. Remember Enron? That debacle led to the demise of one of the world’s biggest firms of accountants, Arthur Andersen. Remember the banks who collapsed? All sailed through their audits, with the auditors picking up multi-million pound fees for offering opinions that proved groundless.
I’m not suggesting that these fees were too high. Perhaps they were too low and worthwhile audits and opinions would be more expensive. However, I am saying that the current reporting regime, with too much emphasis on binary opinions, provides lousy value for money. That is not a minority view. It is the view of the regulators in the UK and USA. It will be interesting to see where the EU moves in this regard.
Testers are not alone
This is far too big and complex an area for me to cover in any detail either now or in my tutorial at EuroStar, even if it is of any interest to any testers except me! However, I think it’s important to understand that there is a big and influential profession wrestling with some of the issues facing testers.
Auditors have to think about how they work, what value they provide, what they should look for, what knowledge they can reasonably provide. Indeed, the more thoughtful auditors are thinking about what knowledge means in their context, how they can “know” things, what constitutes evidence and opinion.
This is epistemology, and it is fascinating. Thinking about this is not some esoteric academic exercise. If we are not clear about what we can know and how we should investigate and report on the knowledge that is available then the danger is that we will end up just faking the whole exercise. We will continue to dress up subjective opinons as “objective” binary verdicts; “yes” this is ok, “no” it isn’t.
Reality doesn’t become clearer simply by pretending that it can be reduced to binary opinions. Quite the reverse, messy reality is obscured by a binary approach. Auditors know that, or at least the clever ones do. There are plenty of smart and capable auditors out there, trying to make sense of what is going on.
The good ones are natural allies of good testers. Seek them out and make them your allies. As for the bad ones, well they are still around as my friend can testify. Their approach is inept and unprofessional. It might not be wise to use these words! It might be interesting to ask them some difficult questions about how they can square their approach with the views of the auditing establishment, the professional bodies and the regulators.
It’s a pity that the self appointed testing establishment, ISTQB and ISO, can’t take a similarly clear line. Sadly their silence effectively endorses binary opinions. Self appointed shouldn’t mean self interested.