This is probably the last in the series of articles running up to my upcoming EuroSTAR tutorial ”Questioning auditors questioning testing”.
In my last blog post I explained my concerns about the way that testers have traditionally adopted an excessively positivist attitude, i.e. they conducted testing as if it were a controlled scientific experiment that would allow them to announce definite, confident answers.
I restricted that article to my rejection of the positivist approach. However, I need to follow it up with my explanation of why I can’t accept the opposite position, that of the anti-positivist or interpretivist. The interpretivist would argue that there is no single, fixed reality. Everything we know is socially constructed. Researchers into social activities have to work with the subjects of the research, learning together and building a joint account of what reality might be. Everything is relative. There is no single truth, just truths.
I don’t think that is any more helpful than rigid positivism. This article is about how I groped for a reasonable position that avoided the extremes.
Understanding, not condemning
My experience as an auditor forced me to think about these issues. That shaped my attitudes to learning and uncertainty when I switched to testing.
I’ve seen internal auditors taking what I considered a disastrously unprofessional approach largely because they never bothered to consider such points. Their lack of intellectual curiousity meant that they unwittingly adopted a rigid scientific positivist approach, totally unaware that a true scientist would never start off by relying on unproven assumptions. That rigid approach was the only option they could consider. That was their worldview. Anything that didn’t match their simplistic, binary view was wrong, an audit finding. They’d plough through audits relying to a ludicrous extent on checklists, without acquiring an adequate understanding of the business context they were trampling around in. They would ask questions that required yes/no answers, and that was all they would accept.
The tragedy was that the organisation where I saw this was in fear of corporate audit, and so the perspective of the auditors shaped commercial decisions. It was better to lose money than to receive a poor audit report. This fear reinforced the dysfunctional approach to audit. Sadly that approach deskilled the job. Audits could be performed by anyone who was literate, so it became a low status job. Auditors were feared but not respected, a dreadful outcome for both audit and the whole company.
Prior to this I had worked as an auditor in a company that had adopted a risk based approach before it became the norm. Compliance, and simple checks against defined standards and controls were seen as being interesting, but of limited value. We had to consider the bigger picture, understand the reasons for apparent non-compliance and try to explain the significance of our findings.
The temptations that can seduce a novice
At first I found all this intimidating. The temptation for a novice is to adopt one of two extremes. I could either get too close to the subjects of my audits and accept the excuses of IT management, which would have been very easy given that I’d just moved over from being a developer and expected to resume my career in IT one day. Or I could veer to the other extreme and rely on formal standards, processes and predefined checklists all of which offer the seductive illusion of a Right Answer.
This is essentially the dichotomy between interpretivism and positivism. Please don’t misunderstand me. I am not rejecting either outright. I am not qualified to do that. I just don’t believe that either is a helpful worldview for testers or auditors.
I knew I had to avoid the extremes and position myself in the middle. This was simply being pragmatic. I had to provide an explanation of what was going on, and reach conclusions. This required me to say that certain things could and should be done better. Senior management were not helped by simplistic answers that didn’t allow for the context, but neither were they interested in waffle that failed to offer helpful advice. They were paying me to make informed judgements, not make excuses.
Maybe not “right”, but as good as I can do
It took me a couple of years to feel comfortable and confident in my judgement. Where should I position myself on any particular complicated problem? I never felt convinced I was right, but I learned to be confident that I was taking a stance that was justifiable in the circumstances and reflected the context and the risks. I learned to be comfortable with “this is as good as I can do”, and take comfort that this was far more valuable to my employers than being either a hardline positivist or interpretivist.
As I conducted each audit I would fairly quickly build up a rough picture, a model, of what I thought should be happening. This understanding was always provisional, capable of being tested and revised all the way through the audit in a constant cycle of revision. Occasionally the audit would have to be aborted because it quickly revealed some fundamental problem that we had not envisaged. In such cases there was no point pursuing the detail in the audit plan. It would have been like reviewing the decoration of a house when the roof had been blown off.
On other occasions our findings were radically different from those that we had expected. We had been sent in because of particular concerns, but the apparent problems were merely symptoms of a quite separate problem with roots elsewhere. It was essential that we had the mental flexibility, and the openness to realise that what we thought we knew was wrong.
The dangers of the extremes
I’m not comfortable positioning myself, and the learning approach that we took, in any particular school between positivism and relativism. It would be interesting to pin it down but working out the right label isn’t a priority. The important point, for my attitude towards auditing and testing, is that I have to be wary of the dangers inherent in the extremes.
In testing, the greater and more obvious danger is that of positivism, or rather an excessive regard for its validity and relevance in this context. In auditing, relativism is also a huge danger, and it’s fatal to the independence and credibility of auditors if they start identifying too closely with the auditees. There can be great commercial and organisational pressure to go with the flow. There have been countless examples of internal and external auditors who blew it in this way and failed to say “this is wrong”.
The dangerous temptation of going with the flow also exists in testing. Indeed, it could be argued that the greatest danger of relativism in testing is to accept the naïve positivism embodied in traditional approaches, and to pretend that detailed scripts, and meticulous documentation can convey the Truth against which the testers have to measure the product. That’s an interesting paradox, but not one I intend to pursue now.
Frustration with “faking it”
Anyway, after learning how to handle myself in a professional and enlightened audit department I couldn’t contain my frustration when I later had to deal with blinkered auditors who couldn’t tell the difference between an audit and a checklist. I don’t regard that as real auditing. It’s faking it, in very much the same style that James Bach has described in testing; running through the process, producing immaculate documents, but taking great care not to learn anything that might really matter – or that might disrupt plans, budgets or precious assumptions.
Inevitably, when I switched to testing, and had to endure the problems of the traditional document driven approach I experienced very much the same frustrations that I had had with poor auditing. It was with great relief that I came across the work of the Context Driven School, and realised that testing wasn’t a messed up profession after all; it was just a profession with some messed up practices, that we can go along with or resist. It’s our choice, and the practical and learning experiences that I enjoyed as an auditor meant I could go only one way.