Training in stuff I know
Last Thursday, October 8th, I went to a training day organised by the Scottish Chapter of ISACA, the international body representing IT governance, control, security and audit professionals. The event was entitled “Audit and Control of Agile Projects”. It was presented by Christopher Wright, who has over thirty years experience in IT audit and risk management.
This is a topic about which I am already very well informed but I was keen to go along for a mixture of reasons. When discussing auditing and the expectations of auditors we are not dealing with absolutes. There is no hard and fast right answer. We can’t say “this interpretation is correct and that one is wrong”. Well, we could, but that ignores the possibility that others might disagree, and they might be auditors who are conducting audits on a basis that we believe to be flawed.
It is therefore important to think deeply about, and understand, the approach that we believe to be appropriate, but also to consider alternative opinions, who holds them and why. Consultants offering advice in this area have to know the job, but they must also be able to advise clients about other approaches. We have to stay in touch with what other people are saying and doing, regardless of whether we agree with them.
I was keen to hear what Christopher Wright had to say, and also to talk to other attendees, who work for a wide range of employers. Happily the event was free to members, which made the decision to attend even easier!
As it transpired I didn’t hear anything from Christopher that was really new to me, but that was reassuring. His message was very much aligned with the advice I’ve been pushing over the last few years to clients, in my writing and in training tutorials. What was also reassuring was that the attendees were receptive to Christopher’s message and there was no sign of lingering attachment to older, traditional ways of working.
I don’t want to cover everything Christopher said. I will just focus on a few points that I thought were key. I will split them into two posts. I will try to make it clear when I am offering my own opinion rather than reporting Christopher’s views.
Agile is potentially an appropriate response to the problems of software development
Firstly, and crucially, agile can be entirely consistent with good governance. The “can be” is an important qualification. You can do agile well or badly. It is not a cop out by organisations that couldn’t make traditional methods work. In many situations it is the most appropriate response. Christopher ran through a quick discussion of the factors involved in making a decision whether to go with an agile or a traditional, waterfall approach. His views were orthodox, current software development thinking rather than a grudging auditor’s acceptance of what has been happening in development circles.
Put simply, Christopher argued that a waterfall approach is fine when the problem is stable and the solution is well understood and predictable. If we know at the outset, with justified confidence, what we are going to do then there’s no problem with waterfall, indeed we might as well use it because it is simpler to manage. We are dealing with a well defined problem and we are not going to be swamped by a succession of changes late in the project. I would argue that that is usually not the case when we are developing new software intended to provide a new solution to a problem.
If the problem is not trivial then it is unlikely that we can understand it fully at the start of the project, and we can only build our understand once the project is well underway. Agile is appropriate in these circumstances. We need to learn what is needed through a process of iteration, experimentation and discovery.
We had a brief discussion about predictable solutions. I would happily have spent much longer mulling over the idea of predictability. My view is that software development has been plagued throughout its history by a yearning for unattainable predictability. That has been damaging because we have pretended that we could predict solutions and project outcomes when the reality has been that we didn’t know, and couldn’t have known at the time. The paradox is that if we try to define a predictable outcome too early that pushes back the point when we truly can predict with confidence what is required and possible. Mary Poppendieck made the point very well back in 2003 with an excellent article “Lean Development and the Predictability Paradox” (PDF, opens in new window).
Agile is scary for auditors
Christopher argued some important points about why many traditional auditors are suspicious of agile, and I was very much in agreement with him. There are many agile methods, and none of them are rigidly defined or standardised. Auditors can’t march into a project with a checklist and say “you’ve not produced a burndown chart in the form prescribed by… ”.
The auditors have to base their work on the Agile Manifesto, but they need to read and understand it carefully. It is not a matter of choosing either working software or comprehensive documentation, but valuing the former over the latter. Auditors have to satisfy themselves that an appropriate balance is being struck, and this requires judgment and experience. The Agile Manifesto can guide them to ask useful questions, but it can’t provide them with answers.
Crucially, auditors have to ask open questions when they are auditing an agile project. This is one of my hobby horses, and I have often written and spoken about it. Auditors must have highly developed questioning skills. They need the soft skills that will allow them to draw out the information they need from auditees. They must not rely on checklists that encourage them to ask simplistic questions that invite simple, binary answers; yes or no.
Christopher told a sadly plausible story of an audit team that reviewed an agile project and produced a report listing 50 detailed problems. The auditors did not have experience with agile and had conducted the audit using a Prince2 checklist that was designed for waterfall projects.
This might seem a ludicrously extreme example, but I’ve seen similar appallingly unprofessional and incompetent behaviour by auditors. I believe that when this happens the best outcome is that the auditors have wasted everyone’s time, failed to do anything useful and trashed their credibility.
What is far more damaging, in my opinion, is a situation where the auditors have real power, regardless of their competence, and auditees start to tailor their work to fit the prejudices and assumptions of the auditors. Managers start to do things they know are wrong, wasteful and damaging because they fear criticism from auditors. Clients become infuriated because the supplier staff are ignoring their needs and focusing on appeasing the auditors.
In this scenario the auditors are taking commercial decisions, but are not accountable for the consequences. They are probably not even aware that this is what they are doing. They lack the experience, knowledge and insight to understand the damage they are doing.
I therefore agree with Christopher that auditing IT projects is no job for the inexperienced. Closed questions are dangerous, and should be used only to confirm information that has already been elicited. The auditor must not ask “do you have a detailed stage plan?”, but ask “can you show me how you plan the work?”. The former question might simply produce a “no”. The latter question will allow the auditor to see, and assess, how the planning is being performed. Auditees are naturally suspicious of auditors and many people will offer as little information as they can. Allowing them to answer with a curt yes or no helps nobody.
Of course, the problem for inexperienced auditors is that if they ask open ended questions they have to understand and interpret the answers then vary their follow up depending on what they are told. That can be difficult. Well, tough. Auditing isn’t meant to be easy. Like testing it should offer valuable information, and redefining the job to make it easy but pointless is deeply unprofessional.
And in part two…
I am splitting this article into two parts and will post the second one as soon as possible. You could easily write a book about this topic, as indeed Christopher has already done. It’s called “Agile Governance and Audit”.
In part two I cover how auditors can work constructively with audit teams, and also discuss testing. I wanted to set the scene in part one before moving on to testing. I don’t think it’s worth treating testing in isolation, and it’s useful to understand first where modern auditors are coming from. The way they should approach testing follows on naturally from the way that they engage with agile in general.