Audit and Agile (part 1)

Training in stuff I know

Last Thursday, October 8th, I went to a training day organised by the Scottish Chapter of ISACA, the international body representing IT governance, control, security and audit professionals. The event was entitled “Audit and Control of Agile Projects”. It was presented by Christopher Wright, who has over thirty years experience in IT audit and risk management.

This is a topic about which I am already very well informed but I was keen to go along for a mixture of reasons. When discussing auditing and the expectations of auditors we are not dealing with absolutes. There is no hard and fast right answer. We can’t say “this interpretation is correct and that one is wrong”. Well, we could, but that ignores the possibility that others might disagree, and they might be auditors who are conducting audits on a basis that we believe to be flawed.

It is therefore important to think deeply about, and understand, the approach that we believe to be appropriate, but also to consider alternative opinions, who holds them and why. Consultants offering advice in this area have to know the job, but they must also be able to advise clients about other approaches. We have to stay in touch with what other people are saying and doing, regardless of whether we agree with them.

I was keen to hear what Christopher Wright had to say, and also to talk to other attendees, who work for a wide range of employers. Happily the event was free to members, which made the decision to attend even easier!

As it transpired I didn’t hear anything from Christopher that was really new to me, but that was reassuring. His message was very much aligned with the advice I’ve been pushing over the last few years to clients, in my writing and in training tutorials. What was also reassuring was that the attendees were receptive to Christopher’s message and there was no sign of lingering attachment to older, traditional ways of working.

I don’t want to cover everything Christopher said. I will just focus on a few points that I thought were key. I will split them into two posts. I will try to make it clear when I am offering my own opinion rather than reporting Christopher’s views.

Agile is potentially an appropriate response to the problems of software development

Firstly, and crucially, agile can be entirely consistent with good governance. The “can be” is an important qualification. You can do agile well or badly. It is not a cop out by organisations that couldn’t make traditional methods work. In many situations it is the most appropriate response. Christopher ran through a quick discussion of the factors involved in making a decision whether to go with an agile or a traditional, waterfall approach. His views were orthodox, current software development thinking rather than a grudging auditor’s acceptance of what has been happening in development circles.

Put simply, Christopher argued that a waterfall approach is fine when the problem is stable and the solution is well understood and predictable. If we know at the outset, with justified confidence, what we are going to do then there’s no problem with waterfall, indeed we might as well use it because it is simpler to manage. We are dealing with a well defined problem and we are not going to be swamped by a succession of changes late in the project. I would argue that that is usually not the case when we are developing new software intended to provide a new solution to a problem.

If the problem is not trivial then it is unlikely that we can understand it fully at the start of the project, and we can only build our understand once the project is well underway. Agile is appropriate in these circumstances. We need to learn what is needed through a process of iteration, experimentation and discovery.

We had a brief discussion about predictable solutions. I would happily have spent much longer mulling over the idea of predictability. My view is that software development has been plagued throughout its history by a yearning for unattainable predictability. That has been damaging because we have pretended that we could predict solutions and project outcomes when the reality has been that we didn’t know, and couldn’t have known at the time. The paradox is that if we try to define a predictable outcome too early that pushes back the point when we truly can predict with confidence what is required and possible. Mary Poppendieck made the point very well back in 2003 with an excellent article “Lean Development and the Predictability Paradox” (PDF, opens in new window).

Agile is scary for auditors

Christopher argued some important points about why many traditional auditors are suspicious of agile, and I was very much in agreement with him. There are many agile methods, and none of them are rigidly defined or standardised. Auditors can’t march into a project with a checklist and say “you’ve not produced a burndown chart in the form prescribed by… ”.

The auditors have to base their work on the Agile Manifesto, but they need to read and understand it carefully. It is not a matter of choosing either working software or comprehensive documentation, but valuing the former over the latter. Auditors have to satisfy themselves that an appropriate balance is being struck, and this requires judgment and experience. The Agile Manifesto can guide them to ask useful questions, but it can’t provide them with answers.

Crucially, auditors have to ask open questions when they are auditing an agile project. This is one of my hobby horses, and I have often written and spoken about it. Auditors must have highly developed questioning skills. They need the soft skills that will allow them to draw out the information they need from auditees. They must not rely on checklists that encourage them to ask simplistic questions that invite simple, binary answers; yes or no.

Christopher told a sadly plausible story of an audit team that reviewed an agile project and produced a report listing 50 detailed problems. The auditors did not have experience with agile and had conducted the audit using a Prince2 checklist that was designed for waterfall projects.

This might seem a ludicrously extreme example, but I’ve seen similar appallingly unprofessional and incompetent behaviour by auditors. I believe that when this happens the best outcome is that the auditors have wasted everyone’s time, failed to do anything useful and trashed their credibility.

What is far more damaging, in my opinion, is a situation where the auditors have real power, regardless of their competence, and auditees start to tailor their work to fit the prejudices and assumptions of the auditors. Managers start to do things they know are wrong, wasteful and damaging because they fear criticism from auditors. Clients become infuriated because the supplier staff are ignoring their needs and focusing on appeasing the auditors.

In this scenario the auditors are taking commercial decisions, but are not accountable for the consequences. They are probably not even aware that this is what they are doing. They lack the experience, knowledge and insight to understand the damage they are doing.

I therefore agree with Christopher that auditing IT projects is no job for the inexperienced. Closed questions are dangerous, and should be used only to confirm information that has already been elicited. The auditor must not ask “do you have a detailed stage plan?”, but ask “can you show me how you plan the work?”. The former question might simply produce a “no”. The latter question will allow the auditor to see, and assess, how the planning is being performed. Auditees are naturally suspicious of auditors and many people will offer as little information as they can. Allowing them to answer with a curt yes or no helps nobody.

Of course, the problem for inexperienced auditors is that if they ask open ended questions they have to understand and interpret the answers then vary their follow up depending on what they are told. That can be difficult. Well, tough. Auditing isn’t meant to be easy. Like testing it should offer valuable information, and redefining the job to make it easy but pointless is deeply unprofessional.

And in part two…

I am splitting this article into two parts and will post the second one as soon as possible. You could easily write a book about this topic, as indeed Christopher has already done. It’s called “Agile Governance and Audit”.

In part two I cover how auditors can work constructively with audit teams, and also discuss testing. I wanted to set the scene in part one before moving on to testing. I don’t think it’s worth treating testing in isolation, and it’s useful to understand first where modern auditors are coming from. The way they should approach testing follows on naturally from the way that they engage with agile in general.


4 thoughts on “Audit and Agile (part 1)

  1. Thank you for posting this. I work in QA at a company that is regularly audited by customers. We prepare for these audits assuming a list of checkboxes and maintain our paperwork with customer audits in mind It sounds like the mindset will need to change on the auditors’ side before we can safely scale back on perfecting the documentation and use this reclaimed time to further explore test scenarios on the software. But do the auditors have incentive to change approaches other than learning from experience that documentation != software quality (necessarily)?

  2. Hi Amanda – as I said, this is interesting.

    I’m writing primarily about internal auditors rather than external auditors (ie the auditors who carry out the formal annual financial audit). Internal auditors are unambiguosly on the same side as internal developers and testers. They have a different perspective, and that’s why they can add value. External auditors are representing the shareholders and other external stakeholders, such as creditors and the tax authorities. That makes their position rather ambiguous, and there’s scope for a conflict of interest. I don’t think that external auditors would usually be taking an interest in individual development projects.

    You seem to be talking about auditors acting on behalf of a client, inspecting a supplier. That is a quite different activity and relationship from an internal audit. It might not even be carried out by professional internal auditors, who would (or should) be working to the standards of the Institute of Internal Auditors. They would be assessing the risks and the controls required, and whether the action being taken was appropriate.

    An inspection by a client would all depend on what the contract stipulated. In my experience contracts are based on what the sales team could negotiate. The sales people are usually incentivised to close the deal, and they focus on price. They would probably concede all sorts of conditions rather than cut the price too far. Likewise, the client would find it easier to get tough inspection conditions than to shave the price.

    The sales team would be long gone (with their bonuses) before the awkward, and possibly expensive implications, of these inspection conditions became apparent. I don’t know how widespread it is, but I have found it infuriating to come along later and find that the sales team had made concessions that were going to make delivering the contract far more difficult than it should have been, and without giving the client any real benefit.

    As a result, “auditors” acting on behalf of a client inspecting the supplier wouldn’t necessarily have the constructive approach that internal auditors should have. There may be an element of point scoring. I would regard that as unprofessional, if they are career, professional internal auditors. They really ought to be open minded and open to evidence based argument.

    I would always recommend going back to read the contract. Often work is done based on a flawed understanding of what the contract requires, which is then distorted further by word of mouth. Sometimes it is possible to comply with the contract by adopting a less painful approach. Focus on what the contract really requires, then argue for the best way to comply. Explain how excessive documentation is a distraction from achieving the best quality. If possible try to get your own internal auditors to back you up, if they are good, constructive internal auditors of course.

    Good auditors want to see evidence that the testers knew what they were going to do, that they have a consistent approach to testing, and that they have sufficient evidence of the actual testing performed to allow the testing to be recreated so that the same conclusion would be reached. That’s good enough for highly regulated industries.

    However, when the contract stipulates something unhelpful then there isn’t much that can be done if the client won’t shift. Whatever the merits, these stipulations become mandatory requirements for the project. I’d want some serious discussions with the sales function in that case.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s