Presentations

This page will carry slide decks for presentations I’ve given at conferences – as I get round to adding them. All the presentations are in PDF format and open in new tabs.

Farewell to “pass or fail”?

I gave this presentation, “Farewell to ‘pass or fail’” at Expo:QA 2014 in Madrid in May 2014. It shows how auditing and software testing have faced similar challenges and how each profession can learn from the other.

Standards – promoting quality or restricting competition?

This presentation is the one that attracted so much attention and controversy at CAST 2014 in New York. It is a critique of software testing standards from the perspective of economics.

ISO – the dog that hasn’t barked

On August 12th I gave a talk at CAST 2014, the conference of the Association for Software Testing (AST) in New York, “Standards; promoting quality or restricting competition.” It was mainly about the new ISO 29119 software testing standard, though I also wove in arguments about ISTQB certification.


I was staggered at the response. Iain McCowatt asked what we should do in response. Karen Johnson proposed a petitition, which subsequently became two. Iain set up a petition through the International Society for Software Testing (ISST), directly targetted at ISO.

Karen’s petition is a more general manifesto for all professional testers to sign if they agree with its stance on certification and standards.

I strongly commend both the petition and the manifesto to all testers.

Eric Proegler also set up an AST special interest group to monitor and review the issue.

This action was not confined to the conference. In the last three weeks there has been a blizzard of activity and discussion on social media and throughout the testing community. Many people have blogged and spoken out. I gave a brief interview to Infoworld.com, and wrote an article for the uTest blog.

Huib Schoots has drawn together many of the relevant articles on his blog. It’s a valuable resource.

However, my own blog has been embarrassingly silent. I’ve not had time, till now, to write anything here. I’ve seen a big rise in traffic as people have hunted down articles I have previously written. August was the busiest month since I started the blog in 2010.

Significant and sustained opposition

Finally I have had a chance to set some thoughts down. I never expected my talk to get such a huge reaction. The response has not been because of what I said. It has happened because it came at the right time. I caught the mood. Many influential people, whom I respect, realised that it was time to speak out, and to do it in unison.

There has been significant and sustained opposition to ISO 29119 as it has developed over the years. However, it has been piecemeal. Individuals have argued passionately, authoritatively and persuasively against the standard, and they have been ignored.

The most interesting response since my talk, however, has been from the dog that didn’t bark, ISO. Neither ISO nor the 29119 working group has come out with a defence of the standard or the way that it was developed.

They have been accused of failing to provide a credible justification for the standard, of ignoring and excluding opponents, and of engaging in rent-seeking by promoting a factional interest as a generic standard.

And their response? Silence. In three weeks we have heard nothing. In their silence ISO have effectively conceded all the points that the Stop 29119 campaigners have been arguing.

There have been some sporadic and entirely unconvincing attempted defences of the standard, mixed up the with odd spot of risibly offensive ad hominem attacks. Collectively the weakness of these defences exposes ISO 29119 for the sham that it is.

Defence #1 – “It’s a standard”

This is possibly the weakest and most wrong-headed argument. ISO are trading on their brand, their image as a disinterested promoter of quality. It is disappointing how many people accept anything that ISO does at face value.

It is important that promoters of any standard justify it, demonstrate that it is relevant to those who will have to use it and that they enjoy a consensus amongst that community. In practice, ISO can spin the argument around.

Once ISO anoints a document with the magic word “standard” too many people suspend their own judgement. They look for opponents to justify their opposition, with reference to the detail they object to in the standard.

In the absence of such detailed arguments against the standard’s content people feel content with saying “it’s a standard, therefore it is a good thing”. That argument might impress those who know nothing about testing or ISO 29119. It lacks any credibility amongst those who know what they are talking about.

Defence #2 – “It’s better than nothing”

Whoops. Sorry. I have to backtrack rapidly. This argument is even worse than the last one. Something that is lousy is emphatically not better than nothing. Just doing something because it’s better than doing nothing? Oh dear. It’s like being lost in a strange city, without a map, and putting a blindfold over your eyes. Well, why not? You’re lost. You don’t know what you’re doing. Blindfolding yourself has to be better than nothing. Right? Oh dear.

This is the politicians’ fallacy, as explained in the classic comedy “Yes Minister”. Thanks to Scott Nickell for reminding me about it (see comments below).


If an organisation’s testing is so disorganised and immature that ISO 29119 might look like the answer then it is ripe ground for a far better approach. Incompetence is not a justification for wheeling in ISO 29119. It is a justification for doing a better job. Next argument please.

Defence #3 – “ISO 29119 doesn’t have to be damaging. It’s possible to work around it.”

The arguments in favour of ISO 29119 aren’t really getting a whole lot more convincing. Sure, you might be able to mitigate some of the worst effects by tailoring your compliance, or ignoring parts that seem particularly unhelpful. However, that is putting effort into ensuring you aren’t worse off than you are if you do nothing.

Also, if standards and rules are so detailed and impractical that they cannot be followed in full then it exposes people to arbitrary enforcement. Once users start tailoring the standard they will leave themselves open to the accusation that they did not comply in full. If things go well there will be no thanks for tailored compliance.

If there are problems, and there are always problems, then any post mortem will reveal that the standard wasn’t followed in full. The implication will be that the problems were caused by deviation, not that the deviation was the only reason anything worthwhile was achieved. Testers and developers will rightly fear that retribution will be arbitrary and quite unrelated to the level of care and professionalism they brought to their work.

Further, ISO 29119 is being marketed with an appeal to fear, as a way of protecting individuals’ backsides if things go wrong. If managers buy into the standard on that basis, are they really likely to take a chance with tailoring the standard?

ISO 29119 is also being pushed to buyers who don’t understand what good practice in testing means. Are they not likely to play safe by insisting in contracts and internal standards that ISO 29119 is complied with?

No, the possibility that we can “work around it” is not a credible argument in favour of ISO 29119.

Defence #4 – “The dissenters are just self-interested”

The standard is apparently a threat to our “craft” and to our businesses. Well, there’s more to it than that, but even if that were the only objection it would still be an entirely valid one. We have argued strenuously that the standard is anti-competitive. A riposte that we fear it will hit us financially is merely to concede the argument in a graceless way.

Anyway, if I were chasing the money I could have made a lot more, an awful lot more by taking lucrative test management contracts to do poor quality work managing the testing process on dysfunctional projects.

I can do great documentation. I am literate, organised and have the knack of getting tuned into corporate politics. My ability to churn out high quality, professional looking documents that would get the money in from clients was a great asset to my last employer.

Testing? Sorry, I forgot about that. I thought we were talking about documentation and money.

Defence #5 – “They’re whingers who wouldn’t get involved.”

This point ignores the argument that the ISO process for developing standards is carefully designed to exclude those who disagree. Again, the “whingers” line spins the argument around. It is not our responsibility to justify our failure to get involved with rent seekers and try to persuade their committees to desist.

I have seen the ISO 29119 working group’s schedule for meetings worldwide over the last few years. In the unlikely event that I would have been allowed to join, my expenses would have wiped out a huge slice of my income. It would certainly have cost me far more than we’ve spent on family holidays in that period. And for what? I’d have sacrificed all that time and money in order to be ignored. Those gallant souls who have tried to fight from the inside have been ground down and spat out by the system. That’s how the system works, how it is intended to work.

No, I didn’t get involved. I had better things to do with my time. In any case, it seems that my campaigning from the outside has not been ignored!

So what now?

We keep going. All this activity has just been the start. ISO is not going to withdraw the standard. However, the blogs, articles and petititions lay down a marker. It shows that the standard was not developed according to any plausible definition of consensus, and that it lacks credibility.

The opposition will strengthen the resolve of testers who wish to stop their organisations buying into the standard. It will provide ammunition to those who want to persuade lawyers and purchasing managers from requiring compliance to ISO 29119.

This one is going to run. We are not going away, and if ISO continue to ignore us then they will simply make themselves look foolish. Do they care? Or are they confident they have enough power and can just sail on regardless?

The problem is not the problem

There are two issues that have been bugging me for a while now and I mean to get them off my chest this week.

They’re not software testing topics but as with so many psychological, organisational or social problems they do have a very real relevance for testers. We don’t exist in a technical or procedural bubble that insulates us from the blizzard of conflicting and confusing influences that swirl around every organisation.

The first issue is that of problems. Very often the problem is not the problem. The problem we see is not necessarily the real, possibly very damaging, problem.

As testers this shouldn’t be news. We’re used to root cause analysis, delving behind the curtain to find out why things have gone wrong and not being deceived by what we initially see. We should also be familiar with the “five whys”, or some variant on that; the discipline of repeatedly challenging answers till we burrow down to the heart of the problem.

That’s not what I want to talk about here. Things go wrong, in life, at work, and especially in software development. That’s the way of it.

What really matters, certainly in the long run, is often not the problem itself but the way we deal with it.

We can deal with the problem in a brisk and efficient manner, resolving the immediate concerns, learning the right lessons and moving on. Or we can blunder into any one of several traps that could hurt us far more than the original problem did.

The first is the cover up. Watergate is the great historical example. The original crime might have been handled without damaging Richard Nixon, but it was the cover up that ended his presidency.

When we, as software developers, hit problems the worst thing we can do is attempt to cover up. Honesty with customers and users isn’t simply honourable and ethical, it also buys you trust and support.

Own up on the smaller problems and it may just buy you the credibility and goodwill to save your skin when you hit big, career threatening problems.

Denial is the sibling of the cover up. It might not be a vigorous, outspoken denial. It may just be a matter of kidding ourselves that there’s nothing really wrong, that we can carry on as before once we’ve sorted the problem. It was just bad luck, or a bit of carelessness. No big deal. Just move on folks. Nothing to learn here.

The next trap is the one you plunge into if you head off in the opposite direction; over-reaction. Losing your temper and savaging staff is obviously damaging and isn’t tolerated in healthy organisations any more, but a more insidious variant is still too prevalent. That is the tendency to blame individuals.

Someone must be to blame, somebody must have screwed up and be held to account. I once worked somewhere with an appalling blame culture. Problems were something to run away from, not something to fix. No-one wanted to jump in and proactively resolve problems in case they got implicated in the management investigation and fallout.

A more respectable form of over-reaction is the assumption that the problem justifies sweeping changes or drastic action rather than careful tweaking. Instead of a careful analysis of what actually went wrong and what the cause was, there is the presumption that it was confirmation of a trend, or deeper problem, that was previously suspected.

That leads me on to the fourth trap, confirmation bias. We convince ourselves the world should work a certain way. Any and every problem can be interpreted as supporting evidence.

The glaring example in the history of software development was the conviction that software development should be essentially the same as construction engineering. Early experts saw practitioners struggling to manage projects with a more or less structured and orderly linear process. They concluded that the answer to the problem was to try and make the process even more rigidly structured, more ordered and more linear. We got Structured Methods (shudder!).

After all, nobody builds a bridge incrementally, opening it bit by bit. No-one erects a building in several iterations. So software engineering shouldn’t be any different. That was more or less the line of thought. Life might be simpler if software development were a highly predictable and controllable process, but the truth is that development is not like that.

I referred to this problem on my blog earlier this year when I explained the predictability paradox.

The correct response to the problem would have been to go with the grain of the problem, embracing the uncertainty, building iteration and incremental development into the development process rather than pretending that we understand the business problem sufficiently well to proceed down a rigid, linear route.

The problem was assumed to be that software engineering was not sufficiently like construction engineering. Divergence between the two activities was seen not as evidence of a genuine difference, but as proof that the software developers were out of line. The truth was that software development is a quite separate discipline and the real problem was that developers were forced to contort their practices to fit a wildly inappropriate paradigm.

The real problem wasn’t the problem the profession saw. It was their response to the problem.

None of this is new to marketing people. Apparently BMW owners who have a problem with their car are more likely to buy another BMW than those who have no problems. Why? It’s because of the way that BMW deal with problems. Is it true? Well, I don’t have access to the statistics, but that’s BMW’s reputation. Problems have become an asset, rather than remaining a problem.

What does this mean for any of us? It’s not easy to draw a neat solution to this problem of problems. All I can pass on is something I have found useful.

Often we plunge into one of the traps because we choose to respond in a way that suits our instincts or prejudices, or which is just the easy, comfortable response.

Take a breath before committing yourself and ask yourself whether your response is what ought to happen. Or is it the response that you’d really like to be the right one? Is it the easy one? Are you maybe dodging the real issue, going for short term expediency rather than long term results?

Run through the traps I’ve discussed. Do any of these make you wonder about your response?

Is your reaction to the problem going to be seen as the real problem further down the line?

After I wrote this I checked the phrase “the problem is not the problem” to see if anyone had used it before. Initially I was a bit disappointed to see that Tom Peters has, but then I cheered up when I realised he’s a pretty respectable ally! Watch the video in the link or, if you’re in the office, check out the transcript (PDF, opens in a new window). It’s very much in line with my argument, but he puts a bit of extra spin on it.

The other thing I want to get off my chest this week is about making decisions. I’ll be back in a day or so.

Please add any comments. I do like to see what people think.

The picture in the header

This was taken in Glen Shirra in Inverness-shire, looking north-east over Loch Crunachdan and upper Strathspey; Ordnance Survey map reference 540918.

I took this photo on my cycle ride from Perth to Inverness in June 2009. That day was one of the best of my trip. I’d taken the train from Rannoch Station to Corrour Halt, then cycled through Strathossian and along Loch Laggan. I took the short and scenic, but very rough, route over the top from Aberarder Lodge and down into Strathspey.

I was on a high when I took this picture. After slogging up the steep track from Loch Laggan I was now bumping downhill into Strathspey in glorious warm sunshine through fantastic scenery. Scotland was looking at its very best that day and it was great to be alive.