Standards – a charming illusion of action

The other day I posted an article I’d written that appeared on the uTest blog a few weeks ago. It was a follow up to an article I wrote last year about ISO 29119. Pmhut (the Project Management Hut website) provided an interesting comment.

”…are you sure that the ISO standards will be really enforced on testing – notably if they don’t really work? After all, lawyers want to get paid and clients want their projects done (regardless of how big the clients are).”

Well, as I answered, whether or not ISO 29119 works is, in a sense, irrelevant. Whether or not it is adopted and enforced will not depend on its value or efficacy. ISO 29119 might go against the grain of good software development and testing, but it is very much aligned with a hugely pervasive trend in bureaucratic, corporate life.

I pointed the commenter to an article I wrote on “Teddy Bear Methods”. People cling to methods not because they work, but because they gain comfort from doing so. That is the only way they can deal with difficult, stressful jobs in messy and complex environments. I could also have pointed to this article “Why do we think we’re different?”, in which I talk about goal displacement, our tendency to focus on what we can manage while losing sight of what we’re supposed to be managing.

A lesson from Afghanistan

I was mulling over this when I started to read a fascinating looking book I was given at Christmas; “Heirs to Forgotten Kingdoms” by Gerard Russell, a deep specialist in the Middle East and a fluent Arabic and Farsi speaker.

The book is about minority religions in the Middle East. Russell is a former diplomat in the British Foreign Office. The foreword was by Rory Stewart, the British Conservative MP. Stewart was writing about his lack of surprise that Russell, a man deeply immersed in the culture of the region, had left the diplomatic service, then added;

”Foreign services and policy makers now want ‘management competency’ – slick and articulate plans, not nuance, deep knowledge, and complexity.”

That sentence resonated with me, and reminded me of a blistering passage from Stewart’s great book “The Places in Between”, his account of walking through the mountains of Afghanistan in early 2002 in the immediate aftermath of the expulsion of the Taliban and the NATO intervention.

Rory Stewart is a fascinating character, far removed from the modern identikit politician. The book is almost entirely a dispassionate account of his adventures and the people whom he met and who provided him with hospitality. Towards the end he lets rip, giving his brutally honest and well-informed perspective of the inadequacies of the western, bureaucratic, managerial approach to building a democratic state where none had previously existed.

It’s worth quoting at some length.

“I now had half a dozen friends working in embassies, thinktanks, international development agencies, the UN and the Afghan government, controlling projects worth millions of dollars. A year before they had been in Kosovo or East Timor and in a year’s time they would have been moved to Iraq or Washington or New York.

Their objective was (to quote the United Nations Assistance Mission for Afghanistan) ‘The creation of a centralised, broad-based, multi-ethnic government committed to democracy, human rights and the rule of law’. They worked twelve- or fourteen- hour days, drafting documents for heavily-funded initiatives on ‘democratisation’, ‘enhancing capacity’, ‘gender’, ‘sustainable development,’ ‘skills training’ or ‘protection issues’. They were mostly in their late twenties or early thirties, with at least two degrees – often in international law, economics or development. They came from middle class backgrounds in Western countries and in the evenings they dined with each other and swapped anecdotes about corruption in the Government and the incompetence of the United Nations. They rarely drove their 4WDs outside Kabul because they were forbidden to do so by their security advisers. There were people who were experienced and well informed about conditions in rural areas of Afghanistan. But such people were barely fifty individuals out of many thousands. Most of the policy makers knew next to nothing about the villages where 90% of the population of Afghanistan lived…

Their policy makers did not have the time, structures or resources for a serious study of an alien culture. They justified their lack of knowledge and experience by focusing on poverty and implying that dramatic cultural differences did not exist. They acted as though villagers were interested in all the priorities of international organisations, even when they were mutually contradictory…

Critics have accused this new breed of administrators of neo-colonialism. But in fact their approach is not that of a nineteenth-century colonial officer. Colonial administrations may have been racist and exploitative but they did at least work seriously at the business of understanding the people they were governing. They recruited people prepared to spend their entire careers in dangerous provinces of a single alien nation. They invested in teaching administrators and military officers the local language…

Post-conflict experts have got the prestige without the effort or stigma of imperialism. Their implicit denial of the difference between cultures is the new mass brand of international intervention. Their policy fails but no one notices. There are no credible monitoring bodies and there is no one to take formal responsibility. Individual officers are never in any one place and rarely in any one organisation long enough to be adequately assessed. The colonial enterprise could be judged by the security or revenue it delivered, but neo-colonialists have no such performance criteria. In fact their very uselessness benefits them. By avoiding any serious action or judgement they, unlike their colonial predecessors, are able to escape accusations of racism, exploitation and oppression.

Perhaps it is because no one requires more than a charming illusion of action in the developing world. If the policy makers know little about the Afghans, the public knows even less, and few care about policy failure when the effects are felt only in Afghanistan.”

Stewart’s experience and insight, backed up by the recent history of Afghanistan, allow him to present an irrefutable case. Yet, in the eyes of pretty much everyone who matters he is wrong. Governments and the military are prepared to ignore the evidence and place their trust in irrelevant and failed techniques rather than confront the awful truth; they don’t know what they’re doing and they can’t know the answers.

Vast sums of money, and millions of lives are at stake. Yet very smart and experienced people will cling on to things that don’t work, and will repeat their mistakes in the future. Stewart, meanwhile, is very unlikely to be allowed anywhere near the levers of power in the United Kingdom. Being right isn’t necessarily a great career move.

Deep knowledge, nuance and complexity

I’m conscious that I’m mixing up quite different subjects here. Software development and testing are very different activities from state building. However, both are complex and difficult. Governments fail repeatedly at something as important and high-profile as constructing new, democratic states, and do so without feeling the need to reconsider their approach. If that can happen in the glare of publicity is it likely that corporations will refrain from adopting and enforcing standards just because they don’t work? Whether or not they work barely matters. Such approaches fit the mindset and culture of many organisations, especially large bureaucracies, and once adopted it is very difficult to persuade them to abandon them.

Any approach to testing that is based on standardisation is doomed to fail unless you define success in a way that is consistent with the flawed assumptions of the standardisation. What’s the answer? Not adopting standards that don’t work is an obvious start, but that doesn’t take you very far. You’ve got to acknowledge those things that Stewart referred to in his foreword to Gerard Russell’s book; answers aren’t easy, they require deep knowledge, an understanding of nuance and an acceptance of complexity.

A video worth watching

Finally, I’d strongly recommend this video of Rory Stewart being interviewed by Harry Kreisler of the University of California about his experiences and the problems I’ve been discussing. I’ve marked the parts I found most interesting.

34 minutes; Stewart is asked about applying abstract ideas in practice.

40:20; Stewart talks about a modernist approach of applying measurement, metrics and standardisation in contexts where they are irrelevant.

47:05; Harry Kreisler and then Stewart talk about participants failing to spot the obvious, that their efforts are futile.

49:33; Stewart says his Harvard students regarded him as a colourful contrarian. They believed that all Afghanistan needed was a new plan and new resources.

ISO 29119: Why is the Debate One-Sided?

This article originally appeared on the uTest blog on February 23rd 2015.

In August last year the Stop 29119 campaign and petition kicked off at the CAST conference in New York.


In September I wrote on the uTest blog about why the new ISO/IEEE 29119 software testing standards are a danger to good testing and the whole testing profession.

I was amazed at the commotion that Stop 29119 caused. It was the biggest talking point in testing in 2014. Six months on it’s time to look back. What has actually happened?

The remarkable answer is – very little. The Stop 29119 campaigners haven’t given up. There has been a steady stream of blogs and articles. However, there has been no real debate; the discussion has been almost entirely one sided.

There has been only one response from ISO. In September Dr Stuart Reid, the convenor of the working goup that produced the standard, issued a statement attempting to rebut the arguments of Stop 29119. That was it. ISO then retreated into its bunker and ignored invitations to debate.

Dr Reid’s response was interesting, both in its content and the way it engaged with the arguments of Stop 29119. The Stop 29119 petition was initiated by the board of the International Society for Software Testing. ISST’s website had a link to the petition, and a long list of blogs and articles from highly credible testing experts criticising ISO 29119. It is a basic rule of debate that one always tackles an opponent’s strongest points. However, Dr Reid ignored these authoritative arguments and responded to a series of points that he quoted from the comments on the petition site.

To be more accurate, Dr Reid paraphrased a selection of the comments and criticisms from elsewhere, framing them in a way that made it easier to refute them. Some of these points were no more than strawmen.

So Cem Kaner argued that IEEE adopts a “software engineering standards process that I see as a closed vehicle that serves the interests of a relatively small portion of the software engineering community… The imposition of a standard that imposes practices and views on a community that would not otherwise agree to them, is a political power play”..

Dr Reid presented such arguments as “no-one outside the Working Group is allowed to participate” and “the standards ‘movement’ is politicized and driven by big business to the exclusion of others”.

These arguments were then dismissed by stating that anyone can join the Working Group, which consists of people from all parts of the industry. Dr Reid also emphasized that “consensus” applies only to those within the ISO process, failing to address the criticism that this excludes those who believe, with compelling evidence, that ISO-style standardization is inappropriate for testing.

These criticisms had been made forcefully for many years, in articles and at conferences, yet Dr Reid blithely presented the strawman that “no-one knew about the standards and the Working Group worked in isolation”. He then effortlessly demolished the argument that no-one was making.

What of the content? There were concerns about how ISO 29119 deals with Agile and Exploratory Testing. Eg, Rikard Edgren offered a critique arguing that the standards tried but failed to deal with Agile. Similarly, Huib Schoots argued that a close reading of the standards revealed that the writers didn’t understand exploratory testing at all.

These are serious arguments that defenders of the standard must deal with if they are to appear credible. What was the ISO response?

Dr Reid reduced such concerns to bland and inaccurate statements that “the standards represent an old-fashioned view and do not address testing on agile projects” and ”the Testing Standards do not allow exploratory testing to be used”. Again these were strawmen that he could dismiss easily.

I could go on to highlight in detail other flaws in the ISO response; the failure to address the criticism that the standards weren’t based on research or experience that demonstrates the validity of that approach; the failure to answer the concern that the standards will lead to compulsion by the back door; the failure to address the charge from the founders of Context Driven Testing that the standards are the antithesis of CDT; the evasion of the documented links between certification and standards.

In the case of research Dr Reid told us of the distinctly underwhelming claims from a Finnish PhD thesis (PDF, opens in a new tab) that the standards represent “a feasible process model for a practical organisation with some limitations”. These limitations are pretty serious; “too detailed” and “the standard model is top heavy”. It’s interesting to note that the PhD study was produced before ISO 29119 part 3 was issued; the study does not mention part 3 in the references. The study can therefore offer no support for the heavyweight documentation approach that ISO 29119 embodies.

So instead of standards based on credible research we see a search for any research offering even lukewarm support for standards that have already been developed. That is not the way to advance knowledge and practice.

These are all huge concerns, and the testing community has received no satisfactory answers. As I said, we should always confront our opponents’ strongest arguments in a debate. In this case I’ve run through the only arguments that ISO have presented. Is it any wonder that the Stop 29119 campaigners don’t believe we have been given any credible answers at all?

What will ISO do? Does it wish to avoid public discussion in the hope that the ISO brand and the magic word “standards” will help them embed the standards in the profession? That might have worked in the past. Now, in the era of social media and blogging there is no hiding place. Anyone searching for information about ISO 29119 will have no difficulty finding persuasive arguments against it. They will not find equally strong arguments in favour of the standards. That seems to be ISO’s choice.