You’ve taken many flights in the past, often at the same time. You know it will take probably about 75 minutes to cover the distance at the busiest time of the day. You know how long it will take to get parked and into the terminal. You allow a bit of padding for safety, decide when you’re going to leave home, and you catch your flight.

What you never do is say, “well, if there are no problems and I’m not held up I can do it in 50 minutes, so I’ll set off at 7:55”.

I’m also very confident that you never say, “I can do the 50 km to the end of the motorway at 120 kph, so that’s 25 minutes, then the last 10km at 60 kph, so that’s another 10 minutes, and if I park in the middle of the car park I will be 150 metres from the check in desk, a distance I can cover in 2 minutes, so if I allow 10% contingency I can leave home at 8:04 and I will be fine.”

That would be insane. You use your experience, you know where things can go wrong and you catch your flight. You always catch your flight! That’s the priority, not impressing people by shaving time off your estimated journey to the airport, or demonstrating your mastery of the trip by building up a pointlessly detailed estimate.

Ok, in the airport case you knew the route and the problems, so you’d obviously go with your direct experience. But would you try doing a detailed estimate for the separate components of the return journey from your hotel to the airport? Would you plan optimistically?

Of course not. You’d rely on your past experience of getting to airports in general and you’d plan the trip cautiously because you’d want to be extremely confident that you’d catch your flight.

So why, when we size IT work, do we habitually go for estimates that are based on best case scenarios, rather than bitter experience, often constructing detailed estimates, full of meaningless spurious attempts at accuracy?

We pile up optimistic guesswork on top of deluded nonsense, then pretend that it has some validity just because it consists of an impressive amount of detail. If it ever works it’s pure luck.

We kid ourselves that we have learnt from past mistakes, that we won’t repeat them, and that any unavoidable problems we hit were bad luck, someone else’s fault or were simply unique to that particular project.

For some reason we find it very difficult to apply the results of past experience to estimating future work. There are various related reasons for this.

We value the specific over the general

Firstly, we value detailed knowledge of the specific task we’re looking at over experience that we, and others, have had with similar work. Logically that’s not an explanation. All I have done is rephrase the problem, but I’ve done so in a way that helps explain why we kid ourselves.

Applying our detailed knowledge maximises the relevant information that we are feeding into the equation. Excluding the results of past experience is rational, but only if one makes completely unrealistic assumptions about risk, probability and our ability to dodge problems that always hit projects.

Kahneman and Tversky (PDF, opens in new window) illustrated this point by discussing how we would predict the life expectancy of an individual. We can call on singular information and distributional information.

Singular information applies specifically to that person. Distributional information ignores the individual and is concerned with the distribution of outcomes; how long human beings live for.

Most people would accept that the most accurate prediction would require both sorts of information. However, we have a tendency to shy away from even the most obviously relevant distributional data.

Kahneman and Tversky found that the more familiar we are with a subject the more likely we are to focus on the singular and to ignore distributional data. The result is that the distribution of predicted outcomes is significantly different from the distribution of real outcomes. Worse, the variation is firmly in one direction and amounts to a bias.

Obviously that bias is towards optimism. We might know that things tend to go wrong, but we don’t know exactly what form the problems will take. We only know for sure how long things ought to take, and we fail lamentably to build in adequate contingency or to acknowledge the probability that life will not be perfect.

We are incorrigibly optimistic

This is the second reason for our inability to use past experience wisely. Regardless of the preference for estimating using singular rather than general information, we are inherently optimistic, and we wildly overestimate the probability of our estimates being accurate.

Or rather, we are hopeless at estimating accurately how long we will take to do something. We are far better at estimating how long someone else will take. We are more likely to look dispassionately at their past performance and not kid ourselves about how long the new task could take in ideal circumstances.

We revere confidence

This ability to apply careful judgement to others people’s work should be valuable in estimating software development projects, and it should help compensate for our preference for the singular over the general. However, any benefit is largely swamped by the effects of the third reason for our failure to learn from experience.

I believe this third factor is the most pernicious of all. It is simply that we value confidence over realism.

I remember a senior IT manager telling me that it was far better to plunge confidently into a project with an impossible deadline, and then emerge sweaty, exhausted and late than it was to tell users at the start how long it would really take. The users just wouldn’t accept realism.

The first time I had to estimate a development project I was concerned that we were using an unproven mixture of tools and technology, and we were applying them to an application for which they’d never been used at that site. I estimated cautiously, and was briskly told that my estimate was far too high.

“You can only estimate for what you know will happen, plus some contingency. You can’t expect to get approval for work to cover problems you don’t know you’re going to have”.

The estimate was halved, and I brought the project in 100% over budget. By a delicious coincidence my initial guess was accurate. I received plenty of praise for my achievement. There was no criticism for being wildly over budget, and that’s certainly not because anyone might have remembered the original estimate!

In such an environment managers who do estimate accurately are likely to get a reputation for “padding their estimates”, not for being right!

The lesson I learned was that an estimate was the highest figure that was politically acceptable. Not commercially acceptable, but the highest figure you could get away with, almost invariably less than the real one.

I’ve got extensive personal experience of this phenomenon, but the evidence isn’t just anecdotal.

In a fascinating study by Jørgensen et al 37 software developers were asked to assess the competence of two developers based on their ability to estimate and deliver work.

Both produced identical estimates and results. However, one was much more confident and predicted narrower margins of error. The other was more realistic about the level of uncertainty and predicted wider margins.

The subjects in the study rated the confident developer as being more competent than the realistic one. In effect, confidence was regarded as a sign of competence. Even when the actual result was outside the confident developer’s margins of error and within the realistic one’s margins it was still the confident developer who was rated higher.

A comment from one of the subjects was revealing.

“I feel that if I estimate very wide (margins of error), this will be interpreted as a total lack of competence and has no informative value to the project manager. I’d rather have fewer actual values inside the minimum-maximum interval, than providing meaningless, wide (margins)”.

This study looked specifically at confidence, rather than optimism, but the constant danger in software development is, of course, over-confident optimism rather than pessimism.

Catch 22? Or catching that flight?

These are consistent biases. The problem is not simply that we get estimates wrong, but we approach them in a way that ensures we consistently err on the side of optimism.

I wonder if there’s a Catch 22 that keeps us trapped in a cycle of poor, over-optimistic estimating.

The less we know about a problem area the less we will know about the complexities and what could go wrong. So we are likely to to be over-confident about our estimates.

But once we have learned more about the problem area, the more tempted we will be to focus on the specifics of the task and ignore the distribution of outcomes, i.e. to ignore the lessons of experience. So we are still likely to be over-confident about our estimates.

There’s a huge exception, however. Remember the analogy I was using at the start of this piece? About getting to the airport to go on holiday.

We always catch our flight. It’s not simply that estimating accurately is easier. It really matters to us. I mean really matters.

Does it really matter as much in software development? I don’t think it does. Just consider the research suggesting we prefer to seem confident rather than right. Perhaps we sometimes need to do a check.

Do we really want to catch that flight? Or do we just want to seem confident?

I’ll expand on possible remedies in my next post. In the meantime if you want to delve further into this area here are a couple of interesting links.

Tversky and Kahneman look at how people have difficulty making sound judgements about probability based on their direct experience.

Buehler et al’s study of our tendency to underestimate task completion times.

The speaker was Neil Allison of the University of Edinburgh’s Website Development Project.

There’s no need for me to discuss Neil’s talk at length because he’s posted it here.

Neil used a phrase that leapt out at me. Prototyping ”helps us take informed decisions at the right time”.

That was exactly what I was thinking about when I wrote in my last blog about taking decisions at the right time, the last responsible moment in Lean terms.

Neil’s phrase summed up the appeal of prototyping perfectly for me. We can take informed decisions at the right time; not take poor, arbitrary or ill-informed decisions when we’ve simply run out of options and have to take whatever default option we’re left with.

In a later email Neil made the point that I keep on trying to make; that usability testing should shape the design, rather than evaluate it.

“When I do usability testing, it’s to check the requirements and the ease of use of a preferred solution. Usually before development begins in earnest as it’s too time consuming and/or costly to backpedal later. (Almost) pointless doing usability testing if you don’t have the resources to take action on the findings”.

Prototyping may have clear benefits but Neil is still trying to raise awareness within the university, spreading the word and trying to widen the use of the technique.

Testers should never see their role as being defined by a rigid process. They should always be looking for better ways to do their job, and be prepared to lobby and educate others.

Please have a look at Neil’s presentation, which contains advice on useful books and tools.

People often have a strange attitude towards decisions. Their instinct is that important decisions require a lot of time and effort. Also, they have an irrational gut feeling that decisions become important and justify lots of our time simply because they are complex. We can get rattled by trivial decisions, where there are surprising complexities that make it difficult to give a definitive right answer.

I’ve been given two pieces of invaluable advice about decision making. The first was from my father when I was still at school.

”Very often the worst decision is no decision. Often the difference between two options is less than the consequences of hanging around, wasting time and not making any decisions at all. Just make your decision and get on with it.”

The second piece of advice was from my manager just after I left university. He went on to become a famous and wealthy businessman. He also had a somewhat cavalier approach to business ethics, which ultimately attracted some very unfavourable press, so I won’t name him. I don’t want to give the impression I endorse his views in general. We disagreed about the way to conduct business, but I’ve found his advice about decision making valuable.

He’d asked me to research a company that we might invest in. I took a couple of days to produce a detailed and thorough analysis explaining all the pros and cons of investing.

He read it, then told me that it was a good, thorough piece, but I’d wasted time on delving into too much detail.

“There are probably no more than about three factors you have to take account of before you make a decision. The rest are either unimportant or irrelevant. The trick is being able to recognise the key points and not waste time on stuff that won’t actually affect the decision”.

Often we get intimidated by the thought of taking responsibility and taking decisions. So we retreat into time-wasting detail. It shows we’re taking our responsibilities seriously and helpfully defers the point when we have to make our decision.

So, yes, my father and my manager were right. An unwillingness to take appropriately quick decisions can have a paralysing effect, and we do have a tendency to plunge into suffocating detail rather than focussing on what really matters.

However, the advice from my father and my manager weren’t rigid rules. They were just guidelines, and shouldn’t be applied inflexibly in every situation.

During the Second World War my father served in the airborne forces, a branch of the army in which making quick decisions and then getting on with it wasn’t a management style, it was a matter of life and death. Later he was a successful salesman and senior sales manager. That career encouraged and rewarded his decisiveness.

My manager was leading a fast moving investment team, buying and selling shares every day. Again, cool and quick thinking were crucial.

The context is crucial when it comes to selecting a style of decision making. In software development we’ve been hopeless at taking decisions and that’s largely because we haven’t recognised the context, which is different from the environment my father and manager thrived in.

Some organisations have a culture that assumes sound decisions require time and detail; if they throw a load of time and analytical effort at a problem they can confidently set their decisions in stone. That’s how the Waterfall, and traditional development methods operated.

Other places favour bold, early decisions, imposing arbitrary implementation dates on projects before anyone knows how much work is involved, or settling on outline design solutions before the requirements are understood (or instead of the requirements, see my blog on Tom Gilb’s thoughts on requirements).

Mostly we’ve mixed up our decision making styles in a way that defies logic and business sense and has given us the worst of both worlds.

Senior executives have applied the advice of my father and manager recklessly, going for ill-informed decisions while project teams have got lost in detail, ignoring big and simple signs that could have let them take important decisions earlier.

The paradox for such teams has been that they might have been taking months to reach irreversible decisions, but the truth was that the decisions were often taken too early in the logical sequence of the project, e.g. the confusion of design and requirements (Gilb again) and the pretence that a viable user interface can be confirmed before the users have tried it.

So, my experience in IT has taught me a third maxim.

The right initial decision is usually to decide what sort of decision we’re looking at.

Should we just do it? Take a quick decision and move on. Either it doesn’t matter, or the right answer is clear now.

Should we park it till later and not waste time agonising over it now, when we don’t have the right supporting information?

Or should we make a provisional decision, act on it and revisit it when we know more?

The trick is to know what sort of problem we’re looking at.

That takes us on to Lean, which has a very pragmatic and helpful attitude towards decision making. It incorporates the principle of taking decisions at the “last responsible moment”.

That doesn’t mean the last possible moment. It’s not a recipe for indecision and procrastination. The last responsible moment could be very early, depending on the circumstances.

This is one of the aspects of Lean that appeals to me. It not only makes sense in software development terms, it also recognises the general point that context is crucial rather than pretending that a single, inflexible approach to decision making applies in every circumstance.

As Mary Poppendieck put it in “The Predictability Paradox”, sadly not available on-line.

“You need to develop a sense of when decisions must be made and then make them when their time has come. It is equally important to develop a sense of what is critically important in the domain and make sure these areas are not overlooked in the decision-making process.”

Mary and Tom Poppendieck expanded on this in their book “Lean software development: an agile toolkit”.

So it’s useful to keep running mental checks, to make sure you’re treating the decision appropriately, and not just taking a decision because you’re feeling gung ho, or deferring a decision because you want more and more information that won’t really affect the final decision.

It’s also important to have confidence in your own judgement and experience. Point out to critics that a quick decision isn’t necessarily a hasty one, and a deferred decision doesn’t mean indecisiveness. It all depends on the context.

Finally, and please, if you know me, don’t inflict on me any more meetings that last for an hour after everyone has reached agreement, but in which people felt they had to drag in more and more ammunition to support the big decision. One day I may just go berserk!

They’re not software testing topics but as with so many psychological, organisational or social problems they do have a very real relevance for testers. We don’t exist in a technical or procedural bubble that insulates us from the blizzard of conflicting and confusing influences that swirl around every organisation.

The first issue is that of problems. Very often the problem is not the problem. The problem we see is not necessarily the real, possibly very damaging, problem.

As testers this shouldn’t be news. We’re used to root cause analysis, delving behind the curtain to find out why things have gone wrong and not being deceived by what we initially see. We should also be familiar with the “five whys”, or some variant on that; the discipline of repeatedly challenging answers till we burrow down to the heart of the problem.

That’s not what I want to talk about here. Things go wrong, in life, at work, and especially in software development. That’s the way of it.

What really matters, certainly in the long run, is often not the problem itself but the way we deal with it.

We can deal with the problem in a brisk and efficient manner, resolving the immediate concerns, learning the right lessons and moving on. Or we can blunder into any one of several traps that could hurt us far more than the original problem did.

The first is the cover up. Watergate is the great historical example. The original crime might have been handled without damaging Richard Nixon, but it was the cover up that ended his presidency.

When we, as software developers, hit problems the worst thing we can do is attempt to cover up. Honesty with customers and users isn’t simply honourable and ethical, it also buys you trust and support.

Own up on the smaller problems and it may just buy you the credibility and goodwill to save your skin when you hit big, career threatening problems.

Denial is the sibling of the cover up. It might not be a vigorous, outspoken denial. It may just be a matter of kidding ourselves that there’s nothing really wrong, that we can carry on as before once we’ve sorted the problem. It was just bad luck, or a bit of carelessness. No big deal. Just move on folks. Nothing to learn here.

The next trap is the one you plunge into if you head off in the opposite direction; over-reaction. Losing your temper and savaging staff is obviously damaging and isn’t tolerated in healthy organisations any more, but a more insidious variant is still too prevalent. That is the tendency to blame individuals.

Someone must be to blame, somebody must have screwed up and be held to account. I once worked somewhere with an appalling blame culture. Problems were something to run away from, not something to fix. No-one wanted to jump in and proactively resolve problems in case they got implicated in the management investigation and fallout.

A more respectable form of over-reaction is the assumption that the problem justifies sweeping changes or drastic action rather than careful tweaking. Instead of a careful analysis of what actually went wrong and what the cause was, there is the presumption that it was confirmation of a trend, or deeper problem, that was previously suspected.

That leads me on to the fourth trap, confirmation bias. We convince ourselves the world should work a certain way. Any and every problem can be interpreted as supporting evidence.

The glaring example in the history of software development was the conviction that software development should be essentially the same as construction engineering. Early experts saw practitioners struggling to manage projects with a more or less structured and orderly linear process. They concluded that the answer to the problem was to try and make the process even more rigidly structured, more ordered and more linear. We got Structured Methods (shudder!).

After all, nobody builds a bridge incrementally, opening it bit by bit. No-one erects a building in several iterations. So software engineering shouldn’t be any different. That was more or less the line of thought. Life might be simpler if software development were a highly predictable and controllable process, but the truth is that development is not like that.

I referred to this problem on my blog earlier this year when I explained the predictability paradox.

The correct response to the problem would have been to go with the grain of the problem, embracing the uncertainty, building iteration and incremental development into the development process rather than pretending that we understand the business problem sufficiently well to proceed down a rigid, linear route.

The problem was assumed to be that software engineering was not sufficiently like construction engineering. Divergence between the two activities was seen not as evidence of a genuine difference, but as proof that the software developers were out of line. The truth was that software development is a quite separate discipline and the real problem was that developers were forced to contort their practices to fit a wildly inappropriate paradigm.

The real problem wasn’t the problem the profession saw. It was their response to the problem.

None of this is new to marketing people. Apparently BMW owners who have a problem with their car are more likely to buy another BMW than those who have no problems. Why? It’s because of the way that BMW deal with problems. Is it true? Well, I don’t have access to the statistics, but that’s BMW’s reputation. Problems have become an asset, rather than remaining a problem.

What does this mean for any of us? It’s not easy to draw a neat solution to this problem of problems. All I can pass on is something I have found useful.

Often we plunge into one of the traps because we choose to respond in a way that suits our instincts or prejudices, or which is just the easy, comfortable response.

Take a breath before committing yourself and ask yourself whether your response is what ought to happen. Or is it the response that you’d really like to be the right one? Is it the easy one? Are you maybe dodging the real issue, going for short term expediency rather than long term results?

Run through the traps I’ve discussed. Do any of these make you wonder about your response?

Is your reaction to the problem going to be seen as the real problem further down the line?

After I wrote this I checked the phrase “the problem is not the problem” to see if anyone had used it before. Initially I was a bit disappointed to see that Tom Peters has, but then I cheered up when I realised he’s a pretty respectable ally! Watch the video in the link or, if you’re in the office, check out the transcript (PDF, opens in a new window). It’s very much in line with my argument, but he puts a bit of extra spin on it.

The other thing I want to get off my chest this week is about making decisions. I’ll be back in a day or so.

Please add any comments. I do like to see what people think.

Yet some companies just can’t get a healthy sense of perspective with documentation. There’s no sense of “just enough”. They always seem to need more than is required, as if documentation is an end in itself.

I’ve been following Michael Bolton’s short blog series on test cases, part 1 and part 2, which I strongly recommend. As well as looking at the specific question of test cases, the wider issue of documentation is discussed, especially in the comments following the second piece.

What really gets me about the demand for skiploads of documentation is that it comes at a huge price. The time I’ve had to spend pointlessly perfecting documents that weren’t needed could have been better spent understanding the application, the business problem and the technology. I’ve sometimes had the dispiriting insight that I knew my way round the documentation far better than I knew the application.

Fiona Charles and Michael refer to a belief in documentation as being a phase that testers go through. Read their comments in part 2 on Michael’s blog. I certainly recognise that pattern.

For me the epiphany, the moment when the last illusions vanished, was on a project where the demand for documentation was insatiable, the deadlines fixed, and the testers too few.

As usual the documentation was produced from the top down; from the contract to the strategy, to the stage test plans, to the detailed test plans, to the test cases.

We were running out of time, but there was no let up in the demand for documentation. Massive test plans had to be refined and reworked even as we were moving into text execution, with only a few sketchy test case scripts in place.

The inevitable consequence was that the test execution was based almost entirely on the judgement, experience and knowledge of the business that the testers had, rather than all the paper plans. The perverse result of confusing documentation with structure was that the test execution was largely improvised and unstructured.

The huge effort that went into the higher level documentation was wasted because it never fed through into the detailed scripts. The test execution was pretty much what would have happened in the absence of all that documentation. Well, maybe not quite like that. If the testers had not been forced to churn out unwanted documentation they would have been actively involved earlier in the project, detecting and forestalling defects.

We could have done so much better for the project if we’d been allowed to select an approach that fitted the real problem, as opposed to a barely relevant process that was driven by an addiction to documentation, rather than by the risk. We could then have planned and prepared for the test execution in a way that would have helped us test better.

The actual testing was well informed and very valuable, but that was despite the documentation, not because of it. So much time was simply wasted in the preparation.

Even during the execution a lot of time was wasted attempting to retro-fit the execution to the test plans so that we could manipulate Quality Center to produce the reports for stakeholders who had such a poor grasp on our efforts that they couldn’t distinguish between the plans and reports and the reality that they were supposed to represent.

My epiphany was the awful realisation that the project was so dysfunctional that the documentation had become the reality.

One of the comments in Michael’s blog referred to the perceived need for test case documentation to satisfy the auditors. This is a red herring. Good auditors don’t need to see documented test cases. They need to see evidence of testing that was appropriate in the circumstances.

Of course there are plenty of bad auditors around, but I think that’s usually a result of inexperience or an unhealthy corporate culture. In any case the best response is to speak to them, to understand them and to change their attitudes if possible.

Whatever you do don’t try to second guess them and produce documentation in the expectation that they require it. When I was an auditor it would drive me up the wall when I found people wasting their time on documentation simply because they wrongly believed I would need to see it.

If you want some more ammunition to try and get auditors onside you could tell them to look at the Sarbanes Oxley guidance material from the Information Systems Audit & Control Association. It’s available to members only on their website. Let me know if you want a copy.

The Institute of Internal Auditors has also provided guidance on Sarbox (PDF – opens in new window).

Sarbox may apply only to the USA, but its requirements are fairly stringent, and you’d have a very strong argument if you could show that not even it requires detailed test scripting. The trouble is that Sarbox has the reputation for demanding extensive documentation, and some auditors go by the reputation rather than the legislation.

Or look at this article by Kanwal Mookhey in the Internal Auditor magazine.

And this article (on my business site) I wrote, in which Kanwal gave me some more explicit quotes on this subject.

Documentation must never be an end in itself. The analogy I drew with eating and drinking at the start wasn’t perfect. You might eat and drink for pleasure, but no sane person produces documentation for the joy of stacks of shelfware. Do yourself, and your projects a favour by challenging the unthinking demand for more and more detailed documentation.

I remember years ago, watching Friends in rapt admiration. Ross the paleontologist took Phoebe’s reluctance to believe in evolution as a personal affront.

Phoebe, the scatty new-ager, dismantled Ross’s certainty in a masterful little cross-examination.

YouTube doesn’t allow embedding of this clip, so here’s the link. The key bit is three and a half minutes in, but the lead up is fun anyway.

Frankly, Ross caved in far too easily, but that was largely the point as far as I was concerned. It wasn’t that he was wrong. The problem was that he was so convinced he was right that he was completely thrown by an astute rhetorical ploy.

He wasn’t used to being challenged, and couldn’t handle it. He was tested and failed.

Phoebe may not have cut it as a conventional corporate software tester, but she did have some qualities that are priceless for testers.

She had a distinct perspective. She looked at the same things that others saw, but saw them in a different way.

She wasn’t afraid to be out of step. She wasn’t worried about ridicule and didn’t care if people thought she was wrong. She was as far from being a “yes person” as you could find.

Above all she challenged conventional wisdom and forced people to question their assumptions. She enjoyed doing so, and managed to do it without truly upsetting people.

If the example had involved Phoebe being unambiguously right, whilst Ross was totally wrong, what would there be to learn from it?

I think most of us would agree that if the project is doing something wrong we should question it. The trouble is that challenging other people, especially senior people, or challenging requirements, can seem ill-mannered or dangerous. So it’s all to easy to persuade ourselves that they’re probably right. We just drift along and don’t ask questions.

Sceptical thinking and challenging questions are worthwhile regardless of whether we know we’re right or wrong. We often don’t know – not for sure. It’s important we don’t act as if we did know, as if certainty were available.

We have to ask the difficult questions to try and help everyone gain a better understanding of what’s going on. Even if our stance as devil’s advocate is wrong, or unsound, the process of questioning should have helped clarify everyone’s thinking.

Ross’s conviction that he was right hadn’t been tested. As soon as Phoebe swung into tester mode Ross crashed with a severity 1 failure. Maybe he needed to sharpen up his thinking and his debating skills? Ross the Debater 0.2 would surely have done far better after failing that test.

When I originally watched this episode of Friends I had recently switched to testing after spending a few years in computer audit. I could empathise with Phoebe’s willingness to challenge, and stir things up. I loved her sense of mischief as she did so A sense of mischief? I’ve never seen that in a job ad for a tester, but it’s not a bad quality.

Come to think of it, I remember a programme manager at IBM calling me an iconoclast. As a Christian of the reformed tradition I was happy to take that as a compliment!

Please! Don’t get over-analytical about the story of Phoebe the tester. It’s not an argument about evolution, or religion or science. As Phoebe said, “that was fun – so who’s hungry?”.

It sought to explain why so much of life just sucks. Given the extremely comfortable circumstances in which I was reading his article I wasn’t entirely convinced that life does suck all that badly. However, I certainly was receptive to the deeper meaning; that we are loften motivated by laziness and the desire for an easy, unchallenging life.

People enter into a sly conspiracy in which they happily trade low quality results, whilst indulging in high quality, but bogus rhetoric, for the record.

Burkeman’s article was based on the work of the Italian philosopher Gloria Origgi. See these articles from her blog; “Kakonomics, or the strange preference for Low-quality outcomes” and “The Kakonomics of Facebook”.

I had no difficulty accepting the gulf between grubby reality and the bogus rhetoric of high flying ideals. That has always been a feature of my working life.

Were people really happy with that gulf? It is of course counter-intuitive. I was sceptical. However, the more I thought about it the more willing I was to accept Origgi’s insights.

Transformation Programmes that transform nothing

I have often been puzzled at how some organisations have been seemingly trapped in damaging, dysfunctional behaviour, with no apparent willingness to break out and transform themselves. They have certainly been too willing to launch Transformation Programmes (note the deliberately disrespectful and cynical capital letters).

These programmes have rarely been designed to truly transform anything. They have simply rearranged anything and everything capable of being rearranged. The real work has carried on in spite of the transformation.

Once I really did see one of those programmes produce genuine and exciting benefits. That those benefits were irrelevant and accidental by-products of the Programme was revealed shortly before it was wound down.

A review of what had been achieved swept the benefits away in an act of casual destruction that effectively reinstated the pre-transformation status quo, but with a superficially different structure and youthful new faces added to the lower layers of the management team.

Was kakonomics at work there? Did the consultants deliver some low grade, but expensive, rubbish, which the management were happy to take on board because it didn’t challenge them to change either their behaviour or the organisation’s culture?

The last minute review ensured that management could carry on as before, but with the enhanced prestige of having “done something” about the abysmal quality and late delivery of IT applications.

Kakonomics and traditional development practices?

I have worked on many developments when you could sense the relief at progress meetings when someone confessed that they’d had problems and would deliver late. By doing so they let the others off the hook and no-one would be required to deliver on time.

That is consistent with kakonomics, but it can also be attributed to a natural human desire not to be the first to admit failure, or an equally natural desire for easier targets. By their actions do people really solicit and encourage poor quality and performance?

The possibility that Origgi might be right seemed stronger when I reflected on occasions when people had stood up and not only insisted on high quality outcomes, but delivered them. Were they lauded and rewarded as heroes?

Well, no. Not consistently.

I remember a friend once saying, shortly before he handed in his notice, that it made no sense to be a high performer at the company where we worked.

The difference in reward was negligible and insufficient to justify the extra effort. Worse, slovenly and inefficient work had its own reward in the form of generous, overtime without scrutiny of what was achieved. Sloppy cowboys could earn more than diligent craftsmen.

The users were untroubled. Everything was late and poor quality. However, the IT people made perfect scapegoats. Requirements could therefore be late, badly defined, and always open to change. IT would take the blame without complaint, largely because they knew it was expected of them, and in the full knowledge that the gap between aspiration and reality would be tolerated and leave them with a substantial comfort zone.

Both there and elsewhere rewards have gone more consistently to those who play the game rather than those who seek and even deliver genuine improvement.

Following practices, such as the Waterfall, V Model and Structured Methods, which are inefficient and ineffective but were widely regarded as “standard” or “best practice” for years reinforced kakonomic exchanges of low quality throughout the IT profession.

If people follow them and the results are bad than they can always pretend that they tried to do it right, but were unlucky. No-one is really to blame, or accountable, and we can all carry on as before. The rhetoric is impeccably high quality. The reality is garbage.

If people deliver something special and high quality then they are probably iconoclasts or rebels who have rejected the methods that would have ensured low quality failure. They have therefore rejected the cosy consensus of the status quo and set a challenge to the organisation.

Whereas those who fail do so conventionally, and can therefore be excused as unlucky but dependable, the rebels are seen as difficult, spiky individuals. Their success is a one off, which can’t offer a pattern, being dependent on individual skill rather than repeatable process.

There’s a blog to be written about whether CMMI is misused so that it encourages kakonomic exchanges by implicitly valuing repeatable, predictable failure above erratic, but sometimes brilliant individualism.

Anyway, the rebels have breached the subliminal contract of the kakonomic exchange, leaving their colleagues distinctly uncomfortable. No-one would admit to the real reason for that discomfort, and they happily ascribe it to concern about the rebels’ cavalier style, which creates friction and harms morale. They are “difficult”, “not team players”.

The conventional team players glide up the organisation, and if the rebels are wise they head for an organisation where sparks are welcomed, rather than feared.

Is Agile a panacea?

I’m not sure there’s much for proponents of Agile to be complacent about here. If Agile is badly implemented then I’d expect exactly the same reinforcement of bad practice. The trendier Agile becomes, the greater is the danger that it will be adopted for the bogus high quality rhetoric.

Perhaps Agile’s greatest strength in the long run is its adaptability and flexibility, rather than any particular technique. Any shrewd practitioner should surely realise that complacency is deadly to Agile, and would be attuned to the dangers of kakonomic exchanges, even if the term and the theory meant nothing to them. In principle, I’d have thought that such exchanges could occur anywhere.

What about testers?

I think that kakonomics can explain some of the daft, damaging and dysfunctional behaviour that happens in organisations. It might explain why people persist in doing things that are entirely inconsistent with their supposed objectives, and why they will do so in the sure knowledge that their behaviour will not be challenged.

Gloria Origgi is rightly critical of the damage this all does. Her analysis is not just an amusing reflection on the vagaries of life.

She explains that the reason “it is a form of collective insanity so difficult to eradicate, is that each low-quality exchange is a local equilibrium in which both parties are satisfied, but each of these exchanges erodes the overall system in the long run”.

Testing should be a comfortable home for those who are happy with creating and suffering a little discomfort! It should be our job not to tell people what they want to hear, or to go through the motions, or to follow the process blindly, but to shine a light on what’s really there.

In doing so it helps to think about why people behave as they do, and to understand why good people do bad things.

I don’t think it always applies, and it’s not a magical insight that explains everything, but maybe kakonomics is one of those things that can help us understand a little better, and to explain just why we are seeing damaging behaviour. It might not make us popular, but unless the organisation is dying there should always be a place for unpopular, truthful insights about why we get poor quality.

One of the things about testing that I like is that it is a profession that has genuine respect for the teller of those unpopular truths!

However, I’ve got to admit that there’s considerable reluctance to acknowledge the problems are real, and that the the solutions are relevant and practical. They won’t guarantee success, but since the traditional alternative load the odds heavily against success they’ve got to be an improvement.

The predictability paradox

There are two uncomfortable truths you have to acknowledge; the general problem and the specific one. In general, you can’t specify requirements accurately and completely up front, and any attempt to do so is doomed. The specific problem is that usability requirements are particularly difficult to specify.

Managers crave certainty and predictability. That’s understandable. What’s dangerous is the delusion that they exist when the truth is that we cannot be certain at the start of a development and we cannot predict the future with the level of confidence that we’d like.

It’s hard to stand up and say “we don’t know – and at this stage we couldn’t responsibly say that we do know”, even though that might be the truthful, honest and responsible thing to say.

It’s always been hugely tempting to pretend that you can set aside a certain amount of time during which you will get the requirements right in a single pass. It makes procurement, planning and management simpler.

Sadly any attempt to do so is like trying to build a house on shifting sand. The paradox is that striving for predictability pushes back the point when you truly can find out what is required and possible.

Mary Poppendieck made the point very well a few years back in an excellent article “Lean Development and the Predictability Paradox”. Unfortunately, it’s no longer available on-line on her website, though you can buy it from the Cutter Consortium’s website at a rather steep $150. However, the important part of the paper is quoted in Skip Angel’s blog.

I tried to argue much the same point in an article in 2008, “The Dangerous and Seductive V Model”. an article that has proved scarily popular on my website for the last couple of years. I guess many people still need to learn these lessons.

Unrighteous contracts?

Trying to create predictability by tying suppliers into tight contracts is a result of the delusion that predictability is available at an early stage. This can have perverse results.

Tight and tough contracts committing suppliers right at the start of a project is an understandable attempt to contain risk, or transfer it to the supplier.

Sadly, clients never truly transfer risk to the supplier if the risk is better handled internally. If the requirements are complex and poorly understood at the start then attempting to transfer the risk to the supplier is misguided.

Awarding a contract for the full development creates the danger that it will be awarded to the most optimistic, inexperienced or irresponsible supplier. Those suppliers with a shrewd understanding of what is really likely to be involved will tender cautiously and be undercut by the clueless, reckless or unprincipled.

When the wheels come off the project the client may be able to hold the supplier to the contract, but the cost will be high. Changes will be fought over, and the supplier will try to minimise the work – regardless of the impact on quality.

There seem to be three options about how to handle the contractual relationship with an external supplier if you want a suitable focus on quality, and especially usability.

The orthodox approach is to split the contract in two. The first contract would be to establish, at a high level, what is required, and ideally to deliver an early prototype. The second would be to build the full application.

This approach was recommended way back in 1987 by the US Defense Science Board Task Force On Military Software (PDF – opens in new window), and its motives were exactly the same as I’ve outlined.

The Agile approach would probably be to write contracts that have a budget for time and cost, and allow the scope to vary, possibly giving the client the right to cancel at set points, possibly after every iteration.

Martin Fowler summarises that approach well in this article, “The New Methodology”.

The third, more radical, approach was mooted by Mary Poppendieck in her article “Righteous Contracts”.

These “righteous contracts” would be written after the development, not at the start. I’m not convinced. It doesn’t seem all that different from simply dispensing with a contract and relying on trust, but I urge you to read her piece. She analyses the problems well and has some challenging ideas. I’d like to see them work, but they do seem rather idealistic.

Righteous contracts would work only if the client and supplier trust each other and are committed to a long term relationship. However, I’ve got to concede that without that trust and commitment the client is going to have problems with their suppliers regardless of the contractual style. You can’t create a productive partnership with a contract, but you can destroy it if the contract is poorly written and conceived.

Enough of the contracts! What about UX?

This piece was really prompted by my concerns about the effects of the contractual relationship between clients and suppliers, so not surprisingly I’ve been focussing on contracts.

However, I must admit I’ve found that rather dispiriting. So in order to feel a bit more positive I’m going to provide some pointers about how to try and handle usability better.

I discussed this in an article in 2009. It’s all about Agile and UX, but that’s where the interesting work is being done, and honestly, I’ve nothing positive to say about how traditional, linear techniques have handled usability. It might be possible to smuggle in techniques like Nielsen’s Discount Usability Engineering, or more ambitiously, Constantine & Lockwood’s Collaborative Usability Inspections. However, remember we’re talking about external suppliers and the contractual relationship. These techniques require iteration, and if the contract doesn’t make explicit allowance for iterative development then the odds are stacked hopelessly against you.

So Agile really seems to be the best hope for taking account of usability when dealing with an external supplier. Rather embarrassingly I came across this excellent article by Jeff Paton only a few days ago. It dates from 2008 and I really wish I’d found it earlier. It lists 12 “best practice” patterns that Paton has seen people using to incorporate UX into agile development. There’s some great stuff there.

A somewhat downbeat conclusion

Perhaps I shouldn’t have talked about “solutions” to the usability problems caused by using external suppliers. Maybe I should have used a phrase like “a better way”. I believe that flexible, staged contracts are vital. They are necessary, but they’re not sufficient. A commitment to usability is required regardless of the approach that is taken. It’s just that the traditional fixed price contract for a linear development torpedoes any hope of getting usability right. The damage is far more widespread than that, but usability is the issue that gets me particularly passionate.

I doubt if I’ve set off a light bulb in anyone’s head and they’re now thinking, “that’s what we’ve been doing wrong and I know what to do now”. However, I hope I’ve pointed some people in a useful direction to help them read, think and understand a bit more about what we do wrong.

I don’t know all the answers. Testers never do, and the big decisions that will affect usability and quality are a long way out of our remit. But we can, and should, be aware of the pitfalls of following particular approaches. We’ve a duty to speak out and tell management what can go wrong, and that just might tilt the odds back in our favour.

One thing that intrigues me is the effect on usability of the differing approaches, and in particular the implications of using external suppliers. It’s not just that different approaches have different consequences. There’s nothing surprising in that. The intriguing thing is that, in this case, the implications are not widely acknowledged. It’s an inconvenient truth.

The more distant, formal and contractual the relationship between developers and users, the less likely it is that usability principles will be incorporated and that the end users will get a product they enjoy using.

This isn’t a problem that arose with e-commerce and web developments. It dates way back to the days when all software developments were applications for employees to use. Academics spotted the problems, but their work didn’t really percolate through to the consciousness of the practitioners.

Lewis & Rieman, Holmlid & Artman (both PDFs & open in new windows), Artman on his own, and Grudin all made similar points regarding procurement, external suppliers and contracts.

Unfortunately only the Holmlid and Artman article is free, though the Lewis and Rieman e-book has a modest suggested donation.

The conclusion of these writers was that using external suppliers was likely to have a damaging impact on the usability of the application. With external contracts there is more pressure and temptation to go for a traditional linear approach.

Effective usability engineering entails iteration, which in turn requires flexible contracts with estimates and costs being revised repeatedly. This is a frightening prospect for both sides; with the supplier scared of being committed to a job which cannot be sized effectively, and the client equally scared of being tied into a contract with no cost cap, and no easy exit without writing off the work already paid for.

In 1995 a workshop was held by the ACM’s Special Interest Group on Computer-Human Interaction to consider the challenges of introducing usability to US government developments. In addition to the general problems faced by all IT projects the participants concluded that very few invitations to tender mentioned usability beyond vague and subjective aspirations. Suppliers naturally didn’t build into their costing any features that were not explicitly mandated, and even after winning the contract were reluctant to provide them lest they get a reputation for cost over-runs.

The research is dated, but I don’t think that anything has essentially changed. The problems weren’t rooted in the technology, or development techniques. The problems were a human reaction to the pressures and incentives of a particular style of procurement. People haven’t evolved to a nobler, more altruistic level since the 90s, and people will still react the same way when confronted with the same approach to contracts and procurement.

Effective contracts have to be clear and objective. It is hard enough to specify usability requirements clearly and objectively. Stating them in any useful way at the contractual level is even harder.

Of course none of this necessarily means that external suppliers will do a bad job; far from it. What it does mean is that the external supplier relationship contains problems that must be explicitly acknowledged and addressed. Too often the implicit assumption is that using an external supplier is a low risk option, provided that they can be tied down to a tight contract. That sets up an adversarial relationship that is poison to the flexible approach that is essential if the relationship is really going to work.

No plan survives contact with the enemy. No prototype design survives exposure to the users unchanged. Why pretend that it can?

What are the implications for testers? Well, we are not mere checkers. It is our job to tell it like it us, and shine a light on those awkward facts and truths that others might prefer to ignore.

If your management is keen to follow a route that will have worrying implications, make sure they know what these are so they take the decision with the best information available. Who better than a tester to give them the news that management might not want, but really need to know?

I’ve tried to keep this fairly brief, so I’m not talking about how to make the situation better. That’s for next time (click here).

However, with complex applications, especially financial systems, it can be harder. You’re expecting a big number, and you get one. It’s hard to be sure if it’s right. If it were easy then you probably wouldn’t be developing a complex application to churn out the big numbers.

What you can do, and what many people seem to forget about, is to run a quick sanity check to see if it makes sense. You can run a quick mental calculation based on what you do know, and a few assumptions.

If the answer is daft, or way out of line with the test result, then you’ve found something out. Either the test result is wrong, or what you confidently think you know is wrong, or your assumptions don’t match the reality. One way or the other, you’re going to find out more about the application, or improve your knowledge about the context of the application.

When I was reading the paper at breakfast this morning I came across a good example of a glaring failure to run a sanity check.

The Association of British Drivers (ABD) is a campaigning group in the UK that fights for the motoring lobby, and against action on green issues, which are dismissed as “scares”.

Yesterday they issued a press release calling on the Scottish Government to rush through investment for the A9, the main highway from central Scotland through the Highlands. The road has a bad safety record, and there has been a long-running debate about whether it should be upgraded to dual carriageway (divided highway).

Naturally, the ABD is all in favour of an upgrade. Their press release claimed that the cost of the Scottish Government’s Car Service was £900 million a year, £300 million more than the cost of upgrading the A9. The Government Car Service provides cars and drivers for ministers and the most senior government officials.

You can imagine their delight at discovering this flagrant example of government waste. Cut it out, divert the money to infrastructure improvement, and produce a net saving.

Unfortunately, no-one ran the sanity check.

Let’s assume the cars are big and flashy, and cost £60,000 each. They’re replaced every three years and have no resale value. Let’s also assume that the cost of a driver is £40,000 a year. So that’s £60,000 per car per year. If the cost of running the fleet is £900 million a year, then the fleet has 15,000 cars. The latest figures I could find showed that there are 15,263 civil servants in Scotland.

So 15,000 government drivers are sitting around, taking it in turns to drive the other 263 people round the country. Now my assumptions were crude, but it’s impossible to change them in any sensible way that produces an answer I can believe. So, oops, sanity check fail!

The ABD were out by a factor of 1,000. The cost of the Government Car Service is £900,000. The fleet has 25 cars. My quick calculation suggested 15 cars could be paid for with £900,000. A 40% error is usually pretty sloppy, but it would have been near enough the mark to spare the ABD the embarrassment of withdrawing a press release because no-one bothered to run the sanity check.

The answer looked so wonderful that no-one wanted to challenge it. My quick check took little more than a minute, including googling for the number of civil servants.

Always, always be wary of answers that seem to be exactly what you hoped for! Always think your way round big numbers. £900,000 and £900 million might both be big, but if you use your existing knowledge, and some crude assumptions, you can tease the numbers apart and start to distinguish the plausible from the ludicrous.

Being wrong by a factor of 1,000 can damage your credibility!