The good old sanity check

Often it’s obvious to testers when something hasn’t worked. The application crashes, there’s no output, or it’s gibberish. Sometimes the output superficially looks ok, but in context it’s patent nonsense, e.g. a schoolchild’s age is calculated at 25.

However, with complex applications, especially financial systems, it can be harder. You’re expecting a big number, and you get one. It’s hard to be sure if it’s right. If it were easy then you probably wouldn’t be developing a complex application to churn out the big numbers.

What you can do, and what many people seem to forget about, is to run a quick sanity check to see if it makes sense. You can run a quick mental calculation based on what you do know, and a few assumptions.

If the answer is daft, or way out of line with the test result, then you’ve found something out. Either the test result is wrong, or what you confidently think you know is wrong, or your assumptions don’t match the reality. One way or the other, you’re going to find out more about the application, or improve your knowledge about the context of the application.

When I was reading the paper at breakfast this morning I came across a good example of a glaring failure to run a sanity check.

The Association of British Drivers (ABD) is a campaigning group in the UK that fights for the motoring lobby, and against action on green issues, which are dismissed as “scares”.

Yesterday they issued a press release calling on the Scottish Government to rush through investment for the A9, the main highway from central Scotland through the Highlands. The road has a bad safety record, and there has been a long-running debate about whether it should be upgraded to dual carriageway (divided highway).

Naturally, the ABD is all in favour of an upgrade. Their press release (NB The Courier has updated its story to report the ABD’s mistake) claimed that the cost of the Scottish Government’s Car Service was £900 million a year, £300 million more than the cost of upgrading the A9. The Government Car Service provides cars and drivers for ministers and the most senior government officials.

You can imagine their delight at discovering this flagrant example of government waste. Cut it out, divert the money to infrastructure improvement, and produce a net saving.

Unfortunately, no-one ran the sanity check.

Let’s assume the cars are big and flashy, and cost £60,000 each. They’re replaced every three years and have no resale value. Let’s also assume that the cost of a driver is £40,000 a year. So that’s £60,000 per car per year. If the cost of running the fleet is £900 million a year, then the fleet has 15,000 cars. The latest figures I could find showed that there are 15,263 civil servants in Scotland.

So 15,000 government drivers are sitting around, taking it in turns to drive the other 263 people round the country. Now my assumptions were crude, but it’s impossible to change them in any sensible way that produces an answer I can believe. So, oops, sanity check fail!

The ABD were out by a factor of 1,000. The cost of the Government Car Service is £900,000. The fleet has 25 cars. My quick calculation suggested 15 cars could be paid for with £900,000. A 40% error is usually pretty sloppy, but it would have been near enough the mark to spare the ABD the embarrassment of withdrawing a press release because no-one bothered to run the sanity check.

The answer looked so wonderful that no-one wanted to challenge it. My quick check took little more than a minute, including googling for the number of civil servants.

Always, always be wary of answers that seem to be exactly what you hoped for! Always think your way round big numbers. £900,000 and £900 million might both be big, but if you use your existing knowledge, and some crude assumptions, you can tease the numbers apart and start to distinguish the plausible from the ludicrous.

Being wrong by a factor of 1,000 can damage your credibility!

withdrawn due to error

What do they know of testing who only testing know?

I know I’m not a particularly touchy feely person. As a child and as a student. I was quite shy and I was more comfortable with books, writing and solving mental problems than I was dealing with other people. In the early part of my career I had a vague notion that my job was all about the technology. Sure, handling people sensitively and with respect was important, but that was so I could get on well with them. It wasn’t really part of getting the job done.

I started to change when I became an IT auditor. It took me maybe two years to get the hang of a job that I gradually came to realise required judgement, experience, intuition and insight; not just technical expertise. Every audit required me to analyse the situation and understand it. “What’s going on here? Why are things being done this way? How could they be improved?”. You can’t answer these questions without understanding the people, and the motives that are guiding their behaviour.

I suppose an important aspect of growing up and maturing is the ability to acknowledge your weaknesses so you can do something about it. Often, if you’re successful, you can con people that your weakness is actually a strength. By the way, I’m terrified of public speaking, but I’ve been able to fool loads of people that I enjoy it!

Client v supplier, or client & supplier?

Thinking about the human side of business proved vital when I moved from internal IT work to a role in which I was a supplier dealing with customers. For many IT professionals this is a minefield. You have to be so careful, and many of us lack the instincts and training to handle it.

Many people have been brutalised by adversarial relationships between suppliers and clients. Some clients see it as their job to challenge suppliers (acceptable), but this can tip over into confrontation, hostility and personal attacks (unacceptable). After some experience of that, supplier staff can start to think that any response short of physical violence is acceptably diplomatic.

When I worked for a big supplier they took care to send the cool heads with long fuses into the more stressful customer facing situations, and to keep the hot heads in the background. I was one of the cool heads. You need people who will be polite and professional, but who are prepared to engage with clients as people first, and clients second. Once you build a personal relationship with the client then the business becomes much easier. It helps if there is continuity on both sides, rather than the supplier rotating key staff every few months.

In my career I’ve met very few people who have been irredeemably unpleasant and obnoxious. Mostly when clients have been difficult it has been in response to pressures from within their organisation. If you understand their position, and understand them as people, then it’s usually possible to work with them pleasantly and constructively. It’s not a fair game, and sometimes client staff will treat suppliers with a lack of respect that they wouldn’t dream of tolerating in return. Suppliers have to be much more careful and sensitive than client staff. That’s just the way it is.

It’s pragmatic, not sentimental

Dealing with people as individuals, understanding their motivation, and showing them respect isn’t just the right way to treat people, but it’s also a highly practical way of handling clients. Once you have persuaded the client that you are doing your best on their behalf, and also of course that you are competent, everything becomes easier. The client perceives that you have hit tough problems that you are trying to resolve, not that you have failed. That’s because you’ve earned their trust and respect. Earned is the key word. It doesn’t just happen. You have to work at it.

There have been many times in my career when the time I’ve invested in early stages of a project getting to know and understand the client staff well has paid dividends later; when we hit serious problems, when I had to persuade the client to change their strategy, when I had to persuade the client that the contract had mistaken assumptions.

I was able to get my way not through bullying, or manipulation, or scare tactics, but because I was speaking to people who already respected, and above all trusted me. If the client had thought I was a chancer, unreliable or concerned only with my employer’s short term interests then discussion would have been closed down. At best I’d have been told brusquely to get on with it. At worst there would have been a shouting match and escalation to senior management to bring me into line.

Obviously, the longer you spend with a particular client the better the personal relationships you can build up. However, even on short term assignments the personal touch mustn’t be overlooked. In fact, the shorter the assignment the more important it can be. If you’re doing a consultancy exercise lasting only a few days then the people might be the highest priority. Instead of a dispassionate analysis of what is being done, and how it is being carried out, you can get a better, quick feel by seeing how people are responding to their environment and gauging their experience and knowledge.

I’ll never be a social worker or a counsellor, and I do screw up, but I think I’ve learnt enough to counter my worst instincts. Homing in on technical issues, statistics and processes might seem rigourous and objective. Often it’s just the comfortable option. In the end, it’s all about people. Come to think of it, they should be the starting point too.

These issues might be more sharply defined in commercial relationships between clients and suppliers, but they are still relevant to internal IT staff who deal only with fellow employees of the same company. They’re also highly relevant when dealing with other members of the same project or team.

What about testing?

What’s all that got to do with testing? Nothing and everything. I’ve quite deliberately reached this stage without mentioning testing.

The purpose of testing is to let the right people have the information they require about the product and the risks. The product is usually destined for people to use. Testing is not a mechanical process of checking the product against the requirements. Ultimately it’s about people, and to understand our role we have to understand the people, whether they are colleagues, managers, clients or end users.

If we only know testing as a technical, dispassionate process then we don’t really know what we’re doing. That’s why I mangled Kipling’s quote about England for my title. What do they know of testing who only testing know?