Often it’s obvious to testers when something hasn’t worked. The application crashes, there’s no output, or it’s gibberish. Sometimes the output superficially looks ok, but in context it’s patent nonsense, e.g. a schoolchild’s age is calculated at 25.
However, with complex applications, especially financial systems, it can be harder. You’re expecting a big number, and you get one. It’s hard to be sure if it’s right. If it were easy then you probably wouldn’t be developing a complex application to churn out the big numbers.
What you can do, and what many people seem to forget about, is to run a quick sanity check to see if it makes sense. You can run a quick mental calculation based on what you do know, and a few assumptions.
If the answer is daft, or way out of line with the test result, then you’ve found something out. Either the test result is wrong, or what you confidently think you know is wrong, or your assumptions don’t match the reality. One way or the other, you’re going to find out more about the application, or improve your knowledge about the context of the application.
When I was reading the paper at breakfast this morning I came across a good example of a glaring failure to run a sanity check.
The Association of British Drivers (ABD) is a campaigning group in the UK that fights for the motoring lobby, and against action on green issues, which are dismissed as “scares”.
Yesterday they issued a press release calling on the Scottish Government to rush through investment for the A9, the main highway from central Scotland through the Highlands. The road has a bad safety record, and there has been a long-running debate about whether it should be upgraded to dual carriageway (divided highway).
Naturally, the ABD is all in favour of an upgrade. Their press release (NB The Courier has updated its story to report the ABD’s mistake) claimed that the cost of the Scottish Government’s Car Service was £900 million a year, £300 million more than the cost of upgrading the A9. The Government Car Service provides cars and drivers for ministers and the most senior government officials.
You can imagine their delight at discovering this flagrant example of government waste. Cut it out, divert the money to infrastructure improvement, and produce a net saving.
Unfortunately, no-one ran the sanity check.
Let’s assume the cars are big and flashy, and cost £60,000 each. They’re replaced every three years and have no resale value. Let’s also assume that the cost of a driver is £40,000 a year. So that’s £60,000 per car per year. If the cost of running the fleet is £900 million a year, then the fleet has 15,000 cars. The latest figures I could find showed that there are 15,263 civil servants in Scotland.
So 15,000 government drivers are sitting around, taking it in turns to drive the other 263 people round the country. Now my assumptions were crude, but it’s impossible to change them in any sensible way that produces an answer I can believe. So, oops, sanity check fail!
The ABD were out by a factor of 1,000. The cost of the Government Car Service is £900,000. The fleet has 25 cars. My quick calculation suggested 15 cars could be paid for with £900,000. A 40% error is usually pretty sloppy, but it would have been near enough the mark to spare the ABD the embarrassment of withdrawing a press release because no-one bothered to run the sanity check.
The answer looked so wonderful that no-one wanted to challenge it. My quick check took little more than a minute, including googling for the number of civil servants.
Always, always be wary of answers that seem to be exactly what you hoped for! Always think your way round big numbers. £900,000 and £900 million might both be big, but if you use your existing knowledge, and some crude assumptions, you can tease the numbers apart and start to distinguish the plausible from the ludicrous.
Being wrong by a factor of 1,000 can damage your credibility!