When documentation is a waste of time

Documentation is like eating and drinking. It’s essential, but in the right quantities, at the right time and in the right place. You don’t want to get obese, and you certainly don’t want to drown.

Yet some companies just can’t get a healthy sense of perspective with documentation. There’s no sense of “just enough”. They always seem to need more than is required, as if documentation is an end in itself.

I’ve been following Michael Bolton’s short blog series on test cases, part 1 and part 2, which I strongly recommend. As well as looking at the specific question of test cases, the wider issue of documentation is discussed, especially in the comments following the second piece.

What really gets me about the demand for skiploads of documentation is that it comes at a huge price. The time I’ve had to spend pointlessly perfecting documents that weren’t needed could have been better spent understanding the application, the business problem and the technology. I’ve sometimes had the dispiriting insight that I knew my way round the documentation far better than I knew the application.

Fiona Charles and Michael refer to a belief in documentation as being a phase that testers go through. Read their comments in part 2 on Michael’s blog. I certainly recognise that pattern.

For me the epiphany, the moment when the last illusions vanished, was on a project where the demand for documentation was insatiable, the deadlines fixed, and the testers too few.

As usual the documentation was produced from the top down; from the contract to the strategy, to the stage test plans, to the detailed test plans, to the test cases.

We were running out of time, but there was no let up in the demand for documentation. Massive test plans had to be refined and reworked even as we were moving into text execution, with only a few sketchy test case scripts in place.

The inevitable consequence was that the test execution was based almost entirely on the judgement, experience and knowledge of the business that the testers had, rather than all the paper plans. The perverse result of confusing documentation with structure was that the test execution was largely improvised and unstructured.

The huge effort that went into the higher level documentation was wasted because it never fed through into the detailed scripts. The test execution was pretty much what would have happened in the absence of all that documentation. Well, maybe not quite like that. If the testers had not been forced to churn out unwanted documentation they would have been actively involved earlier in the project, detecting and forestalling defects.

We could have done so much better for the project if we’d been allowed to select an approach that fitted the real problem, as opposed to a barely relevant process that was driven by an addiction to documentation, rather than by the risk. We could then have planned and prepared for the test execution in a way that would have helped us test better.

The actual testing was well informed and very valuable, but that was despite the documentation, not because of it. So much time was simply wasted in the preparation.

Even during the execution a lot of time was wasted attempting to retro-fit the execution to the test plans so that we could manipulate Quality Center to produce the reports for stakeholders who had such a poor grasp on our efforts that they couldn’t distinguish between the plans and reports and the reality that they were supposed to represent.

My epiphany was the awful realisation that the project was so dysfunctional that the documentation had become the reality.

One of the comments in Michael’s blog referred to the perceived need for test case documentation to satisfy the auditors. This is a red herring. Good auditors don’t need to see documented test cases. They need to see evidence of testing that was appropriate in the circumstances.

Of course there are plenty of bad auditors around, but I think that’s usually a result of inexperience or an unhealthy corporate culture. In any case the best response is to speak to them, to understand them and to change their attitudes if possible.

Whatever you do don’t try to second guess them and produce documentation in the expectation that they require it. When I was an auditor it would drive me up the wall when I found people wasting their time on documentation simply because they wrongly believed I would need to see it.

If you want some more ammunition to try and get auditors onside you could tell them to look at the Sarbanes Oxley guidance material from the Information Systems Audit & Control Association. It’s available to members only on their website. Let me know if you want to talk about it. I have written about the requirements of Sarbanes-Oxley on this blog.

Sarbox may apply only to the USA, but its requirements are fairly stringent, and you’d have a very strong argument if you could show that not even it requires detailed test scripting. The trouble is that Sarbox has the reputation for demanding extensive documentation, and some auditors go by the reputation rather than the legislation.

Or look at this article by Kanwal Mookhey in the Internal Auditor magazine. Kanwal described the checking that auditors should carry out at each stage of a project. He made no mention of the need to see documentation of detailed test plans and scripts whereas he did emphasize the need for early testing.
Kanwal told me.

I would agree that auditors are – or should be – more inclined to see comprehensive testing, rather than comprehensive test documentation.

Documentation of test results is another matter of course. As an auditor, I would be more keen to know that a broad-based testing manual exists, and that for the system in question, key risks and controls identified during the design phase have been tested for. The test results would provide a higher degree of assurance than exhaustive test plans.

I used that quote in this article I wrote for Testing Experience magazine, in which I argued against the adoption of standards that place more emphasis on bulk documentation than testing expertise. I wrote it before the appearance of ISO 29119, so I can hardly claim my article was influential!

Documentation must never be an end in itself. The analogy I drew with eating and drinking at the start wasn’t perfect. You might eat and drink for pleasure, but no sane person produces documentation for the joy of stacks of shelfware. Do yourself, and your projects a favour by challenging the unthinking demand for more and more detailed documentation.