Documentation is like eating and drinking. It’s essential, but in the right quantities, at the right time and in the right place. You don’t want to get obese, and you certainly don’t want to drown.
Yet some companies just can’t get a healthy sense of perspective with documentation. There’s no sense of “just enough”. They always seem to need more than is required, as if documentation is an end in itself.
I’ve been following Michael Bolton’s short blog series on test cases, part 1 and part 2, which I strongly recommend. As well as looking at the specific question of test cases, the wider issue of documentation is discussed, especially in the comments following the second piece.
What really gets me about the demand for skiploads of documentation is that it comes at a huge price. The time I’ve had to spend pointlessly perfecting documents that weren’t needed could have been better spent understanding the application, the business problem and the technology. I’ve sometimes had the dispiriting insight that I knew my way round the documentation far better than I knew the application.
Fiona Charles and Michael refer to a belief in documentation as being a phase that testers go through. Read their comments in part 2 on Michael’s blog. I certainly recognise that pattern.
For me the epiphany, the moment when the last illusions vanished, was on a project where the demand for documentation was insatiable, the deadlines fixed, and the testers too few.
As usual the documentation was produced from the top down; from the contract to the strategy, to the stage test plans, to the detailed test plans, to the test cases.
We were running out of time, but there was no let up in the demand for documentation. Massive test plans had to be refined and reworked even as we were moving into text execution, with only a few sketchy test case scripts in place.
The inevitable consequence was that the test execution was based almost entirely on the judgement, experience and knowledge of the business that the testers had, rather than all the paper plans. The perverse result of confusing documentation with structure was that the test execution was largely improvised and unstructured.
The huge effort that went into the higher level documentation was wasted because it never fed through into the detailed scripts. The test execution was pretty much what would have happened in the absence of all that documentation. Well, maybe not quite like that. If the testers had not been forced to churn out unwanted documentation they would have been actively involved earlier in the project, detecting and forestalling defects.
We could have done so much better for the project if we’d been allowed to select an approach that fitted the real problem, as opposed to a barely relevant process that was driven by an addiction to documentation, rather than by the risk. We could then have planned and prepared for the test execution in a way that would have helped us test better.
The actual testing was well informed and very valuable, but that was despite the documentation, not because of it. So much time was simply wasted in the preparation.
Even during the execution a lot of time was wasted attempting to retro-fit the execution to the test plans so that we could manipulate Quality Center to produce the reports for stakeholders who had such a poor grasp on our efforts that they couldn’t distinguish between the plans and reports and the reality that they were supposed to represent.
My epiphany was the awful realisation that the project was so dysfunctional that the documentation had become the reality.
One of the comments in Michael’s blog referred to the perceived need for test case documentation to satisfy the auditors. This is a red herring. Good auditors don’t need to see documented test cases. They need to see evidence of testing that was appropriate in the circumstances.
Of course there are plenty of bad auditors around, but I think that’s usually a result of inexperience or an unhealthy corporate culture. In any case the best response is to speak to them, to understand them and to change their attitudes if possible.
Whatever you do don’t try to second guess them and produce documentation in the expectation that they require it. When I was an auditor it would drive me up the wall when I found people wasting their time on documentation simply because they wrongly believed I would need to see it.
If you want some more ammunition to try and get auditors onside you could tell them to look at the Sarbanes Oxley guidance material from the Information Systems Audit & Control Association. It’s available to members only on their website. Let me know if you want to talk about it.
The Institute of Internal Auditors has also provided guidance on Sarbox (PDF – opens in new window).
Sarbox may apply only to the USA, but its requirements are fairly stringent, and you’d have a very strong argument if you could show that not even it requires detailed test scripting. The trouble is that Sarbox has the reputation for demanding extensive documentation, and some auditors go by the reputation rather than the legislation.
Or look at this article by Kanwal Mookhey in the Internal Auditor magazine.
And this article (on my business site) I wrote, in which Kanwal gave me some more explicit quotes on this subject.
Documentation must never be an end in itself. The analogy I drew with eating and drinking at the start wasn’t perfect. You might eat and drink for pleasure, but no sane person produces documentation for the joy of stacks of shelfware. Do yourself, and your projects a favour by challenging the unthinking demand for more and more detailed documentation.