Documentation is like eating and drinking. It’s essential, but in the right quantities, at the right time and in the right place. You don’t want to get obese, and you certainly don’t want to drown.
Yet some companies just can’t get a healthy sense of perspective with documentation. There’s no sense of “just enough”. They always seem to need more than is required, as if documentation is an end in itself.
I’ve been following Michael Bolton’s short blog series on test cases, part 1 and part 2, which I strongly recommend. As well as looking at the specific question of test cases, the wider issue of documentation is discussed, especially in the comments following the second piece.
What really gets me about the demand for skiploads of documentation is that it comes at a huge price. The time I’ve had to spend pointlessly perfecting documents that weren’t needed could have been better spent understanding the application, the business problem and the technology. I’ve sometimes had the dispiriting insight that I knew my way round the documentation far better than I knew the application.
Fiona Charles and Michael refer to a belief in documentation as being a phase that testers go through. Read their comments in part 2 on Michael’s blog. I certainly recognise that pattern.
For me the epiphany, the moment when the last illusions vanished, was on a project where the demand for documentation was insatiable, the deadlines fixed, and the testers too few.
As usual the documentation was produced from the top down; from the contract to the strategy, to the stage test plans, to the detailed test plans, to the test cases.
We were running out of time, but there was no let up in the demand for documentation. Massive test plans had to be refined and reworked even as we were moving into text execution, with only a few sketchy test case scripts in place.
The inevitable consequence was that the test execution was based almost entirely on the judgement, experience and knowledge of the business that the testers had, rather than all the paper plans. The perverse result of confusing documentation with structure was that the test execution was largely improvised and unstructured.
The huge effort that went into the higher level documentation was wasted because it never fed through into the detailed scripts. The test execution was pretty much what would have happened in the absence of all that documentation. Well, maybe not quite like that. If the testers had not been forced to churn out unwanted documentation they would have been actively involved earlier in the project, detecting and forestalling defects.
We could have done so much better for the project if we’d been allowed to select an approach that fitted the real problem, as opposed to a barely relevant process that was driven by an addiction to documentation, rather than by the risk. We could then have planned and prepared for the test execution in a way that would have helped us test better.
The actual testing was well informed and very valuable, but that was despite the documentation, not because of it. So much time was simply wasted in the preparation.
Even during the execution a lot of time was wasted attempting to retro-fit the execution to the test plans so that we could manipulate Quality Center to produce the reports for stakeholders who had such a poor grasp on our efforts that they couldn’t distinguish between the plans and reports and the reality that they were supposed to represent.
My epiphany was the awful realisation that the project was so dysfunctional that the documentation had become the reality.
One of the comments in Michael’s blog referred to the perceived need for test case documentation to satisfy the auditors. This is a red herring. Good auditors don’t need to see documented test cases. They need to see evidence of testing that was appropriate in the circumstances.
Of course there are plenty of bad auditors around, but I think that’s usually a result of inexperience or an unhealthy corporate culture. In any case the best response is to speak to them, to understand them and to change their attitudes if possible.
Whatever you do don’t try to second guess them and produce documentation in the expectation that they require it. When I was an auditor it would drive me up the wall when I found people wasting their time on documentation simply because they wrongly believed I would need to see it.
If you want some more ammunition to try and get auditors onside you could tell them to look at the Sarbanes Oxley guidance material from the Information Systems Audit & Control Association. It’s available to members only on their website. Let me know if you want to talk about it. I have written about the requirements of Sarbanes-Oxley on this blog.
Sarbox may apply only to the USA, but its requirements are fairly stringent, and you’d have a very strong argument if you could show that not even it requires detailed test scripting. The trouble is that Sarbox has the reputation for demanding extensive documentation, and some auditors go by the reputation rather than the legislation.
Or look at this article by Kanwal Mookhey in the Internal Auditor magazine. Kanwal described the checking that auditors should carry out at each stage of a project. He made no mention of the need to see documentation of detailed test plans and scripts whereas he did emphasize the need for early testing.
Kanwal told me.
I would agree that auditors are – or should be – more inclined to see comprehensive testing, rather than comprehensive test documentation.
Documentation of test results is another matter of course. As an auditor, I would be more keen to know that a broad-based testing manual exists, and that for the system in question, key risks and controls identified during the design phase have been tested for. The test results would provide a higher degree of assurance than exhaustive test plans.
I used that quote in this article I wrote for Testing Experience magazine, in which I argued against the adoption of standards that place more emphasis on bulk documentation than testing expertise. I wrote it before the appearance of ISO 29119, so I can hardly claim my article was influential!
Documentation must never be an end in itself. The analogy I drew with eating and drinking at the start wasn’t perfect. You might eat and drink for pleasure, but no sane person produces documentation for the joy of stacks of shelfware. Do yourself, and your projects a favour by challenging the unthinking demand for more and more detailed documentation.
Great read James. Just thought I would add that in Canada we have what is known as C-SOX aka Bill 198. For those interested you can read more about it here.
Thanks Lynn. It’s interesting that that is an Ontario act. I suppose the Ontario economy is even more closely linked to the US than is the case with the rest of Canada. I guess there were practical reasons why Ontario would want to pay heed to Sarbox.
Sarbox hasn’t affected us on this side of the Atlantic in quite such a direct way, but it’s something that companies dealing with the US need to be aware of. I worked for IBM, and it certainly affected us. It influenced the culture. It wasn’t a matter of legal compliance.
Good read James,
I am working on incorporating regulatory demands into a generic DoD for information systems affected. I would like to recieve a copy from the Information Systems Audit & Control Association as you offered.
Thank you Jean-Paul. You have mail. Let me know if I can help further.
Fabulous article on documentation. Must read for all the programmers and testers. I would agree with you when you say Documentation is like food and water. (right quantities, at the right time and in the right place).
Pranjal R Nigam
Thank you Pranjal. I’m sorry I didn’t thank you for your comment earlier.
I just wrote an article about test cases in agile for a testing e-zine. After submitting it I heard that you have also blogged about documentation as a waste. So I googled up your blog and funny thing – this post begins with almost exactly the same wording I used in my article without knowing about it.
So, probably needless to say but I somewhat completely agree with the points you make here. It’s a great post and really digs in to the problem of documentation neurosis that so many people seem to be suffering from.
Thanks Petteri. Was your article published? Can you give me a link to it? I’m not surprised other testers have thought exactly the same as me. It’s a very common problem.
I see you’re in Helsinki. I worked there for a while a few years back and loved the city. I was based in Munkkiniemi, working there and in Espoo.
The article is not yet published but it will be in the next issue of “Teatime with Testers” ezine.
Yes, Helsinki is one of the more bearable cities in Finland. 😉
I also like Oulu, but I’ve been there only in the summer!
Every conservative workplace I’ve been at is littered with rubbish PDFs that go for 500+ pages, you open one and the first thing you see is something like “July 1, 2007” stamped across the top. Hours upon hours some fool has sat there, typed all this up and it is absolutely useless. Documentation made sense in 1970 when the same old thing happened on a daily basis for 30+ years. Not so much now. If you start a document, you better commit to maintenance and succession planning. Workplaces should audit documents a person controls before they’re able to leave, in the same way you must return/reassign physical belongings.