When documentation is a waste of time

Documentation is like eating and drinking. It’s essential, but in the right quantities, at the right time and in the right place. You don’t want to get obese, and you certainly don’t want to drown.

Yet some companies just can’t get a healthy sense of perspective with documentation. There’s no sense of “just enough”. They always seem to need more than is required, as if documentation is an end in itself.

I’ve been following Michael Bolton’s short blog series on test cases, part 1 and part 2, which I strongly recommend. As well as looking at the specific question of test cases, the wider issue of documentation is discussed, especially in the comments following the second piece.

What really gets me about the demand for skiploads of documentation is that it comes at a huge price. The time I’ve had to spend pointlessly perfecting documents that weren’t needed could have been better spent understanding the application, the business problem and the technology. I’ve sometimes had the dispiriting insight that I knew my way round the documentation far better than I knew the application.

Fiona Charles and Michael refer to a belief in documentation as being a phase that testers go through. Read their comments in part 2 on Michael’s blog. I certainly recognise that pattern.

For me the epiphany, the moment when the last illusions vanished, was on a project where the demand for documentation was insatiable, the deadlines fixed, and the testers too few.

As usual the documentation was produced from the top down; from the contract to the strategy, to the stage test plans, to the detailed test plans, to the test cases.

We were running out of time, but there was no let up in the demand for documentation. Massive test plans had to be refined and reworked even as we were moving into text execution, with only a few sketchy test case scripts in place.

The inevitable consequence was that the test execution was based almost entirely on the judgement, experience and knowledge of the business that the testers had, rather than all the paper plans. The perverse result of confusing documentation with structure was that the test execution was largely improvised and unstructured.

The huge effort that went into the higher level documentation was wasted because it never fed through into the detailed scripts. The test execution was pretty much what would have happened in the absence of all that documentation. Well, maybe not quite like that. If the testers had not been forced to churn out unwanted documentation they would have been actively involved earlier in the project, detecting and forestalling defects.

We could have done so much better for the project if we’d been allowed to select an approach that fitted the real problem, as opposed to a barely relevant process that was driven by an addiction to documentation, rather than by the risk. We could then have planned and prepared for the test execution in a way that would have helped us test better.

The actual testing was well informed and very valuable, but that was despite the documentation, not because of it. So much time was simply wasted in the preparation.

Even during the execution a lot of time was wasted attempting to retro-fit the execution to the test plans so that we could manipulate Quality Center to produce the reports for stakeholders who had such a poor grasp on our efforts that they couldn’t distinguish between the plans and reports and the reality that they were supposed to represent.

My epiphany was the awful realisation that the project was so dysfunctional that the documentation had become the reality.

One of the comments in Michael’s blog referred to the perceived need for test case documentation to satisfy the auditors. This is a red herring. Good auditors don’t need to see documented test cases. They need to see evidence of testing that was appropriate in the circumstances.

Of course there are plenty of bad auditors around, but I think that’s usually a result of inexperience or an unhealthy corporate culture. In any case the best response is to speak to them, to understand them and to change their attitudes if possible.

Whatever you do don’t try to second guess them and produce documentation in the expectation that they require it. When I was an auditor it would drive me up the wall when I found people wasting their time on documentation simply because they wrongly believed I would need to see it.

If you want some more ammunition to try and get auditors onside you could tell them to look at the Sarbanes Oxley guidance material from the Information Systems Audit & Control Association. It’s available to members only on their website. Let me know if you want to talk about it.

The Institute of Internal Auditors has also provided guidance on Sarbox (PDF – opens in new window).

Sarbox may apply only to the USA, but its requirements are fairly stringent, and you’d have a very strong argument if you could show that not even it requires detailed test scripting. The trouble is that Sarbox has the reputation for demanding extensive documentation, and some auditors go by the reputation rather than the legislation.

Or look at this article by Kanwal Mookhey in the Internal Auditor magazine.

And this article (on my business site) I wrote, in which Kanwal gave me some more explicit quotes on this subject.

Documentation must never be an end in itself. The analogy I drew with eating and drinking at the start wasn’t perfect. You might eat and drink for pleasure, but no sane person produces documentation for the joy of stacks of shelfware. Do yourself, and your projects a favour by challenging the unthinking demand for more and more detailed documentation.

Phoebe the tester

Albert Gareev’s post on the Software Testing Club’s blog about Phoebe Buffay from Friends made me smile.

I remember years ago, watching Friends in rapt admiration. Ross the paleontologist took Phoebe’s reluctance to believe in evolution as a personal affront.

Phoebe, the scatty new-ager, dismantled Ross’s certainty in a masterful little cross-examination.

YouTube doesn’t allow embedding of this clip, so here’s the link. The key bit is three and a half minutes in, but the lead up is fun anyway.

Frankly, Ross caved in far too easily, but that was largely the point as far as I was concerned. It wasn’t that he was wrong. The problem was that he was so convinced he was right that he was completely thrown by an astute rhetorical ploy.

He wasn’t used to being challenged, and couldn’t handle it. He was tested and failed.

Phoebe may not have cut it as a conventional corporate software tester, but she did have some qualities that are priceless for testers.

She had a distinct perspective. She looked at the same things that others saw, but saw them in a different way.

She wasn’t afraid to be out of step. She wasn’t worried about ridicule and didn’t care if people thought she was wrong. She was as far from being a “yes person” as you could find.

Above all she challenged conventional wisdom and forced people to question their assumptions. She enjoyed doing so, and managed to do it without truly upsetting people.

If the example had involved Phoebe being unambiguously right, whilst Ross was totally wrong, what would there be to learn from it?

I think most of us would agree that if the project is doing something wrong we should question it. The trouble is that challenging other people, especially senior people, or challenging requirements, can seem ill-mannered or dangerous. So it’s all to easy to persuade ourselves that they’re probably right. We just drift along and don’t ask questions.

Sceptical thinking and challenging questions are worthwhile regardless of whether we know we’re right or wrong. We often don’t know – not for sure. It’s important we don’t act as if we did know, as if certainty were available.

We have to ask the difficult questions to try and help everyone gain a better understanding of what’s going on. Even if our stance as devil’s advocate is wrong, or unsound, the process of questioning should have helped clarify everyone’s thinking.

Ross’s conviction that he was right hadn’t been tested. As soon as Phoebe swung into tester mode Ross crashed with a severity 1 failure. Maybe he needed to sharpen up his thinking and his debating skills? Ross the Debater 0.2 would surely have done far better after failing that test.

When I originally watched this episode of Friends I had recently switched to testing after spending a few years in computer audit. I could empathise with Phoebe’s willingness to challenge, and stir things up. I loved her sense of mischief as she did so A sense of mischief? I’ve never seen that in a job ad for a tester, but it’s not a bad quality.

Come to think of it, I remember a programme manager at IBM calling me an iconoclast. As a Christian of the reformed tradition I was happy to take that as a compliment!

Please! Don’t get over-analytical about the story of Phoebe the tester. It’s not an argument about evolution, or religion or science. As Phoebe said, “that was fun – so who’s hungry?”.