I was startled yesterday by a series of Tweets from Phil Kirkham.
Shuddering at a sentence that starts ” The basic idea is that we have a Test Script production line…”
It is staffed by a newbie tester and another one new to the project who knows nothing about the app or the business.
Testers churning out test scripts from specs , handing them onto the ‘execution line’ and move onto the next spec.
Now my first reaction was no doubt the same as all right thinking, sensible folk. I shuddered, thought “there but for the grace of God…”, and remembered past projects which had got uncomfortaby close to such an abomination.
Then I thought a bit longer about this, and wondered if the “production line” heresy might offer an opportunity for testers who are wanting to stir things up, to make a difference and improve testing.
All too often companies pay lip service to testing. Everyone believes in it. Plenty don’t want to do what is really necessary to let the testers do a good job. However, the unspoken assumptions and prejudices stay stubbornly hidden under the surface. We know they’re there, but the people with influence never state their deep seated views in public, in a form where it can be questioned and challenged.
I’ve worked on projects where the speed and efficiency with which the scripts could be churned out and then executed was clearly more important than whether the testers were really learning about what the application could do, and what the level of risk would be if it were launched.
Management never said that impressive daily progress cranking out scripts was the real bottom line as far as they were concerned, but we all knew that was the reality.
If I’d suggested that the attitude of management made it clear that we really should have a production line, with operatives specialising in each successive step of the process it would have looked cheeky, verging on the insolent.
Yet the production line is the logical extension of the obsession with scripts, with the number produced and executed. Phil’s case is the “reductio ad absurdum” of the script approach. It’s taking scripting to its logical, but insane conclusion.
The people who’ve proposed it have put their head above the parapet, declared effective testing is a matter of efficient production of scripts, and maybe now they can be challenged.
Scripting is an inefficient way to test. Detailed scripting is especially inefficient, and if different people are writing and executing the scripts then the inevitable consequence is a high level of detail. The test executors have to be told what to do. They will have minimal previous exposure to the application or new functionality. They’re too busy “specialising” in execution.
The production line is simply a way of streamlining a process that is inefficient and ineffective. It makes it clear that the process has become the “end”, not the testing.
Adopting a production line opens up the debate about what testing should be, and how testers can be deployed most effectively. There was an interesting study at the University of Helsinki (PDF – opens in new window) a few years ago in which exploratory testers achieved pretty much the same results as script testers, but with 18% of the effort.
There are all sorts of complaints one could make about the methodology and the analysis. The study didn’t even make any great play of that 18% figure. However, there should be enough in that paper to make even the most blinkered scripting advocate pause and think.
Maybe proposals like the production line give us an opportunity to say, “hang on – just what is it we’re trying to do here?”
I expanded on the subject of heavy documentation in an article on testing and standards that I wrote for Testing Experience in December 2009.
Right now my Saturday job as a football reporter calls, and I’ve got to get off to the game! (PS – we won 🙂 ).