Test script production lines

I was startled yesterday by a series of Tweets from Phil Kirkham.

Shuddering at a sentence that starts ” The basic idea is that we have a Test Script production line…”
It is staffed by a newbie tester and another one new to the project who knows nothing about the app or the business.
Testers churning out test scripts from specs , handing them onto the ‘execution line’ and move onto the next spec.

Now my first reaction was no doubt the same as all right thinking, sensible folk. I shuddered, thought “there but for the grace of God…”, and remembered past projects which had got uncomfortaby close to such an abomination.

Then I thought a bit longer about this, and wondered if the “production line” heresy might offer an opportunity for testers who are wanting to stir things up, to make a difference and improve testing.

All too often companies pay lip service to testing. Everyone believes in it. Plenty don’t want to do what is really necessary to let the testers do a good job. However, the unspoken assumptions and prejudices stay stubbornly hidden under the surface. We know they’re there, but the people with influence never state their deep seated views in public, in a form where it can be questioned and challenged.

I’ve worked on projects where the speed and efficiency with which the scripts could be churned out and then executed was clearly more important than whether the testers were really learning about what the application could do, and what the level of risk would be if it were launched.

Management never said that impressive daily progress cranking out scripts was the real bottom line as far as they were concerned, but we all knew that was the reality.

If I’d suggested that the attitude of management made it clear that we really should have a production line, with operatives specialising in each successive step of the process it would have looked cheeky, verging on the insolent.

Yet the production line is the logical extension of the obsession with scripts, with the number produced and executed. Phil’s case is the “reductio ad absurdum” of the script approach. It’s taking scripting to its logical, but insane conclusion.

The people who’ve proposed it have put their head above the parapet, declared effective testing is a matter of efficient production of scripts, and maybe now they can be challenged.

Scripting is an inefficient way to test.  Detailed scripting is especially inefficient, and if different people are writing and executing the scripts then the inevitable consequence is a high level of detail. The test executors have to be told what to do. They will have minimal previous exposure to the application or new functionality. They’re too busy “specialising” in execution.

The production line is simply a way of streamlining a process that is inefficient and ineffective. It makes it clear that the process has become the “end”, not the testing.

Adopting a production line opens up the debate about what testing should be, and how testers can be deployed most effectively. There was an interesting study at the University of Helsinki (PDF – opens in new window) a few years ago in which exploratory testers achieved pretty much the same results as script testers, but with 18% of the effort.

There are all sorts of complaints one could make about the methodology and the analysis. The study didn’t even make any great play of that 18% figure. However, there should be enough in that paper to make even the most blinkered scripting advocate pause and think.

Maybe proposals like the production line give us an opportunity to say, “hang on – just what is it we’re trying to do here?”

I expanded on the subject of heavy documentation in an article on testing and standards that I wrote for Testing Experience in December 2009.

Right now my Saturday job as a football reporter calls, and I’ve got to get off to the game! (PS – we won 🙂 ).

2 thoughts on “Test script production lines

  1. Excellent post, James.

    “All too often companies pay lip service to testing. Everyone believes in it. Plenty don’t want to do what’s actually necessary to let the testers do a good job. However, the unspoken assumptions and prejudices stay stubbornly hidden under the surface. ”

    Alongside paying ‘lip service’ there is a lot of misunderstanding about what testing is and what it can do for a project. I often hear PM’s comment that “We still have to get this through testing.” or “I doubt this will get past testing.” Both comments instill in people’s minds the idea that testing is this bottleneck – a necessary evil almost – which is really unfortunate. Breaking these misconceptions seems to be a long and tedious task.

    Until we progress in helping people realise what testing is and what it brings to a project, I think we are going to continue having battles against the ‘production line’ proposals.

  2. Thanks Stephen. Some project managers don’t see their job as doing the job, but as following the process.

    If the process of managing a software development project were to say “paint the office purple as soon as the coders have delivered the last module” then they’d bust a gut to make sure the tins of paint were stacked up ready to go when the coders had finished, and they’d want the decorators working overtime to get the new paint up.

    They wouldn’t question whether a spot of interior decoration was actually helpful. It would be just something that had to be done, as quickly and efficiently as possible.

    Is it too cynical to suggest some regard testing in the same light? It’s not about testing as we would understand it. It’s just something that has to be done (yawn).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s