Test management – the good, the bad and the ugly

Paul Carvalho has written a very interesting blog “Test Management is Wrong”, about how we place too much emphasis on test management rather than how the whole project provides value to the stakeholders.

I do agree broadly with Paul, but I think his argument needs to be clarified and refined.

The good – letting testers do their job

Management per se is not bad. It can be done well or badly, tightly or loosely; managers can inspire or demoralise. There always needs to be some form of management to see the big picture; to coach, direct and lead testing by personal example; to report, co-ordinate and assign work; to deal with other managers and stakeholders; to protect the testers from time-wasting crap so that they can be productive. That is all good and worthwhile, and it is all test management.

We need to be clear about the distinction between managing testing and managing the testing process. A couple of years ago I had a great discussion with Fiona Charles at the Test Management Summit in London. Fiona worked it up into a marvellous talk for STPCon (PDF, opens in a new window).

The bad, part 1 – managing the process

The distinction between managing the activity and managing the process is vital and applies equally to other disciplines in software development. Testing has a particular problem. It’s a profession where it is possible to get away with blurring the distinction.

Perhaps that’s because testing, as a concept, is widely misunderstood and therefore poorly defined. When the defined process veers away from the true nature of the activity the eyes of management can remain fixed on the process, unaware of the misalignment.

Sure, the thought leaders of testing are crystal clear about what they are doing, about the distinction between process and task, but do they influence a majority of testers? I’m not sure. They certainly influence those who are listening and whose minds are open. However, there are huge regiments of testers working away in traditional environments, drawing up detailed plans and scripts; test execution becomes a matter of plodding through the scripts, progress being measured in numbers of scripts, or test cases, rather than increasing understanding of the real nature of the application.

Managers can spend all their time managing the process but doing very little that could reasonably be called testing. I don’t mean that they spend all their time managing and don’t have time to perform any testing, though that is a problem. I mean that their management activities are only loosely connected to genuine testing, i.e. providing information to stakeholders about the state of the product under test.

Managers in such companies often don’t really understand testing. They confuse the process with the reality, seeing testing as a process that must be followed, and ticked off on the project plan before the application goes live. Project managers don’t want test managers to manage testing, they want them to manage the testing process so that the project can be tracked neatly through the MS Project plan.

The bad, part 2 – faking the testing

Sadly testing is an activity which it is possible to fake, as James Bach memorably put it (PDF, opens in a new window).

James’ talk was wickedly satirical, but it is based in truth. The deeper truth, the more challenging message of James’ talk isn’t that managers consciously fake the testing. A few test managers might do that; many more do it unwittingly, seduced by the painful effort into the delusion that they must be doing something useful.

The heavyweight, documentation addicted traditional approach to testing is so tough, laborious and stressful that it really does require strong management. Also, it almost guarantees conflict, when the testing window is squeezed, when the defects disrupt the test execution schedule, when 80% of the testing window has elapsed and only 20% of the test cases have been executed. The unpleasantness adds to the illusion that something important and worthwhile must be going on. Surely all that grief, stress and sweaty effort couldn’t be pointless?

Well, actually that’s exactly what it could be. There have been times when I have felt futility and despair when I realised that I was adding nothing to the real testing even though I was the test manager. I was spending most of my time producing meaningless reports for people who didn’t really need them, didn’t understand them and certainly didn’t understand the reality. I was contorting the test execution plan so that I could produce reports showing steady progress through the plan, rather than reports that enhanced knowledge. If my reports had been scrapped all that would have been missed was the chance to tick the box “report produced”.

The ugly – battlefield testing

Whilst I was writing this I received an email asking if I was interested in this role.

lousy test management role

Crisis management? Conflict resolution? I think it’s safe to assume from the skills required that the test lead will be going into a battle zone, expected to flog the testing team along the testing leg of the project death march. How much testing, or even genuine test management, will the poor sucker be able to do? How much time will be spent battling through the bureaucracy and politics.

Test managers can run through the whole testing process entirely plausibly, without any real testing. I find it hard to imagine that happening with other IT disciplines. The folly would be immediately apparent. Yet in testing conflict resolution and crisis management are sadly important for test managers on traditional projects battling their way through the process.

It’s a lousy, unsatisfying way to work; it is inefficient and ineffective. It crushes the people working that way. That’s why we need to be clear about what test management is, and angry when managing the testing is confused with managing the process.

Paul’s follow-up

Paul Carvalho has since clarified his argument in a follow up comment on his blog. His comment confirms that we are pretty much in agreement about test management. Paul’s blog has interesting things to say about development projects providing value to stakeholders, which takes us into territory explored by Tom Gilb and Gojko Adzic.

Testing always requires strong test managers. It’s a valuable role and an important skill. What test management cannot do, especially test management that is no more than management of the process, is to provide value.

Good test management creates the conditions that allows testers to tell people about value, but only if the the whole development has been focussed on delivering value to the stakeholders. You cannot dump responsibility for value, or quality, onto the poor testers at the end. You cannot do it with a beautifully engineered Test Management Process. As Paul put it “saying that you will assure Quality through [only] Test Management is wrong.”

10 thoughts on “Test management – the good, the bad and the ugly

  1. Excellent article James, I’ve been saying some similar things in my Blog (http://itesting.com.au/) and have also discussed briefly with Paul. I think I’ll do a follow up in the next week or two to keep the momentum going on this.
    I particularly endorse you view regarding the majority of Testing endeavors being managed by “un-enlightened” TMs.

    • Thanks Colin. I’ll certainly check out your blog.

      I was carefully vague about the proportion of testers and test managers who are aware of these problems. I’d love to know more precisely. My hunch is that the great majority are “unenlightened”. These testers seldom appear on testing forums. They don’t write blogs. They don’t attend conferences. But they are working away quietly in huge numbers, stressed out but unhappily unaware that things could be different, and better.

  2. Yeh, I’ve sat in meetings where TM’s rattled off about 100% requirement coverage and 80% of tests run and 3 sev 1s but when asked if they knew what exploratory testing meant or boundary conditions or equivalence classes they fell silent. And during defect discussions they had nothing to add as they had no idea about what the product did or what the defect meant. Or why their testers were only running 6 tests a day

    But they were damned good at setting up morning and evening conference calls, lovely Powerpoint reports and lots of stats

    I’m sure there are many more who have been in projects like this – what would your advice be to someone in this situation ?

    • I was surprised I had to approve your comment Phil. I guess this is the first time you’ve commented with that email address.

      What would my advice be? Run! Run for your life!

      “It is difficult to get a man to understand something when his job depends on not understanding it.” Upton Sinclair.

      It’s a tricky one. I took my own advice, saddled up the company car and fled north for the Scottish border when i realised what I was doing was verging on the pointless.

      However, I have to admit it’s not particularly useful advice if your priority is paying the mortgage.

      All I can say is that people have got to keep arguing, talking, pointing out problems. They’ve got to do it in a good humoured, sensitive manner. If you’re an outsider then you have the liberty of bluntness. If you’re a career insider you really have to be more careful. You have to understand the people you’re trying to influence. Different people respond in different ways to different techniques.

      My personal hobby horse is the importance of getting the IT auditors on your side. Experienced IT auditors have thick skins, If they don’t then they don’t last long enough to become experienced! They are independent and if they are any good then they won’t give a damn about unreasonable pressure or ill-informed criticism. They usually occupy an independent slot in the organisational structure that should offer them real protection.

      However, many auditors aren’t up to much and are unlikely to appreciate the difference between the process and the activity. In that case the organisation probably has even more and deeper problems. I refer you to my previous suggestion. Run! Try and get another job.

      • I have too many email addresses to keep track of 😦

        I did run as I knew that even having shown there was an alternative ( better ) way I knew the next project would be the same battle and I was tired of it. Problem then becomes that the people with the good ideas leave and the crap TM’s breathe a sigh of relief that they dont have to deal with the problem of bugs that dont correlate to test cases on their next project…

      • The picture you paint is uncomfortably familiar. Test managers can enjoy better careers, or at least more profitable ones, by accepting the status quo and avoiding difficult questions. You can earn great money contracting at companies with lousy practices and a dysfunctional culture.

        Contractors do have a certain amount of freedom. They know they are not committed to the company for the long term. It’s easier for them to say things are being done wrong and decline a renewal if there’s no prospect of improvement. In the short and medium term that leaves the field clear for less thoughtful TMs. In the longer term the message might get through. That’s more an article of faith than a confident prediction I’m afraid.

  3. Way to continue the conversation, James. I like it! The fun side of me is intrigued by the ugly job posting above and the idea of having “BAT” in my title. Though I agree that it likely stands more for “battle” than “batman.” The professional in me is horrified by what’s actually written in the description. That sounds like an organisation that needs help.

    We need to continue to work to raise awareness in the knowledge work industry regarding the value that testers may provide. Everyone contributes to Quality. It’s a team sport.


    • Thanks Paul. It was a happy coincidence that the job was for a Business Acceptance Testing (BAT) lead!

      I’m pretty sure I know who the company is. There’s one in Edinburgh that is notorious for its working atmosphere.

      Yes, this is something that we all need to keep on and on about. It won’t be easy or quick, but it is so important for the future of testing.

  4. “The good – letting testers do their job”
    Problem is, when you have some lazy and bad tester in your team. Then you have to do a bit more as letting them testing. Sometimes some persons not really want to like their job, will do always only the minimum they have to.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s