Once upon a time there was a project, on which I was the test manager, which prompted me to take the drastic career change I had been mulling over for a year or so.
This project was with a large UK government department. I was the test manager for the supplier who also provided the development team. A rival IT services company ran the Service Delivery function. It was a very interesting (i.e. poisonous) three way relationship!
The development was a six month project to customise an off the shelf package that was the basis for a rule based expert system. It would help users to decide if applicants were eligible on health grounds for welfare payments. The idea was that we would develop a pilot for live running in one particular part of the organisation.
The crucial constraint was that the development had to be fast so that all parties could understand the base product better and learn the right lessons for the customisation required for the rest of the organisation. However, it would be a live application that would shape decisions which would have a huge impact on the lives of poor and vulnerable people. It was a serious matter, not just a throwaway prototype. There was also a strong emphasis on accessibility. If any corners were cut elsewhere there was no question of skimping on accessibility issues.
The stress on accessibility was a key insight into later problems. There were about 100 potential users for the pilot, which was expected to run for only a couple of years till it would be replaced by a more sophisticated version as part of the full programme. So I asked what accessibility needs the current staff had, so I could factor that into my risk assessment. Obviously, or so I thought, the needs of those staff should have the highest priority in testing.
I was bluntly told by the client that I was not allowed to ask that question. We had to assume that any type of user with special needs might be recruited within the lifetime of the application. That was fair enough, but it didn’t sit easily with the expectation that the development would be as fast as possible. It was clear that it would be better to deliver late, or even to fail altogether, rather than to produce an application with accessibility problems. The reason? This government department was responsible for ensuring that other organisations complied with accessibility legislation, so the client was insistent that everything must be done by the book.
I gradually became aware that this was a wider problem than concern about accessibility. At that time there had been a string of embarrassing stories in the British magazine Private Eye about government IT failures. These were well researched and highly critical articles. They were referred to occasionally and it dawned on me that the client was terrified of being in a future article.
When I say they were terrified I mean their fear dictated their whole approach to the job. Working in the private sector I had been used to seeing people take risks, doing what was necessary to get the project in. Sometimes these risks were reckless, sometimes they were shrewdly calculated. But I had never seen a client who was too terrified to take any reasonable risks.
The bottom line wasn’t that the pilot had to be on time, or to work, or to be within budget. The bottom line was that the client managers had to be bullet proof if things went wrong. They were so concerned about protecting themselves that they were refusing to create the conditions for success. They were almost guaranteeing failure, but a failure for which they would not be blamed.
The response of client management was to retain all of the normal processes and project governance. These were specified in the contract. Testing was required to produce the full set of horribly obsolete, IEEE 829 style documentation. Payment to the supplier was staged with each chunk of money being released as each document passed a quality gate.
The only concession to speed was that the project team was expected to do everything faster than normal.
I expended a huge amount of effort, working long hours, writing the test plans. One of my great assets to my employer was that I produced great, high quality, impressive looking documentation.
In this case the documents were produced on shifting sands because neither I nor the development team had an adequate understanding of either the base product or what the tailored version would look like. We were able to acquire that understanding later, once our documents had been written and approved. We were always working half-blind, catching up later with our understanding.
However, everyone was happy. At least everyone at a senior level was happy as my beautiful documents sailed through the quality gates. The client paid out the money, and we had a celebratory night out every time we got another payment.
Meanwhile the testing team were miserable. They were stressed as they worked hard trying to piece together their preparations using unhelpful and barely relevant plans. The situation was rescued by my fantastic deputy test manager. I was the better test manager, in a skilled corporate operator sense. She was extremely experienced and capable, and she was the better hands on test manager. We were happy to admit this, with wry smiles, to each other, and we made a good team in those difficult circumstances. While I was working hard to produce useless test plans that would nevertheless pay our salaries, she drew up a set of informal test plans that the testers could work from.
When it came to test execution we winged it frantically. The official plans were shelfware, and we had to constantly adapt and improvise using the informal plans my deputy had prepared. We could have done it so much better and less stressfully if we had been able to plan and prepare on a realistic basis for the specific problems of this application.
What were the main differences between the two sets of plans?
The official plans did not deal with risk. The client refused to acknowledge in writing the real risk that motivated them, the fear of bad publicity. They insisted, infuriatingly, that everything had to be fully tested. They would not prioritise the testing so we could adapt the plan and concentrate on the crucial, high priority testing if we started to run out of time. They would only sign off a test plan that pretended it would be possible to do everything in time. So our informal test plans had to allow for the inevitable, and assign priorities based on our growing knowledge of the client and the system.
The client also refused to accept that using a new package would inevitably make it hard to predict how long it would take to tailor and develop applications. We were not allowed any flexibility in the schedule. Yet we were obliged to follow the full formal process, until such point as they would concede that we could skip or adapt steps. They would never do so until it was too late, by which time we had already acted unilaterally.
A particular annoyance was the insistence that the detailed plans should identify the type of people required to develop and test the system at each stage of development, and plan for recruiting this set of perfect people. There was a basic team in place, with the skills and experience to do the job, more or less. The schedule meant we had to do the best we could with the people available. That was fine, but I had to waste ridiculous amounts of time arguing about whether they were the right people, or whether we should look for different people. There wasn’t the budget or the time to go looking for a different team, but the plans had to be drawn up on the basis that we would try to develop and test with a perfect team – if we wanted to get the plans signed off and to be paid.
Another source of friction was that the client had completely ignored the non-functional requirements, with the result that the supplier architects had drawn up a requirements specification that was no more than vague mush; a set of motherhood statements and untestable criteria that were useful for a service level agreement, but not testing.
We had to re-open the user workshops to address the non-functional requirements, so we could get some meaningful input from the users as a basis for our testing. This caused real trouble, because it was a direct, explicit statement by the test team that the official plans and process were inadequate, even though they had been signed off by the supplier and client. My depute and I did not budge on this point and we were insistent. It was clear that our stance was considered insulting to the professionalism of the client. I had already decided I was going to resign soon, so I was happy to take all the blame, and unpopularity, for rocking the boat.
What were the lessons learned by the client and supplier? Well, there were no useful lessons taken on board. We were only slightly late, the quality was acceptable given that it was a pilot, there were no serious accessibility issues and we didn’t miss our budget by too much. The fact that the project team, and especially the testers, hated the experience wasn’t relevant. Nor was it acknowledged that the only reason we achieved anything was because we didn’t follow the standard driven, officially documented plan.
That was a crucial insight into what happened. There were two projects in effect. There was the official one, a beautifully planned and documented project, and a messy, chaotic project. There was little relation between the two. The messy project was the one that mattered. It delivered. The pristine document driven project was the only one that was visible and judged, but it did nothing, except bring in the money.
I thought it was a ludicrous way to work, but it was clearly the way ahead at that client. It wasn’t that people were working like this because they thought it was better than Agile, which would have been well suited to that project, if there had been a willingness to do it properly. There was no conscious rejection of Agile. Client management seemed to think vaguely that Agile meant doing the same things, but magically somehow doing them faster. In that organisation people were following prescriptive processes and standards because it gave them a sense of protection when they failed. Perversely this way of working created stress and misery for most of the people on the project. Protection? I don’t think so.
Anyway, I had had enough of this style of working, so I handed in my notice and left. The full IT development programme at that client was scheduled to run for another three years. About a year later it was cancelled.