I wrote this article for Testing Planet a couple of months ago. Most of the feedback was positive, but I received some criticism on Twitter that I’ve been unfair on Structured Methods. I want to make it clear that I’m talking about the big, formal methodologies such as SSADM, and BIS Structured Analysis & Design, the particular variant we used where I worked. These are based on the work of Yourdon, Constantine and DeMarco.
My problem isn’t with individual techniques, many of which are valuable. I’ve always been obsessive about structured programming when I code. I was trained to regard that as simply good, responsible practice. I’ve also long been a fan of IDEF0, which I’ve personally found useful both as an auditor and a tester to clarify my understanding of what is going on, and what should be happening but maybe isn’t. IDEF0 is definitely a technique from structured methods.
So the problem isn’t with individual components. It’s with the way they are bolted together into a full blown, formal methodology. The development process is linear and rigid. Practitioners are encouraged, often mandated to follow the whole method, and nothing but the method. It is a brave project manager who deviates from the formal method knowing that if he/she delivers anything that is less than perfect there will be hard questions about failure to follow the prescribed process, which implicitly assumes that it is the only route to the Correct Solution. You then see the sort of scenario I’ve described in my article.
And here is the article
When I was a child my father often used to refer to me with an exasperated smile as “lawyer Jim”. He was amused by the way I would always argue and question, not in a disobedient or truculent way. I just wanted to know what was going on, and why things were happening the way they were. I demanded good explanations, and if the explanations didn’t make sense then the adult could expect a thorough cross-examination till we’d got to the heart of the matter.
I didn’t end up as a lawyer, but it’s not really surprising that I did become an auditor and a tester. I’ve always wanted to understand what’s happening around me. It has always struck me as a challenge if someone says “just because”, or “that’s the way we do things round here”.
When I left university I struggled for the first few years to make any sense at all of the places where I worked. I started off training to be a chartered accountant, but I hated that. It wasn’t the day to day auditing, it was the culture. I struggled to see how we were really doing anything useful, how our work added up to valuable information for the stakeholders. It all seemed a charade, and I wasn’t convinced anyone thought differently; it was just a good way to make money.
Structured Methods – something I just didn’t get
I ended up in IT, at a great place where I learned a huge amount; about business, people, organisations, IBM operating systems and utilities, how to have fun coding, all sorts. What I didn’t really learn much about was how to make the prevailing Structured Methods work.
Structured Methods were “The BIG Thing” then, in the mid 80s. They were intended to revolutionise the way that systems were developed. No longer would a bunch of geeky cowboys hack their way through the code till some ramshackle application could be dumped on the poor users. Structured Methods meant that rigorously trained, professional, software engineers would meticulously construct applications that perfectly reflected the requirements of the users.
There were a couple of problems with this, apart from the obvious one that the results were lousy. Firstly, you can’t specify requirements perfectly in advance in the way that Structured Methods assumed. The whole rickety shebang depended on some pretty ludicrous assumptions about requirements; that these could be specified precisely, accurately and in detail up front, and also that they could be largely inferred from the existing system and its flaws. It was a pretty limited view of what was possible.
The other huge problem was the attempt to make development a mechanical process. You churned out the artefacts from one stage, fed them into the next stage and so on along the production line. Fair enough, if there actually was a production line. The trouble was that that the results of the analysis weren’t really connected to the resulting design, not in any coherent and consistent way. The design depended on old-school, unfashionable things like skill, experience and judgement.
Developers would plod agonisingly through the process, generating a skip-load of incomprehensible documentation that no-one would ever read again, then at the crucial point of turning requirements into a physical design, they’d have to wing it.
Anyway, I didn’t get Structured Methods, and managed to arrange my career so I could go my own sweet way and keep enjoying myself.
I knew that Structured Methods didn’t work. That was pretty obvious. What I didn’t realise was that that was irrelevant. People weren’t using them because they worked, or even truly believed deep down that they worked. So what was going on?
Let’s wind back a few decades for some interesting insights. Isabel Menzies Lyth was a distinguished psychoanalyst whose particular specialism was analysing the dynamics of groups and organisations.
In 1959 she set off a bomb under the medical profession in the UK with a paper called “The functions of social systems as a defence against anxiety” (PDF, opens in a new tab). She was writing about nursing, and argued that the main factors shaping an organisation’s structure and processes were the primary task, the technology used and the social and psychological needs of the people. The dominant factor was not the task or even the technology; it was the need for managers and nurses to cope with the stress and anxiety of a tough job.
As a result “social defences” were built to help people cope, and specifically to help the managers cope. The defences identified by Menzies Lyth included rigid processes that removed discretion and the need for decision making by nurses, hierarchical staffing structures, increased specialisation, and nurses being managed as fungible (i.e readily interchangeable) units, rather than skilled professionals.
These defences solidified the culture, structure and processes in hospitals. The rigidity damaged performance of the primary task, i.e. caring for patients. This benefited the management but racked up the stress for the nurses themselves. The defences were therefore strengthened. The response was to adapt the job so that nurses became emotionally distanced from the patients and their outcomes. Patients were increasingly regarded as subjects for treatment, rather than people. Sub-standard performance had to be defined as standard. Acceptance of sub-standard performance became part of the defence mechanism.
Practices in the health professions have certainly changed over the last half century, but Menzies Lyth’s insights into how people deal with stressful jobs in organisations have an important lesson for us.
David Wastell and transtional objects
In the 1990s a British academic, David Wastell, linked Menzies Lyth’s insights with work by Donald Winnicott, a paediatrician and psychoanalyst. Winnicott’s big contribution was the idea of the transitional object (PDF, opens in a new tab).
This is something that helps infants to cope with loosening the bonds with their mother. Babies don’t distinguish between themselves and their mother. Objects like security blankets and teddy bears give them something comforting to cling onto while they come to terms with the beginnings of independence in a big, scary world.
Wastell studied development shops that used Structured Methods. He interpreted his findings in the light of the work of Menzies Lyth and Winnicott. Wastell found that the way that developers used the method, and their mindset, meant that Structured Methods had become a transitional object, i.e. a defence mechanism to alleviate the stress and anxiety of a difficult job (see “The fetish of technique, methodology as a social defence”, not free I’m afraid).
Wastell could see no evidence from his own studies, or from the research literature, to suggest that Structured Methods worked. The evidence was that the resulting systems were no better than the old ones, took much longer to develop and were more expensive.
I could recognise the patterns Wastell was describing. Managers became hooked on the technique and lost sight of the true goal.
“Methodology becomes a fetish, a procedure used with pathological rigidity for its own sake, not as a means to an end. Used in this way, methodology provides a relief against anxiety; it insulates the practitioner from the risks and uncertainties of real engagement with people and problems.”
Teddy bears are certainly great for toddlers. They are cute and cuddly, and they help children as they learn about the world. However, Wastell was at pains to point out that it is deeply harmful when developers cling on to their own transitional objects. Systems development has to be a process of learning, he argued (see “Learning dysfunctions in information systems development…”). Seizing on Structured Methods as transitional objects, and obsessing about compliance with a rigid technique, prevented learning.
In chasing a warped vision of “maturity” the proponents and users of Structured Methods were perversely refusing to grow into true maturity. Practitioners became trapped, and never grew beyond the teddy bear stage.
Like the nurses and managers in Menzies Lyth’s study, developers were defining sub-standard levels of performance as the professional standard. The rigid techniques were not really helping the nurses who treating patients. The managerial processes did help the managers to deal with stress and anxiety, but they made the job harder for the nurses to cope with, which in turn led to nurses concentrating on process rather than patients.
Likewise in systems development techniques that help the managers to cope with stress and anxiety, and which give them an illusory, reassuring sense of control, are harmful to the team members. They have to cope by focussing on the technique, mastery of the tool, or compliance with the standard. In doing that they can feel that they are doing a good job – so long as they don’t have to think about whether they are really working towards the true ends of the organisation.
Trusting the method, or asking awkward questions?
Sadly, the more mature and thoughtful nurses in Menzies Lyth’s study were less likely to complete their training and stay in the profession. It was the less thoughtful, less emotionally mature nurses who could adapt more easily to the regime.
Perhaps that was why I was uncomfortable with Structured Methods, and with the idea of standards and rigid processes. My parents always encouraged my awkward questioning. It was important to question, argue and think. If you want to grow up you have to face the reality that worthwhile jobs are going to bring some stress and anxiety. We have to be honest enough to face up to that, rather than pretend that a cute and cuddly new standard, method or technique will conjure up an easier life. Maybe I was right in the first few years after I left university and the people I was watching really didn’t understand why they were working the way they were, or how their activities helped achieve anything worthwhile.
Tough jobs don’t become easy just because you redefine them in a way that makes them feel easier. That way you just risk detaching your activities from the true objectives you’re being paid to achieve. Sooner or later someone is going to catch on and realise that you’re getting paid for something that’s of little value. If you’re not doing anything useful there will always be someone who can do it cheaper.
High quality systems depend on good people, with the right skills and experience. Their managers have to be strong enough, and honest enough, to confront problems, and recognise psychological strains. They shouldn’t evade them. If they want something to help them cope with the stress and anxiety, buy them some teddy bears!