Best practice, or just okay practice?

The testing community has been conducting a lively debate on Twitter over the last week about the merits of “best practice”. Rex Black kicked it off with this provocative tweet.
RexBlacktweet

Not surprisingly Rex got a vigorous response and soon the debate was raging. Many people provided valid objections to the idea that there are such things as best practices, but Rex had no intention of backing down.

In response to criticism that “best practice” is just a marketing term Rex wrote;

It’s not a marketing term. “Best practices” is a widely used management term. By refusing to use this term in its common meaning, you’re just removing yourself from mainstream discourse.

Yes, I’d agree with the first part of that, but not in a way that would make Rex feel comfortable because the reasons for my agreement undermine Rex’s position. It is a weaselly and damaging management term, however popular it might be. The second part, about dissenters “removing themselves from mainstream” is a non-sequitur. Disagreement and disengagement are very different things.

“Best practices” certainly is a widely used management term, and thoughtful testers do need to engage with it. We must not remove ourselves from the mainstream. However, by refusing to use it “in its common meaning” we are bringing much needed clarity of language and thought to the problem.

“Best practices” may be widely used but it is also bollocks. I’d like to offer just some of the reasons why.

”Best practices” hinder experts

My first objection is that the idea of “best practice” constrains practitioners. Beginners need the rules and formal structure that “best practice” and standards provide. Experts do not, and their creativity is stifled by “best practices” that are usually defined by people with less experience and skill.

I touched on this in an article I wrote about testing standards a few years ago.

I referred in that article to a talk that Lloyd Roden gave at Starwest 2009. I’ve just found this video of Lloyd’s talk on YouTube. It is under nine minutes long and well worth watching. It’s had only eight views so far and deserves many more.

”Best practices” foster mediocrity

If experts are constrained and frustrated then that leads on to my second objection to “best practice”. The term implies that certain methods and techniques cannot be improved upon and are required in all circumstances. Paradoxically “best practice” encourages mediocre conformity.

Practitioners have to conform to an industry norm and are discouraged from improving upon “best practice”. If we have already achieved “best” then how can we get better? Any deviance must be inferior. That, I’m afraid, may be utter nonsense, but it is the subtext to much of what goes on in reality.

Dilbert nails “best practices” in this classic strip, “stop making mediocrity sound bad”.

Don’t dismiss Dilbert’s view as being cynical and unfair, at odds with serious professional opinion. Read this.

I have always had difficulties with the term “best practice.” Who is to say which practice is best, which is almost as good, which is really not good enough? The members of the standards committee have been appointed (who appoints them, anyway?) to define best practice at a point in time, but as I stated previously the best sinks to “just okay” practice with the passage of time. A standard is obsolete the day it is published.

Is a standards committee empowered to describe what they believe are generally the best methods, tools and techniques as evidenced by what most organizations are doing? If so, the committee is describing standard practice that has been overtaken by best practice, however defined. The process of standardization drives the thought process to the middle. Carried forward over time, standard practice is mediocrity.

This isn’t a tract from the “context-driven/RST” camp. It is an extract from an article by Steven Ross of Risk Masters Inc entitled “Just okay practice” in ISACA Journal, vol 2, 2013. This is the official magazine of the Information Systems Audit and Control Association, of which Ross is a former president.

Unfortunately the article is available online only to members. Is it a controversial view? I doubt it. Two months after it was published the only comments are ones that agree with the writer.

Context comes first, practices come second

Although Steven Ross was talking in general about standards the article was prompted by a specific concern about the effect of standards on information security practice. So do the auditors think differently when it comes to software development?

I don’t think so. Try this.

Internal auditors should not expect organizations to fully implement PMBOK, PRINCE2, COBIT, or any other large set of best practices. Rather, they should expect to see that these practices have been customized and integrated into the organization’s project management methodology.

“Not … fully implement… best practices”, “customized”. This pragmatic statement comes from “Global Technology Audit Guide 12 – Auditing IT Applications”, an official research paper issued by the Institute of Internal Auditors to its members.

COBIT (Control Objectives for Information and Related Technologies) is the governance framework for IT created by ISACA. The latest version, COBIT 5, makes it clear that “best practices” are neither mandatory nor inflexible. They should be tailored to each organisation’s objectives and needs. Understanding the context comes first. Choosing, refining and deploying the techniques come second.

Does anyone believe “best practice” means what it says?

Clearly “best practice” doesn’t mean exactly what it says. “Best” is only good practice in a particular context. That is probably not contentious, and I doubt if supporters of “best practice” such as Rex Black would contest the point.

However, in practice many influential people would dispute the point and they do expect “best practice” to mean exactly what it says.

Whatever the defenders say, it is used loosely as a marketing term. It is used to persuade clients that the highest professional standards will be deployed. Clients can also insist on suppliers complying with “best practice” in the naïve belief that this will protect them.

During the Twitter debate Suresh Nageswaran made a very revealing comment in defending “best practice” with the analogy of checks one should perform before driving.

Before car starts, checking tire pressure, fuel,coolant, water is best practice. Cars have numerous points of failure. This practice will reduce the chances of running into them. Ergo best practice. “Best practice” reduces risk in the path to achieving a goal.

Rex Black backed up Suresh saying that he was “exactly right”.

I think he was exactly wrong. It would be best practice to perform such checks before a journey only if there is time available and the consequences of failing to reach the destination on time were sufficiently serious to justify the delay.

Given the reliability of modern cars the risk does not justify performing these checks every time. If you have to get to work on time it would be prudent to trust in the reliability of your vehicle and perform checks at leisure. It is dangerous nonsense to insist that whatever reduces risk is “best practice”.

However, if we neglect to perform the checks then we are guilty of ignoring a “best practice”. In business neglecting documented and agreed “best practice” might get the lawyers interested. So clients and suppliers agree to use “best practice”. It suits both to pretend that it raises quality and protects them.

If you use words loosely don’t be surprised when people believe you mean what you say. “Best practice” can’t be treated as innocuous management terminology when it shapes contracts and introduces damaging inflexibility to working practices.

If you mean that a particular technique is a good practice in certain contexts then say that. Don’t pretend it is a best practice. If the auditors are careful to qualify the term shouldn’t we also be cautious about how we use “best practice”? Some lawyer might just believe you actually mean it! Why on earth would we want to use a term that suggests we are negligent if we don’t follow “best practice”?

Postscript – I intended to leave the matter at that, but I was challenged in a comment by Suresh Nageswaran (see below) that “best practice” is an important form of protection for clients and users. Thinking back on my own experience I reflected that this wasn’t the case, and I decided to write a second article explaining why I’ve found the idea of “best practice” unhelpful.

Advertisements

18 thoughts on “Best practice, or just okay practice?

  1. No matter how advanced or reliable your vehicle is, if you have insufficient air in your tyres, you are opening yourself out to risk on the highway. Your car may get damaged, worser still, you may lose your life.

    If getting to the destination is *so* important, it is even more important to ensure you have enough fuel. Again, the reliability of the car will not help if you run out of gas.

    Another best practice is to observe the speed limit on the roads. Again, statistically, you may flout this limit and get away with it for the most part. But you don’t anticipate the presence of oil on the road, and your car skids.

    Best practices exist for a reason. The vast majority of drivers on the road find utility in them. Again, the operational term is majority, assuming that the average skill level is low.

    Now you may be Michael Schumacher and find this does not strictly apply to you. And that is okay. The exception makes the rule.

    Even you will be at pains to tell your teenaged son not to observe these best practices because, after all, “it’s all contextual”.

    The point you make about customers insisting that suppliers comply with best practices is correct. Again, this sentiment did not magically appear out of a vacuum. For years, customer had to deal with problematic developers and projects managed poorly with little or no standards. The best practices and standards came into being to address this shortfall of maturity and to prevent the same mistakes from being made over and again.

    I have an issue, much as you do, with excessive reliance of standards. I have worked in software organizations that have become bureaucracies that insist on paper and electronic trails for anything and everything without discrimination. They assail both developer and test engineer with reams of useless forms and documentation that nobody will ever see or read. That is an abuse of the very concept of best practices.

  2. Pingback: Five Blogs – 7 May 2013 | 5blogs

  3. Spot on. “Best practice” is good practice minus context plus hubris.

    Great quotes from the audit industry too. They’re less conservative than one might imagine.

  4. Thanks Dave.

    Hi Suresh, thanks for taking the trouble to come along and disagree with me politely.

    To be clear, I am not arguing against any particular practice. My problem is labeling them “best practices”. In fact, my strongest objection isn’t to “best practice”, but the subtly different phrase “best practices”.

    Using “best practice” to describe PMBOK or COBIT is undesirable in my opinion, but just about acceptable provided it is qualified in the way that ISACA and IIA do. If one uses the term “best practices” and then categorises individual techniques as being “best practices” that is more dangerous because the inference that any rational observer would draw is that such practices are the best to employ. It is a poor argument to claim that we don’t really mean what we say and that “best” means “usually best, but not necessarily so”.

    The example you give of checking the fuel level is a good practice. But you say it is best. I check the fuel level frequently while I’m driving and always when I get home after every journey. I then know whether I need to allow extra time to get fuel before I start my next journey. That ensures that I never (well, never so far) run out of fuel and means I won’t be late at my destination. This works well for me.

    However, if checking the fuel before I start is classed as best practice and someone comes along to check whether I comply with best practice the answer would be “no”. That has real implications in business. It implies that I have been sloppy, unprofessional and even negligent. The definition of “best practice” may reflect a limited awareness of what is possible and practical.

    I don’t want to get into the analogy of speed limits. Complying with the law and adhering to “best practice” are different matters.

    As for protecting clients, I think talk of best practice is unhelpful. You say that slavish adherence to documentation and standards is an abuse of best practice. Talking of “best practice” has the natural effect of encouraging uncritical adherence, so what you see as abuse I see as a likely consequence.

    There is a lot more I want to say on the question of whether “best practice” protects clients, but I think it requires another blog post, rather than just a comment.

  5. Let me try another analogy. The FAA checklists are an orderly and sequential collection of “best practices” for configuring an aircraft for safe flight. Checklists must often be accomplished amid a host of competing cockpit priorities–obtaining clearance, responding to calls from ATC, consulting charts, taxiing for takeoff, and communicating with the cabin crew, to name just a few.

    The FAA mandates that these be followed, but of course, does not enforce. That is left to individual airline firms.

    So my question to you is – given a choice between traveling on an airline that enforces and institutionalizes these best practices, to one that does not – which airline would you prefer to fly with?

    And if you can make that choice, why hold customers of software development firms to a different bar?

    What is important to understand is that the term ‘best practice’ is not an end in itself. And best practices must be continually improved upon. If you find a better way to do something in your context, that is a best practice as well.

    But I reject your premise that uncritical adherence is the inevitable consequence of relying on such best practices. My opinion is that this goes really to the DNA of an organization, the culture that lies beneath.

    And even such consequence may not always be bad. After all, organizations choose the cultures that work well in the line of business they are in, and what helps them produce results. If process-related ballast slows down software development to the point that time-to-market is affected, you will find the business pushing back to make that adjustment.

  6. Thank you for returning. It’s an interesting discussion.

    I would obviously prefer to fly with the airline that carried out proper checks before each flight. However, I don’t buy your follow on claim that software development should be carried out in the same way. Software development is different from flying a passenger jet for all sorts of reasons. For one, operating a proven machine is very different from trying to envisage and then design a more nebulous creature such as a possible software application.

    I agree that “best practice” must never be regarded as an end in itself, but calling it “best” encourages that mindset. If someone finds a better way then that means that there are two different, competing “bests” in the same organisation at the same time. That leads to conflict, and often leads to the innovator being squashed and told to conform. After all, the old (and poorer) practice is established as “best”. I know you think that shouldn’t happen, but I’ve seen it time and time again. If people were encouraged to think not of “best”, but “good and needing to get better” then that would help enormously.

    I think we’d probably agree on the right things to do at any given time, but where we disagree is probably in our view of human nature and the culture of organisations. I probably have a bleaker view of human nature, and I have seen how rigid and unresponsive to change, and to problems, large organisations can be.

    I once explained to a senior manager what was wrong with certain things we were doing, which were regarded as best practice. He said that he agreed with me, but I’d be surprised how far up the organisation I could go and find people who would agree with me, but say, “sorry, we can’t change”.

    I think it’s important to work with the grain of people’s instincts, which may be good or bad, and to ensure that corporate structures, processes and cultures positively encourage flexibility and willingness to improve. I think the term, and concept, “best practice is unhelpful.

  7. The pilot of a private plane does the kind of walkaround that you describe prior to getting in and starting the engine. I would imagine the potential consequences of a point of failure in an airplane justify the expenditure of a few minutes.

    Still, is that a “best practice”? I have trouble describing it as such. It’s more a trained practice that ensures nothing obvious is wrong. We can, and under some circumstances should do more.

    I think the definition is where we are getting wrapped around the axle. I see best practices as highly contextual, almost more like heuristics described within decision trees. A single statement of a certain activity under all relevant circumstances is more like a practitioner exercising trained common sense.

    • It’s an interesting point whether pre-flight checks are correctly described as “best practice”. They’re unquestionably the right thing to do, and if calling them “best practice” means they’re more likely to be performed then that’s fine.

      I remember the procedures we had to follow on the rifle range when I was in the Air Cadets at school. They were very precise, and were intended to ensure no-one got hurt and no bullets were unaccounted for. I’m sure these procedures would have been called best practice, and again I’m fine with that. When you’re dealing with teenage boys and rifles then certain practices are non-negotiable. There was only one acceptable way to do it. But none of that is relevant to the way that we develop software and we should be wary of analogies of dubious relevance. They fog the issue rather than clarify it.

  8. Wrapped around the axle, indeed. There are a number of ways this analogy fails.

    For starters, operating a vehicle (flying or rolling) is not the same thing as testing software. It’s not apples and oranges, it’s apples and playing a piano.

    A vehicle has a defined set of inputs, with outputs mechanically engineered to be predictable. The operator follows a route that has been traveled many times before, and as long as they operate the vehicle correctly and don’t collide with anything, they will be successful in reaching their goal.

    A software project is creating something new, from nothing. The people, timeline, project, architecture, implementation – they are different, every time. You start with nothing, working together with other people, sharing senses and knowledge to produce something in the end that will be flawed, but can be great.

    Since always, there have been people applying industrial quality thinking to software testing – paying attention to defect densities, output measurements, trying to apply Sick Sigma analysis techniques, and so forth. It always fails, because software is not widgets. There is no blueprint or CAD drawing to say what it should be or how to build it – so why would you treat it as a compliance with design exercise?

    Comparing piloting to software testing is no better. Software is people working together to make something out of nothing. We need to find analogies that fit that. Such as:
    – Writing a book
    – Making a movie
    – Sculpture
    – Writing a symphony

    Karen Johnson once observed that like other context-driven testers, I was very interested in definitions and placed a lot of value in understanding them. And of course, she was right.

    But in this case, my objections to Best Practice when Rex Black asked for them were not just about the definition of Best. Like with others – and as James has stated it brilliantly here – the problem is with the idea that if we follow a Best Practices recipe, we not only don’t have to do our own thinking, we shouldn’t.

    I participated in a couple of peer workshops where we identified heuristics that we use as testers. James Bach’s definition has stuck with me: “A heuristic is a fallible method of solving a problem or making a decision.” More here: http://www.satisfice.com/blog/archives/462

    Do Best Practices include instruction on when they are shouldn’t be applied? Do they acknowledge where they fail?

  9. A most interesting summary James! Did you know that, in 2008, Gartner said that Programme and Portfolio Management best practices are optimized for a world gone by (included governance).

    But, what next? Here is a suggestion:
    – Ignore the debate around best practice, method and the likes
    – Whatever it is called, use what delivers value in the specific environment
    – Avoid what creates damage;
    – Approach it from: optimum value for the clients, the employees and the enterprise as a whole

    How? Perhaps this helps: “We Selected the Simple Solution. The Complex Solution Became a Worldwide Standard”. http://bit.ly/15X9sbX

  10. “Context” has been mentioned numerous times here, but only in passing.

    The context of the particular kind of problem space you happen to be dealing with dictates the appropriate approach, per the Cynefin framework. http://en.wikipedia.org/wiki/Cynefin

    In particular, “best practices” (poor naming aside) really only apply to the “simple” context, and “good practices” to the “complicated” context.

    Anything truly interesting/hard/creative (such as my domain of software engineering) is going to reside in the “complex” or “chaotic” contexts, the approach to which are probe/act, sense, and respond, from which the practices emerge.

  11. Pingback: Neotys Testing Roundup, May 2013 Issue 2

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s