Archive

Monthly Archives: August 2012

Why Does Agile Work?

From the response to my recent post, Barking Up The Wrong Tree, it seems like I did a good job of “burying the lede“. So here’s the key thought:

We don’t know why Agile works.

Oh, there’s a whole bunch of theories, beliefs, assumptions and what have you. But no definitive evidence as to the key ingredients, and how to combine them. Like a Christmas Pudding, which ingredients do we need for a good pudding? Which can we safely leave out? Which ingredients simply add to the flavour or texture, and which constitute the essence of a Christmas Pudding? How does the mixing and cooking affect the result? How should it be brought to the table and presented? What ingredients does a software development team or organisation need to be effective, whilst still calling what they do “Agile”?

And most importantly of all, can we have a nice dessert without having a Christmas Pudding? Would Plum Duff or Mince Pies serve us just as well?

It seems like a lot of folks – including consultants, coaches, etc., seem unconcerned over this. “Given that the developers are happier, I don’t need to know any more” seems to be the general position of some. I gasp in astonishment at this. Maybe it’s just me. I was always taking things apart to find out how they worked when I was a child (but see the section “Complexity”, below). And I subscribe to William Kingdon Clifford’s arguments about the ethics of belief, too.

Let’s set aside the issue of why Agile doesn’t work, for another time (more Agile adoptions fail to deliver the expected benefits than succeed, by a ratio of at least 3 to 1).

Instead, let me make the case for why knowing matters:

Why We Need to Know

“He who loves practice without theory is like the sailor who boards ship without a rudder and compass and never knows where he may cast.”

~ Leonardo da Vinci

If we have no understanding of why our practice(s) work, we are like Leonardo’s sailor, rudderless and at the mercy of fickle fortune. With even a modicum of theory, however, we can at least begin to make choices about where we’re going. Actually, the thought occurs to me that maybe some Agile consultants and the like don’t worry because they have no destination (future state) in mind. Maybe, for them, just being “at sea in the Agile boat” is all they need. Especially if there’s wonga in it. Sounds like being “all at sea” to me.

And maybe some know the waters in which they sail, and the regular tides and the winds, well enough to arrive where they intend. All very well for the knowledgable, in familiar waters.

Do you need to know?

Complexity

Of course, a development team or organisation is a social system, not a machine. So we’re in the realm of complexity, and complex adaptive systems. This means that we can’t simply take the system apart, to examine how it works by considering its individual constituents. That would be Analytic thinking, in any case.

So I don’t propose it’s easy to understand why Agile works (when it works, if it works), or even that it’s possible, in any absolute, mechanistic, definitive sense. Nor do I claim to understand it myself.

But I do claim to understand some of the attractors and barriers involved, and so, some of the ingredients of our Christmas dessert. And in my experience, there’s only a limited overlap between those and the ingredients of the Agile Christmas Pudding.

We all know the proof of the pudding is in the eating, but what proficient chef would not want to know the likely ingredients involved?

– Bob

Further Reading

The Ethics of Belief (1877) ~ William Kingdon Clifford

An Open Letter to the Project Management Community

Occasioned by this discussion on LinkedIn, I wanted to write to Project Managers everywhere…

Dear All

Dear Project Managers everywhere,

I hear you have mixed views about the recent, er, “developments”, in the field of Software Development, commonly referred-to as “Agile Software Development practices”. I won’t call them “advances” as we may not be able to agree that they have, in fact, advanced anything. Incidentally, I share some of your likely skepticism on that front.

I am writing to you today to share some opinions and observations about the changes in train in the software development field, globally. Whilst patchy in their uptake, changes are afoot. I can relate to your professional concerns that we retain the best of what we have learned from decades of successful project management (this also, we have to admit, being very patchy, too).

Many who look to advance the field of software development also have concerns. Concerns that some of the received wisdom of project management professionals has been rendered redundant or even dysfunctional by recent advances in fields such as psychology, neuroscience, sociology and evidence-based management.

These bilateral concerns have lead to understandable, yet vexing, tensions and misunderstandings between the various communities. Nowhere have these been more evident, perhaps, than between ‘traditional’ project managers and the Agile crowd.

I find it helpful to characterise this conflict as a clash of world-views. In a nutshell, a clash between what McGregor has called “Theory X” and “Theory Y”.

I hope I’m right in thinking that we all share a common objective – a desire to see better outcomes for our customers, delivered within timescales and at a cost that delights everyone involved. Oh, and maybe improving effectiveness of the organisations within which we work, too.

Whilst it may appear the arguments and contentions arise from our different ways and means for achieving this objective, I’d like to suggest that the conflict – as a product of conflicting world-views – is more deep-seated, and all the more pernicious for that. We can hardly expect folks, of any persuasion, to change their world-views overnight, if at all. Nor blame them for that aspect of their humanity.

And given the fundamental differences between these world-views, it seems overly optimistic to expect these world-views even to coexist peacefully and productively.

All we might hope for is a little more understanding, a little less fractiousness, and a future where we can all at least agree to disagree.

More optimistically, we might also realise that everyone has much to learn – and unlearn – from each other. That, perhaps, is something we can all work on together.

Thanks for listening,

– Bob

Further Reading

Power And Love ~ Adam Kahane
Power and Love – RSA video

Barking Up the Wrong Tree

Blind Faith

What if we’re wrong? What if we’re all unwitting participants (and victims) in a mass delusion of biblical proportions?

“Not creating delusions is enlightenment.”

~ Bodhidharma

What if the past thirty years of so-called progress in the field of software development has all been one vast waste of time?

What if we’ve fooled ourselves by one huge placebo effect? Or by a combination of placebo effect and other similar pernicious delusions and cognitive biases?

“It is only when we forget all our learning that we begin to know.”

~ Henry David Thoreau

What if what we think we’ve learned turns out to have no validity at all?

Scrum, Waterfall, Agile, Kanban, Xp, etc.. “Process” itself. Could these all in fact be no more than the most egregious of red herrings?

What if it’s really some other factor – or factors in combination – that accounts for some, or indeed all, of the differences we observe from improvement initiatives? Honestly, I don’t think we can discount this possibility. Personally, I am coming round ever more to this belief.

Let’s take a look at some of the pernicious delusions and cognitive biases that may be at play here:

The Hawthorne Effect

The central idea behind the Hawthorne Effect is that changes in participants’ behavior during the course of a study may be “related only to the special social situation and social treatment they receive”.

The Feedback Effect

Improving folks’ performance by improving e.g. their skills may be a consequence of their receiving feedback on their performance (and not as a consequence of any improvement in skills per se). An “agile adoption” may give folks feedback for the first time in their working lives.

The Observer-Expectancy Effect

The observer-expectancy effect (also called the experimenter-expectancy effect, expectancy bias, observer effect, or experimenter effect) is a form of reactivity in which a researcher’s cognitive bias causes them to unconsciously influence the participants of an experiment.

“Any of a number of subtle cues or signals from an experimenter can affect the performance or response of subjects in the experiment.”

Sounds pretty much like agile coaching or scrum mastering, just about everywhere? Of course, the role of a coach or Scrum Master is indeed to affect their team(s) in such ways (at least, for the better).

The John Henry Effect

The John Henry effect is an experimental bias introduced into social experiments by reactive behavior by the control group (i.e. a group of people, not the subject of the experiment, used as a “control” against which progress in the subject group can be compared.)

As applied to organisations adopting agile, this effect may account, at least in part, for the improvement (if an) in teams, and other departments, not immediately part of the agile adoption (a.k.a. pilot).

The Pygmalion Effect

The Pygmalion effect, or Rosenthal effect, refers to the phenomenon in which the greater the expectation placed upon a group of people, the better they perform.

In agile adoptions, managers typically place a great deal of expectation on the first agile team(s). According to this effect, these teams may improve simply as a consequence of those expectations (and not, for example, as a consequence of any changes to the way the work works).

The Placebo Effect

The placebo effect refers to the phenomenon in which people receiving a fake or otherwise intentionally ineffective treatment improve to more or less the same extent as people receiving a real, intentionally effective treatment.

“Placebos have been shown to work in about thirty percent of patients. Some researchers believe that placebos simply evoke a psychological response. That the act of taking them gives you an improved sense of well-being. However, recent research indicates that placebos may also bring about a physical response.”

The Subject-Expectancy Effect

The subject-expectancy effect is a form of reactivity that occurs when someone, e.g. a research subject, expects a given result and therefore unconsciously affects the outcome, or reports that expected result.

When people already know what the result of a particular “improvement” is supposed to be, they might unconsciously change their reaction to bring about that result, or simply report that result as the outcome – even if it wasn’t. Some researchers believe that people who experience the placebo effect have become classically conditioned to expect improvement from a change. Remember Dr. Ivan Pavlov and the dog that salivated when it heard a bell? In the case of people and placebos, the stimulus is e.g. the “ceremonies” of the new development method, and the response is real (or perceived) improvement and feelings of well-being and positivity.

“The expectation of pain relief causes the brain’s pain relief system to activate.”

The Novelty Effect

The novelty effect, in the context of human performance, is the tendency for performance to initially improve when a new approach to work is instituted – not because of any actual improvement in learning or achievement, but in response to increased interest in e.g. the new approach.

Self-determination Theory

Self-determination theory is concerned with the motivation behind the choices that people make, absent any external influences. The theory focuses on the degree to which an individual’s behavior is self-motivated and self-determined. Key studies that led to emergence of this theory include research on intrinsic motivation.

In effective Agile adoptions, for example, increased self-determination (self-managed teams and the like) may be a causal factor in increased motivation, and thus in increases in e.g. productivity, quality, or what have you. Note here I’m saying that the benefits accruing (if any) are not the result of any material changes in the process (the way the work works), but in the social, motivational context for the work.

Summary

Just as in the Hawthorne experiments, we who (merely) observe are part of the system too. Objectivity is delusional. How much else of what we induce and convince ourselves to believe, is delusional too? And how would we know? As part of the “system”, could we ever know?

The Hawthorne experiments – contention over their validity and interpretation notwithstanding –  stand as a warning about us viewing even simple experiments on human participants as if the people are only mechanical systems.

“If history repeats itself, and the unexpected always happens, how incapable must Man be of learning from experience?”

~ George Bernard Shaw

Given all the research into how our brains work (and more often, fail to work), should we not be at least open to the possibility that the results we think we have achieved in the world of software development have little or nothing to do with the things we think are important?

What do you think?

– Bob

Further Reading

The Nocebo Effect – A Contributory Factor in Failed Agile Adoptions?

Grassed Up

Software folks always seem to be on the lookout for analogies to explain software development to interested lay colleagues, friends, family and the like. Here’s one I haven’t come across before (with photos!).

Grass-cutting

Software development is like… mowing your lawn. Let me explain…

Firstly, there’s getting started. We need some developers (“the team”, analog: the lawn-mower, in this case a venerable Murray 125/96 ride-on tractor mower). The task is to cut all the grass on the lawn.

What the customer wants is a nice, neat, stripey lawn:

The team is going to deliver on this requirement by converting, inch by inch (a.k.a. continuous delivery), un-mown lawn into mown lawn. Let’s take a look at what’s involved.

The team has three basic controls; height of cut, speed over the ground, and direction.

Height of cut. This governs the height of the rotating blades carried under the mower, which in turn controls the amount (length) of each stalk of grass removed as the mower passes.

Speed over the ground. This is the rate at which new work (uncut grass) enters the team.

Direction. This allows us to select which work (area of grass) to work on next. Our choice is somewhat limited owing to the customer wanting nice, straight stripes. Sometimes (e.g. at the end of a stripe) we have to raise the blades and NOT cut some  grass (i.e. run the team without doing any real “work”) in order to best position the team (mower) for the next stripe.

Grass in Progress

The amount of grass cut, but not yet ejected from the mower, is the “grass-in-progress”.  The mower pictured is not very tolerant of high levels of grass-in-progress. Its motor is relatively small, and setting the height of cut too low (cutting too much of each grass stem in one pass) or the speed over the ground too high (increasing the rate of entry of new grass into the mower) will increase the level of grass-in-progress to the point where the motor will stall out. Listening to the motor’s note, it will complain by sounding increasingly laboured just before reaching the point of stall. Anyone who has driven a vehicle with a petrol engine will recognise this situation instinctively.

Further, damp or wet grass, and long grass, increases the friction of the grass-in-progress.

So, to mow happily at top speed, we need, primarily, to limit the grass-in-progress at an optimum level, through judicious and continual adjustment of the speed over the ground and the height of cut (and taking into account the condition of the grass, too).

One serious constraint on throughput (how quickly we can mow the lawn) is the little bastard seen in the next picture. This is the grass exit nozzle. Cunningly (mis) designed so as to clog up with cuttings anytime we try to have too much grass-in-progress. But it has one redeeming feature, too. If we notice that cuttings have stopped exiting the nozzle, the operator can reach down and lift up the plastic chute, allowing a clod of cuttings to vomit directly out of the blade housing, a bit like a hairball.

Of course, life, and this team (mower), is not perfect. In particular, corner-cases and edge-cases are problematic. The limited manoeuvrability (skill set) of the team, and the awkward shape of some parts of the lawn and its boundaries, means that some areas of the lawn don’t get cut (some corners of the requirements are left undelivered). A conscientious operator (product owner) might choose to use another, more specialised team (like a strimmer) to finish up these little pieces more neatly.

And the operator is at risk of injury too, not from the team –  which has numerous safety features – but from things in the wider environment, not directly part of the work (lawn), but close enough nearby to intrude. Here we see some pretty berries, but what we cannot see are the vicious thorns on the branches carrying the fruits.

Such risks suggest some combination of:

  • a risk-mitigating tour around the lawn prior to starting to mow, with a pair of secateurs to remove the risks
  • a not-fully-mown lawn (avoidance of the danger areas, requirements partially unmet), or
  • some inevitable damage to the operator

Fuelling the Team

No team runs on air, rather it needs fuel in order to run, and do the work. Here’s the fuel container (jerry-can) for this team. This team runs on petrol, but a software team runs on knowledge – both knowledge it already has in its tank, and new knowledge added to the tank (through e.g. learning, recruitment, etc.), as the work requires. (and no, not on wages, nor even beer & pizzas).

Here’s the fuel funnel for this team. In software development, it would carry learning into the team, rather than two star. I almost named this the “HR and training department/funnel”.

Rework,  Refactoring and Technical Debt

The thing that started me off on this analogy was the topic of refactoring. As the mower (team) works along one stripe of the lawn, it deposits a thin layer of cuttings on the adjacent stripe (no cuttings catchment here) – sometimes an already-cut stripe, sometimes a yet-to-cut stripe. When deposited thinly, they present no real problem if cutting them again on the pass down the next stripe. But ingesting the hairballs I mentioned above, suddenly and unpredictably increase the grass-in-progress for the team (technical debt). This generally requires operator intervention (lifting the nozzle), and sometimes can stall out the machine. An alternative intervention is to anticipate the problem, and reduce the grass-in-progress (by raising the cutting height and/or slowing the team) in advance of the hairball – i.e. refactoring.

As for rework, this simply means cutting the same area of grass more than once. Necessary either when we’ve ‘missed a bit’ (failed to meet quality standards) or when we’re repositioning the team and can’t be bothered to raise the blades. (Analogy: Type II and type I muda, respectively).

Team Maintenance, Breakdowns and Reliability

This poor mower rarely sees any maintenance. Hence it’s always a gamble whether it’ll start when it’s needed. And the jockey wheel/tensioner for the drive belt powering the gearbox and wheels has definitely seen better days. Often, at the end of a day’s cutting, the belt will jump the idler, requiring some faffing around to set it up again next time the mower is used.

Recap

For the literal-minded, here’s the analogous terms, laid out for you:

  • Mower: Software Development Team
  • Operator: Product Owner
  • Lawn: Work
  • Thorns: Risks
  • Petrol: Know-how
  • Stripes: User stories
  • Grass in/under the mower: Work in progress
  • Jerry-can: Sources of new know-how
  • Strimmer: alternate “finishing” team or specialist(s)
  • Cut grass: Code (and other artefacts)
  • Mossy patches: Hmm, what would you say?

– Bob

Balkanisation

Balkanisation. The dis-integration, fragmentation and breakdown of cooperation within e.g. organisations.

I’ve previously both written, and spoken publicly, about alienation in the workplace. Given alienation’s deleterious impact on the effectiveness of organisations everywhere, I don’t see it ceasing to be an issue in my lifetime.

“Modern science is characterized by its ever-increasing specialization, necessitated by the enormous amount of data, the complexity of techniques and of theoretical structures within every field. Thus science is split into innumerable disciplines continually generating new subdisciplines. In consequence, the physicist, the biologist, the psychologist and the social scientist are, so to speak, encapusulated in their private universes, and it is difficult to get word from one cocoon to the other…”

~ Ludwig von Bertalanffy, General Systems Theory

For those who have experienced the Agile way of being, the benefits of collaboration, co-location, integration of specialists from different domains and different parts of the business are manifest.

The Analytic mindset, founded as it is on the idea of breaking an organisation down into parts, and attempting to manage the parts separately, offers fertile conditions for balkanisation – and the various forms of alienation arising from it.

The Synergistic mindset, by way of contrast, with its emphasis on the whole, significantly reduces the scope for balkanisation, and thus the impact of at least one source of alienation in the organisation.

Why then do we see so few organisations where these issues are even recognised, let alone addressed?

Attempting to address symptoms of organisational dysfunction, like lack of innovation, poor engagement and motivation of staff, or high staff turnover, without understanding the root conditions, seems to me to be a fool’s errand.

– Bob

Further Reading

The Concept of Alienation in Modern Sociology ~ Igor S. Kon
General Systems Theory ~ Ludwig von Bertalanffy
Ackoff’s Fables ~ Russell L Ackoff

Dare to Disagree

Although in the same general vein as “On the Morality of Dissent“, a post I wrote some months ago, this TED video featuring Margaret Heffernan (a five time CEO) actually makes a great argument for Thinking Differently (in case you hadn’t noticed, the title and theme of this blog).

“We have to seek out people with different backgrounds, different disciplines, different ways of thinking and different experiences – and find ways to engage with them… that’s a kind of love.”

~ Margaret Heffernan

If it took 25 years for doctors to stop killing children (with X-rays), what chance do we have to bring significant change to the way organisations work – in our lifetimes?

“How do organisations think? Well, for the most part, they don’t. And that isn’t because they don’t want to. It’s really because they can’t… And they can’t  because the people inside of them are too afraid of conflict… so they can’t think together.”

~ Margaret Heffernan

– Bob

Further Reading

The Five Dysfunctions of a Team ~ Patrick Lencioni

Zen and the Art of Organisational Enlightenment

The Enlightened Organisation

I think it’s fairly safe to say that most organisations lack enlightenment. That is, few indeed are those that perceive their true nature, are self-aware (in the sense of consciousness of their own thought processes, motivations and behaviours) and can act on that perception for positive change. I think it fair to say that most organisations exist in a perpetual state of dukkha.

Does this matter? Does it, for example, impact the bottom line? I’d say yes on both counts (yes it matters, and yes, it impacts the bottom line).

The Buddha described it thus:

Insight into the Four Noble Truths we call “awakening”. This awakening allows us to attain the unattained supreme security from bondage.

Ok, enough of Eastern mysticism already. (BTW There’s a similar Western tradition, Transcendentalism, exemplified by Emerson, Thoreau, et al.).

How does this all relate to “the enlightened organisation”?

Self Awareness

If a person lacks awareness of themselves, of their own thinking, of their way of being in the world, then:

“The more asleep we are, the more out of touch we are with what we are doing, the more unaware we are likely to be of consequences, and the more unaware we will be regarding how what we are doing is affecting us and others – so the fewer opportunities we will have to recognize how often we create our own problems…”

~ Milton Dawes

Or from the Mahout and the Elephant perspective: the reflective, self-aware part of our brain governs our ability to overcome the strong psychological hurdles to our understanding ourselves – and why we do things.

In the context of the collective organisational psyche, I suggest that self-awareness (the organisation’s collective awareness of and sense of self) poses the same kinds of challenges and offers the same kinds of benefits if achieved – but at the organisational level.

By What Method

If the goal is a healthy, self-aware organisation, then how can we set about making this happen?

“A goal without a method is nonsense.”

~ W.E. Deming

Personally, I’d suggest taking a look at how individuals go about transforming their outlook and self-awareness. Effective techniques I myself have used include:

  • Meditation
  • Zen and zazen
  • Coaching
  • Therapy (i.e. help from experienced helpers)
  • Dialogue on the subject (i.e. with others)
  • Reading and study (including much study of Koans and the ineffable Tao)

For me, the organisations I see are much like those individuals trapped in a cycle of self-ignorance – unwitting prisoners of their own psyche.

In Conclusion

A thunderclap under the clear blue sky
All beings on earth open their eyes
Everything under heaven bows together
Mount Sumeru leaps up and dances.

~ Wumen

– Bob

Further Reading

Satori – Japanese term for “Enlightenment”
Samadhi – Buddhist term for “mindfulness” or “no mind” (Japanese; mushin), Flow
The Wedge of Consciousness ~ Online article by Milton Dawes
Knowing Why Beats Knowing How ~ Whitney Hess (blog post)
Innovation and the Art of Riding an Elephant – Online article by Bengt Järrehult
Self-awareness is Vital to Self-improvement ~ Blog Post at PsychologyToday
The Polar Opposite of Self-Awareness: Image Management ~ Steve Beckow (blog post)

%d bloggers like this: