The Nature of the Challenge
The Nature of the Challenge
“If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.”
~ Albert Einstein
We’ve had something like fifty years to solve the problem of reliable, effective software development. And not only have we not solved the problem, it looks like we’ve many more years ahead of us before we get to that elusive “solution”.
I’m not even sure there’s any kind of consensus on the nature of the problem, or even that we have a problem.
I’ve been studying and researching and exploring and thinking about the state of software development – à la Einstein – for the best part of thirty years. This post is about my take on the nature of the problem, as I see it. Maybe you see things differently. Either way, I invite you to share here with me and others how you see things.
Whatever the real challenge is, we seem to have a surfeit of possible solutions. Solutions which rarely get applied in the real world. In the myriad of organisations making software. Or attempting to.
So, as I see it, a key question is “why does so little of our research and new knowledge get adopted and applied?”.
We can point the finger in various directions, but I’m not looking to apportion blame.
It might be fair to ask “Who needs it? Who needs reliable, effective software development?”. It’s been my experience that precious few organisations, despite their protestations and pretensions, appear to need things to radically change for the better.
The Core Issue
There’s the rub. Mostly, people don’t seem to need things to get better. Executives, shareholders, managers, workers, customers – everyone whinges from time to time, but makes little concerted effort to actually do anything.
I’d call this a lack of motivation.
Awareness, Responsibility, Commitment
From my coaching days, I remember the A.R.C. mnemonic. This reminds us that commitment (to actually do something) is a product of people choosing to take responsibility to do something, and that this choice depends on awareness. Awareness that change is possible. Awareness that things can be better. Awareness that things are better, in some few places. And awareness that someone will have to do something before things will get better.
So, for me, I believe there is a problem. Maybe fifty years ago it was a different problem. Maybe back then, it was much more about lack of knowledge, lack of reliable technology, lack of tools, lack of importance (of software, to the world).
But now, we have the knowledge but aren’t applying it. We have reliable tech and tools, and these aren’t making much difference. And software is hugely more important to our products, businesses and societies that ever it was.
Yet a problem remains. and I believe the prime symptom of the problem is that people are unaware of the possibilities, unaware of how much better things could be, unaware of the advances in fields like psychology, sociology, group dynamics and neuroscience. And yes, unaware even of the real benefits of things like Agile and Lean – and how to realise them.
Awakening Awareness
But awareness is not the heart of the problem. If it was just a lack of awareness, then people could make themselves aware. After all, the knowledge is out there. If not on the intarwebs, then in books, periodical and the heads and hands of the (few) people who have done this stuff.
What makes for more awareness? Curiosity? What factors influence whether someone will sit up and wonder about their problems – and seek solutions to them?
I’d say motivation. Motivation to become curious. And then to pursue that curiosity.
Dan Pink suggests (intrinsic) motivation depends on three factors: autonomy, mastery and purpose. In this context, though, I subscribe to Marshall Rosenberg’s insight: motivation (to action) stems from people having (unmet) needs.
So here’s my bottom line: Reliable, effective software development won’t become widespread, won’t become the norm, until people need that to happen. And in most organisations today, I just don’t see that need manifest. Or even discussed. You?
– Bob
Think Different 
If we started delivering stuff that worked then the big consultancies would no longer have a market for their failure. I mean in terms of Seddon’s Failure Demand – I think it’s what keeps the market for this consultancy running and lots of these products with millions of lines of code in them alive.
I’ve been reading his latest book “The Whitehall Effect” – he reports of being in meetings and being told that his method of working to demand and not creating failure was bad for business.
It occurs to me that we’ve had several technologies come and go, the old green screen, client server, plain html, web 2.0 blah blah – this mean that the old solutions won’t work or needed to be redone. That’s fine. But within those solutions there was only ever a need to create something that was adequate for current needs – and then the shareholders wanted the money spent on something else. So we have a tension between old fashioned, siloed, projects and the yearning for continuous improvement. Short termism doesn’t want CI – it doesn’t care.
It will only change when the underlying model and the needs it encapsulates change. You can’t look at the software companies use divorced from the culture that created it.
I have a new client – outside my normal domain of space and defense – that has brought me to the realization you mention here. This is a new experience for me. In space and defense the “100% Mission Success” is a banner of one client. And pretty much that’s what they produce and certainly that is what informs their work.
I’ve come to realize that is not the “motto” of the firm I am now working in the Program Management Office. Thanks for the thoughts. Intentional failure is not the basis of the issues. I’d conjecture from my A&D experience as well as process control, power systems, and petrochemical software intensive systems, that the “Mission of Success” is missing. In an A&D the developers would be mortified if a defect “leaked” through the unit testing, integration testing, and test and evaluation of the “fit for purpose” and fit for use” lifecycle. “Oh well we’ll fix it in the next release” or “we need an emergency session of an all-nighter to fix that bug.”
Mission is the glue to connects people on the project. In the same “mission” connects most human endeavors to strive for excellence. As we have tried to teach our children (now young adults) – “it’s not about you, find a higher calling, change the world, make a difference.”
Thank you very much for this post: i find it very inspiring. I have been, as well, wondering for years why all that we know about software engineering is not used. However, I’ve not been as clear to state the problem as you are.
In his book “The Leader’s Guide to Radical Management: Reinventing the Workplace for the 21st Century”, Steve Denning writes:
In a bureaucracy, order trumps performance.
I think that bureaucracy is a significant factor that hinders quality in software development (and in many industries as well). Hierarchical bureaucracy is pervasive in companies and administrations, to the point that it has become synonymous of management. However, self-organization is key to real performance. But self-organization endangered order and bureaucracy cannot tolerate self-organization.
Most PMOs in the world are in charge of fighting self-organization, and are defeating performance. Most PMOs are in charge of keeping developers busy, by allocating so-called resources to precisely predefined tasks. That’s a typical pattern of what I call the “bureaucratic attitude”, as opposed to the “getting results attitude”. In a bureaucracy, survival is the watchword. People use their creativity to survive and prosper, to make politics, rather than to make great products.
I live and work in France, and France is a very bureaucratic society. Every citizen believes in widespread interventionism and everybody loves premature optimization. Higher management in large companies has mainly been trained to manage administrations. Many French startups think bureaucratic.
As far as I know, at least in New York and in San Francisco, there are companies that are less bureaucratic and more successful.
“And in most organisations today, I just don’t see that need manifest. Or even discussed. You?”
Well, I feel like I routinely see an imposter for it. It goes like this:
(1) Complain about the latest mess – because it’s not good enough (for some undisclosed definition of good enough).
(2) Find someone or some organisational ill such as a structural problem to blame.
(3) Fix the blamed (if that’s replacement of an individual, they usually go get the same sort of person again).
(4) Go back to (1).
The definition of insanity is doing the same thing over and over, expecting different results. Yet, that’s what most I’ve seen do. They seemingly believe they have the methods to address the issue but cannot confront the fact that things get no better in spite of the blaming and changing.
So, is there motivation present? I can’t tell. Given all the complaining, it sure seems so. Action is certainly taken in an attempt to fix it. Perhaps the missing motivation is to face up to this repeated failure and try something truly different.
But then when folks try something different they almost invariably just bend the new back to the old seemingly without noticing they did it. Seems like a form of self-deception or maybe another failure in having sufficient motivation? Speaking of self-deception, I’ve seen folks state that all of this is just the nature of software development! Never mind that it can be seen in many other disciplines as well!
At least right now, I’m tempted to attribute much of this to our nature. You know, the sort of thing Kahneman has written on. WYSIATI, the disconnection between our internally referenced abstract models and how things really work etc. That can’t be the only contributor though. There’s the education system, ego and such as well, surely?