What’s Wrong with the Project Approach?

What’s Wrong with the Project Approach?

Some time back the late, great Grant Rule wrote this paper on the problems with “projects” as an approach to organising software development. As the original has now ceased to be, I’ve taken the liberty of reproducing it here for posterity:

What’s wrong with the project approach to software development?

January 11th, 2011
Author: pg_rule

Pretty much since “software” was first invented (60 years ago?), numerous folk have been promoting an ‘engineering-led’ approach to ‘software projects’. Yet this advice goes largely unheeded, with the result that the relative success of IT projects is poor, and has improved very little during all my years in IT (38 and counting). Given that such admonishments seemed to have had such little effect in all that time, I also find myself asking, “Do I think it likely that further exhortations to those involved in ‘software projects’ to change their project practices is likely to achieve improved value delivery to stakeholders?”

And I have to conclude that the answer is “No”.

Following Albert Einstein’s adage that, “The definition of insanity is doing the same thing over and over again and expecting different results”, it seems to me that we need an entirely new approach. A new approach which goes to the root causes of what actually goes wrong in the end-to-end process. Why are the honest endeavours of software developers often so disconnected from the delivery of customer and stakeholder value?

Observation of what actually happens in organisations suggests there are two root conditions to the problem:

  1. Those responsible for business strategy are disengaged from, and know relatively little about, software-intensive systems and technology. So they structure their organisations so that software & technology people are segregated into silos. In those silos, the inmates talk amongst themselves in whatever arcane language they choose. But importantly, they don’t communicate (or interfere) with the ‘real business’.
  2. Everyone conspires to pretend that software-related activities should be managed as ‘projects’. That is, as chunks of work that start and end (on more or less clear and agreed dates), that have more or less well-defined goals, that contain a list of activities (tasks) that are assigned by a ‘project manager’ to a project team to which ‘resources’ are assigned for a limited period.

The results of studies too numerous to mention shows that most software projects are ‘challenged’ or ‘fail’. One study suggested that the majority of experienced project managers (and I am sure, folk playing other roles) expect at least 1 in 3 of the projects they lead to fail! As systems become more complex, and larger, they employ more teams combining projects into programmes… which further reduces the likelihood of successful achievement of the overall goals.

My conclusion is that we need a complete change of mindset. We need to move away from the inherently batch & queue concept of the ‘project life-cycle’ (as promoted by organisations including BCS, APM, PMI, OGC, SEI, NCC, ISO, IEEE, IET, etc. etc.) to a different approach.

I suggest that the required new mindset will accommodate the ideas of flow production and lean systems thinking that first began to be developed systematically (in e.g. automotive engineering) around 100 years ago. (Of course, one can trace elements of flow production & lean at least back to Carthage c.300-200 BC, but let’s skip over the history for now.)

I posit that Tom Gilb’s Evo method, and other agile methods such as XP, Scrum, Flowchain, and software Kanban, etc., begin to achieve ‘better’ results compared to ‘traditional’ big-design-up-front, wholly batch & queue methods, precisely because they encourage workers to focus on smaller batches of stakeholder value. In other words, value in terms the software developer can get to grips with.

Agile methods are one or more steps nearer to the ideal of ‘single piece continuous flow’. BUT… they are inherently limited because they continue to create & disband teams, to establish & abandon value streams, to create & throw away know-how, at – it seems – every opportunity. And crucially, they allow the C-suite and ‘business-side’ managers to ignore their responsibilities for the system of work and for the desired outcome.

Flow production (toward which Evo, Flowchain and Kanban currently make the nearest approaches IMO) would:

A) Make the entire end-to-end, whole-life, ‘concept to consumption & retirement’ process of defining, deciding, acquiring, designing, developing, operating, supporting, maintaining & replacing software- intensive systems a visible, inherent part of normal business operations… forcing issues onto the management horizon so they can be addressed as business issues – and not just something technologists worry about.

B) Because it would be apparent that software & IT issues were causing interruption to (or even cessation of) the flow of value, C-suite executives would have to recognise the pressing need to engage with software & IT related issues just as much as they do with other kinds of business issue. Conversely, the engagement of the ‘systems and software engineers’ with ‘the business’ would also be stimulated, the role of each and the communication between them finally becoming acknowledged as a main artery of the organisation’s lifeblood.

Flow production can only work effectively with the active engagement of all involved. For this reason it is a far more sustainable business model than other, perhaps more familiar, approaches. It embeds the ability to flex and respond to market forces deeply into the organisational culture. The focus on ‘the unforgiving minute’ forces constraints and problems out into the open where everyone can see them. It won’t allow problems to be swept under the carpet, or passed from one department to another like the proverbial buck. Hidden problems will always result in debts in one or more of the five kinds of capital. And such hidden debts, whether financial, technical, intellectual, social or environmental, all too often bite you when you least expect it. They will injure or kill the project – and destroy stakeholder value. Even if the project avoids repaying its hidden debts, this usually means that the debt has been passed-off onto one or another unsuspecting stakeholder group (sometimes, the end-customer or taxpayer). Which must be judged as unethical behaviour.

Unfortunately, not only have most people in the software industry been taught to sit in their silos and focus largely on coding, they and their masters have developed a cultural love affair with the project concept. To the extent that everyone assumes that all work has to be compartmentalised into projects, the very epitome of batch & queue thinking. Tell me what you think. Has the software project had its day, or is there another way of revolutionising workpatterns in the software industry?

– P Grant Rule

  1. Well, good, somebody noticed Gilb and Evo.

    I have seen no evidence that any part of Agile comes close. Some value chain analysis seems a bit better than Agile, which so many seem to take to mean just skip the documentation.

    Does anybody else start with making the (ten or fewer) stakeholder satisfying goals explicit and follow up with making explicit how progress toward each goal will be measured?

    Does anybody else evaluate the apparent possible next one week or less steps to guestimate the progress toward each goal that will be achieved by each proposed step in choosing which to do next?

    Does anybody else deliver working improved software every week to actual or stand-in end users to test if intended progress was achieved?

    Just let me know if any of these critical parts are even discussed, much less demanded as they are in Gilb’s Evo.

    • Hi Richard,

      Couldn’t agree more with your welcome words of wisdom. Been a student and applier of Gilb for decades now.

      – Bob

  2. I don’t work in IT (used to, won’t again) and at the company I work for the software I support is used daily by around 600 people, with about a thousand distinct users.

    • Sigh, the thing didn’t let me finish.

      Because I work in a “Business Unit” my team gets very little support from IT proper. This just adds to the difficulties… Anyway:

      I research, analyze, build, deploy, maintain and retire the software I build. Most of the tools I have built are still in use, but a handful have been retired. I consider any issue with the software, such as deployment, user support, upgrades, stability, error-handling, and user education to be my responsibility. As such, evidently, there is a ton of work, so there is never a day when I tell my boss: “Gee, I don’t have anything to work on this week”. However, I keep my priorities straight, debug before introducing new code, document, design, test, and deploy software all by myself, which frees him from all coordinating activities.

      Essentially, when the business units we support require applications built, they just send me in, and I meet with users, collect requirements, understand the value-added, the pain-points, and then build the software and add it to my list of things to maintain. I become the point of contact for the business unit, and my boss essentially have absolutely nothing to do. Needless to say there is a huge risk to the company should something happen to me, but it’s a risk they live with because the alternative is to do IT projects that take two years to get delivered.

      As far as silos, I treat any disruption in my ability to do my work as a bug and I work around it. As such, I ruffle feathers. My boss’ boss and his bosses like my work and value the value I deliver to the users and completely shield me from, hum, undesirable events.

      Of course, this hinges on my ability to deliver, and as such to not over-extend myself. As such, I rarely give target dates, even if pressed, and in the rare cases where I do, I caveat them with a large “If everything goes completely without problems” which the user then understand that I gave them a date to make them happy. So far nobody’s made a fuss about not getting dates, because I usually just build the software instead of performing CYA activities.

      One one-piece-flow: I have strudied Toyota’s production system, and Ford’s before, and applied to software. I get a lot more done when not multitasking at all. If I’m setting myself to do a task, I complete it fully before moving on. This is impossible to do in deadline crunch, so deadline crunches had to go.

      I looked at Agile development, and I realized there were still too many meetings, updates to be marked, etc. I just focus on the task of building the software, and do nothing else. No status reports, no update to bug tracking (unless needed by software), no meeting with boss-n-team. I just get down, do the work, deploy, train the users on use, let them go at it, and stay accessible for support or questions. That’s pretty much it. No time for anything else.

      • nitpicker077 said:

        You have also revealed a small problem I ran into as well. When I undertook to save PL/I from 22 new on conditions to accomplish something like COBOL’s Report Writer feature, I did it alone, at home, in a couple of weeks, but I could not teach anyone else how to stuff all that understanding into one’s head and spit out 600 lines of nearly bug free pre-processor code. That distresses me since it has no future. You don’t speak of writing down what you do in detail, or of taking on any apprentices to teach to work that way too, so I suspect the successes will die with you, too.

  3. @nitpicker077
    That’s true. I do document, but it’s not my job to train someone in the art of computer programming. I write the documentation for someone skilled in programming. If they need to replace me, they need to replace me with someone skilled. That person will need to know their skill, and they will most probably use their own way to do things, so I can’t give them a click here, click there manual. They just need the high-level information.

    Generally, what you point out is true, that I am not replaceable, but that would be true no matter what trade I was into. Also, this does not distress me. I am not married to the code I produce for the company. I consider it a tool, nothing more.

    • Dave C said:

      Christopher – enjoyed the post and I found myself thinking all the way back to when I “were a young programmer” – I wasn’t very good at coding – great at solving the problem – but my code was often a bit “clunky” – and the more I watched the great developers the more I really felt it was more art than science. Even more the fact that I think the development language itself influenced my creativity and ability and I only really got a bit better at coding when I tried Cobol (which must mean something – probably bad!). As I read your post I found myself nodding furiously in agreement and I remember the small software house I worked in – we wrote OK documentation within the code and more or less it was adequate (or fit for purpose!) – the good developers were usually happy with it – the weaker ones – ie me – took longer to get up to speed and asked more questions.

  4. Dave C said:

    Bob – love this post. So much so I liked it in linked in – where it’s had a few more comments. I have I think, found my spiritual home in thinkdifferent and rightshifting and will be spending more time reading some of the great stuff you guys write about. My journey into thinking different started when i read a book five years ago by Dee Hock (founder of visa) call the Birth of the Chaordic Age – I loved it and since then have become more and more intrigued by what some (less fun people) call Complex Adaptive Theory (I preferred Chaordic!)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: