Why has Agile been such a depressing failure?

There’s been a decade of grumbling about the problems with Agile (one of the original Agile Manifesto authors called time-of-death in 2014) but there had already been quite a lot of hand-wringing before that. And you can watch dozens of videos on YouTube and go to countless conferences telling you that the solution to bad Agile is just more & better Agile. I think this is fundamentally wrong - what this misses is the why. Why is Agile so hard?

Success consists of going from failure to failure without loss of enthusiasm.

Firstly, what Agile got right, and why it changed the world: Agile was successful because it recognized that the participants in software development were humans. And that humans have certain limits and abilities, and if you can factor those limits & abilities into your process, you have a much better process.

For example, we humans are really bad at estimating how long things will take. And really bad at predicting the future. And keeping our minds made up for long periods of time. Waterfall is a terrible process because it requires humans to be brilliant at exactly these things, and we just aren’t. But we’re really good at iterating - starting with anything and making it better. And we like frequent deadlines and frequent feedback and a continuous stream of small rewards. So Agile brilliantly addressed these limitations with humans and provided a great way to leverage our strengths. So why does it so often go off the rails? Why do so many people feel like Agile works only everywhere else, and never where they actually work?

An overemphasis on teamwork and consensus can cause a kind of mental lethargy. If you subordinate your own expectations to the expectations of the group, you’re absolved of the responsibilities for the choices the group makes.

The fundamental problem with Agile is it assumes all the humans in the process are the same humans. But any good team is going to have a diversity of personalities, and this is where Agile starts to fall down.

In my consulting, I do a lot of Agile remedial work. Invariably, people do Daily Standups and so I ask to attend those as a first step of diagnosis. And, invariably, they are awful. You only need to look at the faces of those who aren’t currently speaking - they are completely bored, and probably looking at their phone if they bothered to show up at all. So why does everyone dread them? In fact, why do most people dread most of the Scrum ceremonies (sprint planning, review, retrospectives…) and why can almost noone - when asked - point to any tangible benefit of doing them?

Wrong questions! Good questions, sure, but missing an important point. If a particular team’s standups are so awful, why does that team still do them? The reason tells you something about the behaviour of groups of people, and this is what Agile misses. While each individual in the team might understand and agree with ‘Individuals and interactions over processes and tools’, as a group they behave in the opposite way, acting slavishly to the Scrum methodology.

Another example is Story Points, another of the most derailing concepts to ever hit software. Now, we all know that humans are terrible at estimating how long things will take, and Story Points offer a brilliant escape by taking the time out of estimation and replacing it by counting similar items. But when you Google ‘story points’, you see what people really want to do with them:

Now, the people in a team who are especially abstract thinkers will understand Story Points and how they should be done. They might even convince the team to ‘do it the right way’ for a while, but they will lose the battle, sooner or later. There’s a natural ‘sink’ that teams will roll down into with Agile.

Much of the Agile Manifesto actually works against the instincts and intrinsic rewards of teams. Sure, it’s better than the rolling disaster that was Waterfall, but we haven’t yet discovered a way of working that aligns with a diversity of team personality types. Sure, we’ve recognized that the participants of software development are humans.That was a great first step. But the next step would require us to recognize the different humans involved, and Agile doesn’t and can’t do this, at least in its present form.

So what’s depressing about this?

Well, the sad conclusion is that there likely isn’t a philosophy or methodology that ‘just works’. Not Scrum, not Kanban, not XP, not SAFe, not nothing! At best, these are collections of tools one can draw inspiration from, to be wielded well or poorly by the leaders who hold them. The dominant criteria for success are actually the same as they ever were: culture, leadership and group dynamics.