Why don't we see more Agile in records management programs?

The agile manifesto represented a set of realisations about the way software was being designed and coded.

The main one (at least to my reading) was that trying to design everything up front didn't work (this is what waterfall tries to do).

Part of it was the problem that George Bernard Shaw outlined when he said that "the biggest problem in communication is the illusion that it has taken place."

The other problems are that we're people, and as much as we think that we can tell people what we think, mostly we can't - and there are lots of good working theories that take this into account - from Christ Argyris theory-in-use vs. theory-espoused to economics' revealed preference theory (basically that you can't trust people to tell you what they want, you have to get them to buy something with real money) to the Dunning-Kruger effect.

Dunning-Kruger is particularly interesting.

When we're building a system, we're starting from a place of low knowledge about the system, and then we start designing.

Along the way, we gain a certain level of knowledge and certainty about what we're designing - and we start to believe in it.

And we fall victim to the endowment effect - which just says that we value things that we own more than they're worth.

And this is critical.

Because it says we can fall in love with our own system before it meets the real world, before we have knowledge about how it meets the real world (and at the time when we've fooled ourselves that we do have it), at the time when we're dealing with what people have told us about how the world works - vs. how it actually does, and the time when we're dealing with what we thought they were saying rather than what they actually were.

Agile teaches us that value needs to be delivered collaboratively, can only be decided by the people who "buy" the system (even if they only buy it by opting to spend their time using it vs something else), and is something that we almost always fail to deliver if we try and design too much ahead of time.

These are things that software design has known for at least 20 years - which is why Agile is eating the world - indeed, it's now been around long enough that lots of people are starting to think that it has come to the end of its success.

Why haven't we learned from these lessons?

Why haven't we learned, that designing a system up front results in a system that doesn't deliver value that people want, in a way that they will use?

In the face of so much evidence.

More than anything, records has to work.

The things we do have to result in meaningful capture rates, and meaningful business improvements.

The evidence shows that this isn't our primary concern.

The evidence shows that we're much more concerned about being right about the design principles, than being right about what works in the real world.

It's the triumph of dogmatism over pragmatism.

Why haven't we learned from Agile?