Why do project post-mortems miss the real cause? Learn how survivorship bias hides missing capacity and capability in IT project data and dashboards.
These days, almost everything can be measured.
And IT IS!!
From social media algorithms to Fitbit watches to IT projects.
IT projects generate mountains of data: budgets, milestones, resource allocations, risks, issues, defects, change requests and delivery metrics. Dashboards turn this information into colourful indicators of project health, giving us the impression that, with enough data, we can understand what’s happening and make better decisions.
BUT what if some of the most important factors influencing project success never make it onto the dashboard in the first place? What if the real problem isn’t hidden somewhere in the data, but lies in something that was never there to be measured?
First, a quick history lesson.
THE MISSING BULLET HOLES: WHAT WWII CAN TEACH US ABOUT DATA
During World War II, Allied bombers were being shot down at catastrophic rates, and commanders needed to add armour. But armour is heavy, so covering the entire aircraft was impossible. They had to reinforce only the most critical areas.
To find them, the military mapped the bullet holes and shrapnel damage on the aircraft that returned from combat. A clear pattern emerged: the wings, fuselage and tail were riddled with holes, while the engine and cockpit were largely untouched. The “obvious” conclusion seemed written in the data: armour the wings and body.
But a mathematician named Abraham Wald saw what the military brass had missed. They were falling victim to survivorship bias. They were studying only the planes that made it back.
Wald pointed out that the holes on returning planes proved those areas were survivable. A plane could take a hit to the wing and still limp home. The clean, untouched areas, the engines and the cockpit, were missing from the data because planes hit there – never returned.
So, he recommended armour where the bullet holes weren’t.
I see this lesson regularly in IT project post mortems, but here the missing bullet holes are often the capacity and capability nobody recorded.
THE PROBLEM WITH THE PROJECT POST-MORTEM
Fast-forward to a modern IT project that has gone wrong. It ran three months late. Requirements changed repeatedly. Defects increased during testing. The budget was exceeded and stakeholders were unhappy. The post-mortem identifies weaknesses in requirements management, estimation and change control. Actions are agreed, lessons are documented, and everyone moves on.
Except, perhaps, the real problem was never identified.
Imagine the project had no dedicated business analyst. Requirements gathering was squeezed between other responsibilities, business stakeholders were stretched, and developers were left interpreting what users actually needed. Misunderstandings were inevitable. Requirements changed, rework increased, and testing uncovered functionality that didn’t meet business expectations.
The data faithfully recorded the consequences. It didn’t reveal the cause.
The retrospective may well have focused on improving requirements management processes, when the project lacked the people to manage requirements in the first place. You can improve your processes until you’re blue in the face, if nobody has the time or capability to make them work, you won’t get the outcome you’re looking for.
The same applies to project management itself.
Picture a project that repeatedly misses decision deadlines, leaves dependencies unresolved and escalates risks too slowly. The project manager is criticised for weak governance. But they’re also responsible for another major delivery, dealing with operational issues and sitting through a seemingly endless succession of meetings. The resource plan says the project has a project manager. The reality is that it doesn’t have enough project management capacity.
Assigning someone to a project isn’t the same as giving that project the capability it needs. A name against a role says nothing about competing priorities, available hours or the focused attention the work receives. Meanwhile, the usual response, more reporting, tighter controls and extra governance meetings, can add to the very pressures that caused the problem.
WHEN SUCCESS HIDES THE PROBLEM
Here’s where it gets particularly interesting, because this isn’t just about why projects fail.
Imagine a project delivered on time and within budget. Milestones met, testing completed, solution accepted by the business. By conventional measures, that’s a success.
BUT … behind the scenes, developers worked late to resolve defects, the project manager spent evenings catching up on planning and reporting, and business stakeholders repeatedly dropped other responsibilities to keep decisions moving. The project succeeded through a considerable amount of informal effort that never appeared in the official reporting.
The lessons learned celebrate the delivery approach, and the organisation decides to replicate it. Except the next team is different, the circumstances are less favourable, and nobody is available to compensate for the same underlying weaknesses. This time, the project struggles.
The first project’s data recorded a successful outcome accurately. What it didn’t capture was how that outcome was achieved. Success can conceal weaknesses just as effectively as failure can. Sometimes a project succeeds because its delivery model is effective. Sometimes it succeeds despite that model, thanks to individual effort and workarounds that aren’t sustainable. If we don’t investigate the difference, we risk mistaking exceptional effort for a repeatable process.
LOOKING BEYOND THE DASHBOARD
None of this is an argument against data. Good project data is essential. It helps us spot trends, challenge assumptions and make informed decisions. Without it, we’re left relying on memory, intuition and whoever tells the most convincing story in the project review.
But data is only as useful as the questions we ask of it. When a project misses its milestones, we should examine the schedule and also ask whether the assumptions behind it were realistic. When requirements change, we should investigate the changes and also ask whether the project had the business analysis capability to manage them.
The challenge for project managers, sponsors and PMOs is to recognise that the absence of evidence isn’t evidence that something wasn’t important.
Sometimes the missing information concerns work that happened but was never recorded.
Sometimes it’s a capability that was never provided.
And sometimes it’s a problem nobody recognised because nobody asked the right question.
THE MOST IMPORTANT QUESTION
Perhaps the most valuable part of a post-mortem isn’t the dashboard, the lessons-learned log or the final status report. Perhaps it’s the moment someone asks a question nobody thought to ask before.
Did we have the right capabilities? Were we under resourced? Did our people have enough time to do their jobs properly? Did success depend on extraordinary individual effort? Were we measuring the project’s actual health, or simply the indicators we had chosen to represent it?
At Stoneseed, we “rescue” a lot of projects and, even better, we get to use the experience to help shore up projects from the start. Often, the most important lesson from a project isn’t found in the data it produced, but in what the data never told us to look for.
