The Number That Tells you Nothing
On choosing the manageable number over the understanding that matters
Every year, organisations across every sector spend significant money finding out how engaged their people are. The answer comes back as a number. Sixty-two percent. Seventy-one. A red, an amber, a green. And then, in most cases, very little changes — because a number, however precisely measured, does not tell you what to do next.
This is not a failure of effort. It is a failure of method. The question being asked — how engaged are people, on a scale of one to ten — is structurally incapable of producing the intelligence needed to act. It tells you the temperature of the room. It does not tell you why the room is cold, which walls are letting in the draught, or whether the people in it have stopped expecting anyone to fix it.
The pattern holds across every sector. An NHS trust running annual staff surveys. A private equity-backed scale-up tracking employee sentiment through quarterly pulse checks. A charity trying to understand what its service users actually need from a redesigned programme. A number where there should be a conversation.
What the number misses
The case for quantitative engagement measurement is not wrong. Trends matter. Comparability matters. Knowing that satisfaction dropped twelve points between January and March, or that one team consistently scores lower than the rest of the business, is genuinely useful. The problem is not that organisations measure; it is that most treat the measurement as the understanding, rather than as a signal that understanding is needed.
Qualitative intelligence — what people actually say, in their own words, about their own experience — produces something different. It surfaces the specific, the unexpected, and the things no survey designer thought to ask about. It also surfaces disagreement within a group that aggregate scoring conceals. A team with an average engagement score of sixty-eight might contain people who are deeply committed and people who are quietly preparing to leave. The number averages them into false coherence.
The difficulty has always been scale. Reading and coding open-text responses from three hundred employees, or three thousand service users, is expensive and slow. The analysis introduces its own biases. And the time between collecting the data and acting on it is long enough that the context has already shifted. So organisations default to the form that is easiest to process — the number — and accept its limitations as the cost of manageability.
That trade-off no longer holds as firmly as it did. The combination of open-response collection, peer voting, and AI-assisted analysis — the approach taken by tools like Gobby, which works in this space — means that qualitative data can now be gathered and synthesised at a speed and scale that was not previously practical. People respond in their own words. Those responses are surfaced to peers, who vote on what resonates. The result is not just a fuller picture; it is a picture that has been validated by the community itself, not just interpreted by the analyst.
The pattern across sectors
In the public sector, the gap between what is measured and what matters is often widest. Patient experience surveys ask whether care was clean, timely, and respectful — categories defined by the regulator, not the patient. What they rarely capture is whether the patient felt heard, whether the treatment made sense in the context of their actual life, or whether the system’s definition of a good outcome matched their own. The number that comes back is real. The experience it describes is partial.
Local government consultation has the same problem at a different scale. A planning consultation that asks residents to rate a proposal on a five-point scale produces a mandate for or against. It does not produce understanding of why, which aspects are acceptable and which are not, or what a modified proposal might need to look like to bring the sceptics with it. The binary produces a decision. It does not produce the intelligence that would make the decision stick.
In the private sector, the equivalent is the employee engagement survey and its close relative, the customer satisfaction score. Both have become so ritualised that the measurement has partially replaced the thing it is supposed to measure. Teams know how to answer the survey. Customers know that the score goes nowhere. The process continues because it produces a defensible number, not because it produces understanding. When engagement drops, the standard response is a communications campaign — because the survey told you sentiment, not cause.
The charity and social enterprise sector is arguably most exposed. Organisations whose entire purpose is to serve and represent specific communities often have the least developed means of understanding whether they are doing so. Impact reporting is dominated by output metrics — number of people served, sessions delivered, referrals made. What is missing is the voice of the person at the end of the service, speaking in their own terms about whether it changed anything for them. Not a satisfaction score. A considered account of what happened.
Understanding you can act on
The value of more detailed engagement data is not primarily analytical. It is operational. When the reasons behind a trend are visible — not inferred, but stated by the people who are living them — the path to action becomes clearer and the likelihood of getting it wrong decreases.
A portfolio company workforce that scores low on engagement is a problem. A portfolio company workforce that is able to articulate, collectively and specifically, that the integration plan removed the three things that made the culture work is a solvable problem — or at least one where the board can make an informed decision about the trade-off. A service user group that can say, in their own words and with their peers’ endorsement, that the new referral pathway is harder to navigate than the one it replaced gives a programme team something to act on rather than something to commission further research about.
My Opinion
Every organisation I’ve worked with runs some version of the same survey — historical questions, coded to a scale, returned as a number. I’ve seen what happens when you replace that with something closer to in-the-moment: a city authority that put real-time word clouds — drawn from live citizen responses — on a screen in the corridor where the chief executive walked every morning. The conversation in that building changed. Not because the data was new, but because it was immediate, in people’s own words, and impossible to walk past. That is the difference between recording sentiment and understanding it.
The shift required is not primarily technological. The tools exist. What is required is a change in what organisations believe engagement measurement is for. If the purpose is compliance — to show that a survey was conducted and a score was recorded — then the number is sufficient. If the purpose is understanding — to know what is actually happening and why, well enough to do something about it — then the number is a starting point, not an answer.
The organisations that close that gap are not the ones that find a better survey platform. They are the ones that decide the distinction matters.
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.