Casey stood in the kitchen on a Sunday morning wanting to do something small and

ordinary. The lights were already on — the right level, the right temperature, the system having made the calculation sometime before waking. The heating was already at the temperature the house knew a Sunday morning required. The coffee machine had already started its cycle.

Not one of these things had been consciously chosen that morning.

The intent had been to make a deliberate choice about the day — to begin it with a moment of decision rather than discovery. To turn the lights up brighter than usual, perhaps, because it was overcast outside and the mood called for it. To make the coffee later than normal because there was no urgency. Small acts of self-determination in an ordinary morning.

The system had already decided. It had observed enough previous Sunday mornings to know what this one required. The preferences were not wrong — they were, in fact, accurate. They reflected genuine past behaviour, real choices made on real mornings over months. The system had learned well.

That was the problem.

The disappearance of friction

There is a version of this story that reads as convenience. The house anticipates. The morning runs smoothly. That is what the technology was designed to do, and it is doing it.

What that version of the story does not account for is what friction was for.

The small acts of choosing — the light level, the temperature, the pace of a morning — are not merely mechanical. They are the continuous practice of preference: the low-stakes exercise of deciding who you are and what you want on any given day. They are how a person remains in contact with their own inclinations. When those acts are automated away, the automation does not just remove effort. It removes the opportunity to notice whether the choice would have been the same.

Casey’s house had learned a version of Casey built from past behaviour. Not from yesterday’s behaviour, from the cumulative pattern of months. A version that was accurate in aggregate and potentially wrong in any specific instance. The system had no mechanism to register that people change, that moods shift, that a Sunday in January and a Sunday in March might call for different things even if the previous fifty Sundays had called for the same ones. The actual person was no longer quite required.

The moment of choosing had disappeared so gradually that it was only noticeable in its absence. Not when the system got it wrong, it rarely got it wrong in any obvious way. But when the desire arose to do something differently, and the system had already done it, and the act of undoing it felt like more effort than simply accepting what was already there.

That is the trap. Not coercion, convenience so complete it becomes its own constraint.

From the kitchen to everywhere

The smart home is the most intimate version of this pattern, because the home is the space where people are most fully themselves. But it is only the beginning of the scale.

Step outward from the kitchen and the same architecture is running at every level. The streaming platform that has watched enough viewing history to know what will be selected before the selection is made. The news feed that has modelled enough reading behaviour to decide what is worth knowing today. Each of these systems is doing the same thing as the smart home, in a different register: replacing the act of choosing with the delivery of a prediction so accurate that choosing no longer feels necessary.

Step further out. The e-commerce platform that has learned spending patterns well enough to surface only what will be bought — not what might be wanted, not what might be surprising, but what the model says is coming next. The navigation app that routes not by the driver’s preference but by the algorithm’s assessment of the optimal path, such that after a year of use a driver may no longer know the roads they travel every day. The workplace tool that pre-populates responses, pre-schedules meetings, pre-drafts the email — reducing the human contribution to the act of approval.

At each step the friction removed is real. The convenience is not manufactured, it is delivered. And at each step something else is happening simultaneously: the system’s model of the person is deepening, the person’s practice of self-determination is narrowing, and the gap between who the system thinks they are and who they actually are is quietly widening.

The model is built from what was. The person is living in what is. Those two things are not the same, and the longer the system runs without the person having reason to notice the divergence, the larger the gap becomes.

Step to the outermost scale. The political information environment, the social media feed, the search result page, all of them running the same logic at civilisational scope. Not just anticipating individual preferences but shaping them, because a preference exercised without friction, without encounter with the unexpected, without the resistance of a world that does not already agree, is a preference that stops developing. The system does not just reflect who you are. Over time, it shapes who you remain.

The distinction nobody offered

The question of how much anticipation is enough is not rhetorical. It is a real dilemma.

Friction is not inherently good. The effort of choosing is not valuable in itself, it depends entirely on what is being chosen and why. The friction of adjusting the thermostat is not the same as the friction of deciding what to read or what to believe or what kind of person to become. Not all convenience is erosion. The task is to distinguish between the friction that was serving a purpose and the friction that was simply in the way, and to recognise that those two things look identical until the moment of removal.

Most people have not made that distinction deliberately, because it was never offered to them as a choice. The anticipatory systems arrived and removed friction and the question of which friction was worth keeping was never asked. The default was anticipation. The person who wanted to preserve the act of choosing had to actively resist the convenience of not choosing, and resistance is itself a form of friction that most people do not have time or inclination to apply.

The deeper consequence is not about convenience it is about the model. Every anticipatory system builds a representation of the person it serves. That representation is the basis for every subsequent prediction, every pre-emptive action, every choice made on the person’s behalf. The richer and more accurate the model, the more completely it can substitute for the person’s actual judgment. And the more completely it substitutes, the less opportunity there is for the person’s actual judgment to develop, correct, or surprise itself.

A person who has never had to choose their own route no longer knows the roads — a 2024 study confirmed this is not a metaphor: habitual GPS use produces a measurable, causal decline in spatial memory, even in people who previously navigated well. A person whose reading is pre-selected no longer encounters what they did not know they were missing. A person whose home has already decided never fully inhabits the decision to be at home. The capacity that is not exercised does not remain dormant. It diminishes.

That is the cost that does not appear in the product specification. That is the human loss that convenience optimises away.

Who is this for

The question the anticipatory system never asks, because it was not designed to, is who benefits from the anticipation.

The person using the system receives convenience that is real. But the system is not running in their service alone. Every preference recorded, every choice predicted, every behaviour anticipated is data. Data that trains the model, refines the prediction, deepens the platform’s understanding of what the person will do next. The more accurate the anticipation, the more valuable the person becomes as a data source and the more dependent they become on a system whose primary obligation is not to them.

The smart home manufacturer knows the daily rhythms of the household. The streaming platform knows attention patterns more precisely than the viewer knows themselves. None of this knowledge is held in the person’s interest by default. It is held in the platform’s interest, governed by the platform’s terms, subject to the platform’s commercial decisions about how that knowledge is used, shared, or sold.

The anticipatory trap is not just about agency. It is about the terms on which that agency has been surrendered and to whom. Casey’s house is learning Casey — but who owns what has been learned, and what happens to it when the subscription ends, the company is acquired, the terms are changed? The system was presented as a tool serving the household. The data flowing out of it tells a different story about who is serving whom.

The system that anticipates your needs perfectly has built a more accurate model of you than you have of yourself. That is impressive. It is also alarming, not because the anticipation is wrong, but because the entity holding that model is not you, does not answer to you, and did not ask your permission to build it in quite the way it has.

The friction that was removed was not just effort. Some of it was the resistance that kept you in contact with your own preferences. Some of it was the randomness that kept you encountering the unexpected. And some of it was the inconvenience that made you notice, every now and then, that a choice was being made at all.

What it was replaced with is not neutral. It serves someone. The question is whether you know who.

What you might find you’ve forgotten

Think about the systems in your daily life that anticipate your preferences — the home, the phone, the feed, the platform. When did you last make a deliberate choice in a domain where anticipation now operates? Not overriding a suggestion — making an original decision, from scratch, without a pre-populated option in front of you?

The model these systems hold of you was built from your past behaviour. How accurate is it to who you are now? Have you changed in ways the system has not registered? Have your preferences shifted, your interests expanded, your circumstances moved, and has the anticipation moved with you, or is it still running on data from a version of you that no longer quite exists?

Who holds the model? Not in terms of which company’s name is on the product, in terms of what happens to what has been learned about you. Is it portable? Is it deletable? Is it yours, in any meaningful sense? Or has the convenience of the service been partly funded by the permanent transfer of something you did not consciously offer?

And the question underneath all of these: if the anticipation were removed tomorrow — if the systems stopped predicting and simply waited for instruction — what would you find you had forgotten how to choose?

My Opinion

The technology itself is not the problem. What makes ambient systems genuinely difficult is their invisibility. A tool you can see, you can use deliberately. A system that operates below the level of conscious attention has already made the choice about when and how it operates. The line between the two is not technical — it is a design decision, made by someone other than you, before you arrived.

Most ambient technology was not designed to be controlling. It was designed to be easy. But easy and controlling share a direction of travel. The easier a system makes a decision, the less often you will make that decision yourself — and at some point along that gradient, the decision stops being yours. It stopped leaving room for it.

Not everyone needs this. That sounds obvious but it is rarely said plainly. There are households where ambient technology solves a real problem — accessibility, health monitoring, the household stretched across too many responsibilities to manage routine decisions without help. In those circumstances it is doing what it was built for. As a default for everyone, replacing the ordinary friction of a morning with a prediction of what the morning should be, it is a different proposition — one that takes something from the quality of ordinary life that is hard to name until it is gone.

The question is not whether to use these systems. It is whether you know what you are giving in exchange. Convenience offered at no visible cost is rarely free. What is given in return is usually the data that makes the system more accurate, and the habit of not deciding that makes it more necessary. Both transfers happen gradually, without announcement, and by the time they are noticeable, reversing them requires more effort than most people are willing to apply on a Tuesday morning.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan opens the app on a whim. It is one of those data-transparency tools that have become briefly fashionable — the kind that pulls together what advertisers, brokers, and platforms believe about you and shows you a summary. Morgan is curious, not anxious. It takes about thirty seconds to load.

What comes back is strange. The profile is accurate in the way a caricature is accurate — it captures something recognisable while getting most of the important things wrong. Morgan is categorised as a “high-anxiety urban professional,” likely to respond to time-pressure messaging and financially cautious. There are interest tags: travel, wellness, mid-range electronics. A risk band. An inferred household income. Morgan reads it twice, unsure whether to laugh or feel uneasy. Neither feels quite right.

When behaviour becomes identity

The discomfort is worth paying attention to. Not because the data is false — most of it is close enough to be recognisable — but because it is being treated as complete. The profile was assembled from clicks and purchases and search patterns, and somewhere along the way it stopped being a description of behaviour and became a verdict about a person.

That shift happens quietly. A system designed to predict preferences begins to define them. A tool built to segment audiences starts to assign identities. Nobody decided this was the purpose. It emerged from the logic of optimisation, from the pressure to know more in order to sell more, to lend more wisely, to assess risk at scale. The profile is a side effect of a process that was never really about the person at all.

The profile is accurate about behaviour and wrong about the person. The danger is not the bad data — it is that the system believes data is enough.

A trace treated as a portrait

Here is the underlying problem: data captures what someone did, in a particular moment, under particular circumstances. It does not capture why. It cannot hold the week someone was caring for a sick parent and spending on delivery food because there was no time to cook, or the period of job uncertainty that explains a cautious financial pattern, or a single impulsive holiday purchase that rewrote the travel interest score. The data is a trace. The system reads it as a portrait.

This matters because these profiles are not just sitting in a database somewhere. They are active. They shape what someone is offered, what rate they are quoted, sometimes what opportunities appear and which do not. A mortgage application, a job screening, an insurance premium — in each of these, the statistical shadow is treated as the person. And the person has almost no practical way to correct it, because most of the time they have no idea it exists.

The portability of these profiles makes it worse. An assumption made in one context follows into another. A risk score generated for one purpose bleeds into decisions it was never designed to inform. The data travels; the context does not. What began as a description of what someone did last Tuesday is now being used to decide what they deserve next year.

What the profile decides

The human cost here is not always dramatic. Sometimes it is a loan offered at a higher rate than someone with an identical financial history but a different data footprint would receive. Sometimes it is a job application that does not progress past the screening layer. Sometimes it is simply the low-grade unease of being looked at by systems that cannot see you, being assessed by processes you cannot access, being defined by a version of yourself that you did not author and cannot revise.

For those who are already disadvantaged — people in financial precarity, people with gaps in their histories, people whose lives do not fit the patterns the models were trained on — the distortion is not low-grade. It is consequential. The system is not malicious. It is simply indifferent to the difference between a data point and a human being. That indifference is its own kind of harm.

What the design is actually doing

In the original use case — recommendations, relevant offers, faster credit decisions — there is a genuine service on offer. But it holds only while the profile is accurate, and only while the person has some say in what it contains. Where the profile is wrong, or where it follows someone into contexts where it has no business being, the service becomes a constraint.

The efficiency gains for the institutions using these systems are real. What has been added for the individual is less clear: a permanent, portable verdict about who someone probably is. Whether that addition has value depends entirely on who is doing the measuring — and for whom the system was designed.

Where it fails most visibly is in adaptation. The profile is static in the ways that matter most. It captures behaviour but not situation. It updates when you click, but it does not reset when your life changes. It does not know the difference between who you were and who you are, and it has no mechanism for registering who you are trying to become. A system that cannot hold that kind of ambiguity is not adaptive. It is just persistent.

This is one of the questions at the centre of The Cognitive Crucible — not just what the system records, but what it misses by design.

My Opinion

We are living through a period of digital identity fragmentation. Pieces of us — clicks, purchases, search patterns — are being gathered by third parties and reassembled into a composite. The problem is not just the assembly. It is the premise: that this composite represents a person.

It does not. We are not one thing. We are many, depending on context — different at work and at home, under pressure and at ease, at a particular age, in a particular relationship. A system that flattens all of that into a single profile is not describing us. It is describing a version of us that has never existed.

Taking back agency over this will be hard. It may require something counter-intuitive: choosing, deliberately, to define and manage our own digital identities — profiles that represent genuine aspects of who we are in a specific context, rather than ceding that definition to systems that were never designed with us in mind.

The questions underneath the discomfort

There is a reasonable question underneath the discomfort the opening described. Not “how do I get off this list” — that is almost certainly impossible — but something more fundamental: who is this profile actually for?

The profile was not built for the person. It was built for the systems that process them. That distinction changes the nature of the conversation. The question is not whether the data is accurate. The question is whether data should be sufficient.

1. Have you ever encountered a version of yourself through a system — a recommendation, a decision, a risk band — that felt both recognisable and completely wrong? What did you do with that feeling?

2. If you could see your full data profile across every platform and institution that holds one, what do you think you would find — and would you want to know?

3. If you could change one thing about how these systems work — how they are built, what they are allowed to infer, how they can be challenged — what would it be?

4. The system knows your clicks. Should that be enough to know you? And if not — what else should be required before a system is allowed to make decisions about your life?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The people report comes toward the end of the board pack. Somewhere after the financial performance slides and before the risk register. It contains an engagement score — this quarter seventy-one, up three points from the previous period — and a brief commentary from the people director explaining the movement. The board notes the improvement. Someone asks whether the score is above or below sector average. The answer is above. The item closes.

Six months later, a significant portion of the senior leadership team has left. The reasons, when they eventually surface, are not complicated: a sustained period of organisational change had been managed without involving the people most affected in the decisions being made, and those people had concluded, rationally, that the direction was not one they wanted to follow. None of this was invisible. It was legible, in specific terms, in the free-text responses that sat beneath the engagement score the board had accepted as reassurance.

The board had asked the right governance question — how engaged are our people — and received an accurate answer. The number was real. What it failed to convey was whether the organisation understood the conditions producing it, and whether those conditions were stable or deteriorating beneath a score that was still, technically, above average.

This is not an unusual story. It is a governance pattern.

What a people metric is designed to do

Board-level people reporting has evolved to serve a specific function: to give non-executive directors sufficient assurance that the human risk in the organisation is being managed. The engagement score performs that function efficiently. It is comparable across periods, benchmarkable against sector peers, and defensible in the event of a regulatory or reputational challenge. It answers the governance question the board is equipped to ask.

The problem is that the governance question and the strategic question are not the same question. Is engagement above or below threshold is a governance question. Do we understand what our workforce, our customers, or our communities are actually experiencing well enough to anticipate what is coming is a strategic one. The first produces a number. The second requires intelligence. And most board people reporting has been designed to produce the former while giving the impression it addresses the latter.

The distinction matters because boards make consequential decisions about people — through strategy, through resource allocation, through the acquisitions and restructures they approve — and the quality of those decisions depends on the quality of the understanding they rest on. A board that believes a score of seventy-one means everything is broadly in order is making different decisions than a board that understands why the score is seventy-one, which parts of the organisation are driving it, and what would need to change for it to move in either direction.

The score alone cannot support the second set of decisions. It was not designed to.

What boards are not seeing

The gap between what boards receive and what would constitute genuine people intelligence is wide in most organisations, and wider still in specific situations: post-acquisition integration, significant organisational change, sustained periods of uncertainty, or industries where the workforce is distributed, contracted, or otherwise at a distance from the centre.

In each of these situations, the aggregate score becomes less representative precisely when it most needs to be reliable. The integration that is going well for the acquiring team and badly for the acquired workforce produces an average. The restructure that has settled the leadership layer while creating deep anxiety in the operational layer produces an average. The number consolidates the variation into a single point and presents it to the board as a picture of the whole.

What the board is not seeing, in most cases, is the distribution beneath the average — where the score is concentrated, where it is fragile, which parts of the organisation are carrying the weight of others, and what the people in the lower quartile are specifically saying about their experience. Nor is it seeing the qualitative register: the specific concerns, the named conditions, the articulated reasons that would allow a board to distinguish between an engagement dip that is transient and one that is structural.

Boards that have tried to close this gap by asking for more granular data — scores by function, by tenure, by level — get closer to the distribution problem but do not solve the intelligence problem. Granular numbers are still numbers. They show you where the variation is. They do not show you why.

The question no one asks in the boardroom

There is a version of this problem that is specific to the board’s role, and it sits just beneath the surface of every board pack.

Boards are responsible for oversight, not management. The distinction is important. A board that receives an engagement score and accepts it as adequate people intelligence is not failing to manage — that is not the board’s job. It is failing to ask the questions that oversight requires: what does this number actually tell us, what does it not tell us, and what would we need to know to be confident we are not surprised by something we should have seen coming.

In most organisations, no one in the boardroom asks those questions routinely. The people director presents the data they have been asked to present, in the format that has been agreed, against the benchmarks that have been established. The non-executives receive it in a pack that also contains financial performance, risk, and compliance items that require more active scrutiny. People data gets the time allocated to it, which is rarely enough to interrogate it deeply.

The result is that people risk is governed at a level of abstraction that would be considered inadequate in other domains. A board that accepted a single aggregate technology risk score without understanding what it represented, where the exposure lay, and what conditions were driving it would be considered negligent. The equivalent in people governance is standard practice.

The shift required is not primarily about tools or data. It is about what boards decide to demand. Engagement intelligence that is specific, distributed, and grounded in what people actually say — rather than how they respond to a scale — is achievable. Gobby, a UK platform built on qualitative-first surveys, shows what this looks like in practice: respondents answer in their own words, then validate each other’s responses rather than having them averaged by the system. What reaches the board isn’t a number — it’s what the workforce actually said. It requires a different kind of question at board level, and a people function equipped and willing to answer it.

The score will always be easier to present than the understanding. The question is whether the board is prepared to settle for it.

My opinion

I left an organisation once because the leadership had changed five times in five years and no one seemed to know what direction they were heading in. When I left, no one asked why. The exit process was procedural. I was not a data point anyone was going to interrogate. In another business I left because of a executive level change which resulted in an organisational restructure, after the very first meeting with the new executive I knew what was coming and it wasn’t going to be pretty. The thing which I loved about that business was the one thing that was going to be destroyed - Culture. So I resigned, said my piece and everything I said came true.

The people who collect this kind of information are usually not hiding anything deliberately. They are responding to incentives. They have learned, over time, what the board finds useful and what it finds uncomfortable. They present the former. They convert the latter into a score.

That is not intelligence. It is data dressed as reassurance. And if a board cannot hear the difficult alongside the comfortable — if the culture in the boardroom makes hard truths unwelcome — then the problem is not the measurement. It is a leadership failure at the top of the organisation.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam has been reading the same paragraph for the third time. The words are there. The comprehension isn’t. Something, unclear what, has been pulling at the edge of attention for the last twenty minutes without ever resolving.

The message arrives that Sam isn’t going to read yet. The alert that Sam already half-knows exists. Some unfinished conversation sitting just below the surface of thought, unanswered, unresolved, quietly taking up space in the mind that was supposedly committed to this page.

Sam isn’t distracted. Sam is present. Physically here, eyes on the text, doing the right thing. The presence is real. The attention is somewhere else entirely.

The attention you were certain you had

In 1999, two psychologists at Harvard asked participants to watch a short video and count how many times a basketball was passed between players wearing white. The task was specific and demanding.

Halfway through the video, a person in a full gorilla suit walked into the frame. Faced the camera. Beat their chest. Walked out. Fifty per cent of participants never saw it. Not because it was subtle. Because their attention was committed elsewhere, and the gorilla wasn’t part of what they’d been asked to track.

We believe we’re present because we’re physically in a situation. The eyes are open. The body is there. But presence and attention are not the same thing, and the gap between them is where a great deal of modern life is quietly disappearing.

Inattentional blindness — the brain’s tendency to miss even obvious stimuli when focus is elsewhere — isn’t a rare laboratory phenomenon. It’s the operating condition of most people’s daily lives.

The brain doesn’t see everything in front of it. It samples, filters, and constructs a picture based on what it has been primed to look for. When that priming mechanism is overwhelmed — when there are too many things competing for focus — the filter becomes less a tool of selection and more a tool of exclusion.

What has changed since the gorilla experiment isn’t the cognitive architecture. It’s the scale and sophistication of the systems competing to own the priming. Every notification, every feed, every platform is fighting to be the thing your brain gets told to track. When they win, everything else becomes the gorilla. Invisible. Present. Missed.

We don’t lose the ability to pay attention. We lose the choice about where it goes.

The things that disappear without announcement

The gorilla in modern life isn’t a person in a suit. It’s the conversation not had. The warning not noticed. The colleague who needed support and didn’t get it because you were technically in the room while mentally somewhere else entirely.

It’s the child who asked a question and got a distracted non-answer, then stopped asking. The friend who was struggling and mentioned it once, sideways, and you were there but not there when it came up. The decision made quickly because the depth of focus required to make it well wasn’t available that day.

The cost isn’t always visible. It rarely arrives labelled. It accumulates in the quality of what you produce, the depth of the relationships you maintain, and the gradual narrowing of what you actually notice about the people and moments that are directly in front of you.

We have been taught to understand this as a personal failing. A concentration problem. A discipline issue. The technology industry is clear, in its habits if not its language, that this framing suits it perfectly. Because if the problem is you, you don’t look at the design.

What the design is actually doing

A system that competes for the priming of your attention, that inserts itself into the filtering mechanism the brain relies on to distinguish signal from noise, is not supporting you. It is working against the very cognitive function it claims to support.

The argument that social platforms augment connection holds only if the connections they produce are of comparable quality to those they replace or displace. The gorilla experiment suggests a direct relationship: the more demanding the competing task, the more complete the blindness. Connection that comes at the cost of presence isn’t connection. It’s substitution.

The deepest failure is this: any system that makes you selectively blind to your immediate environment cannot claim to respond to your actual circumstances. The person sitting with their child. The person driving. The person in the conversation that matters. A system that doesn’t know — or doesn’t care — which of those moments it’s interrupting has no basis for calling itself designed for you.

This is one of the questions at the centre of my book The Cognitive Crucible — not just what technology does, but what it costs us without announcing the price.

My opinion

Attention is the most finite resource we have. I don’t think the platforms taking it are confused about what they’re doing — the design is not accidental, and the framing of distraction as a personal discipline problem is not a misunderstanding either. What concerns me most is the picture of society these platforms have constructed: not a reflection of how people actually live, but of what drives engagement. The two are not the same thing, and recovering the ability to tell them apart — to notice what is actually in front of you rather than what you’ve been primed to track — is, in the most direct terms I can offer, how you get your cognitive agency back.

What this asks of you

The point is not to understand inattentional blindness. The point is to know where it is operating in your own life.

1. What have you noticed yourself missing lately — at work, at home, in conversation — that you’d have caught if you weren’t distracted?

2. Is there someone in your life who has been trying to tell you something you haven’t fully heard? What would it take to change that?

3. What would you need to change about your environment to make genuine presence your default rather than your exception?

4. If someone was watching how you spend your attention across a typical week, what would they conclude about what matters most to you?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

There is a particular kind of silence that follows a consultation. Not the silence of consideration, where something heard is being weighed. The silence of completion, where a process has been concluded and the findings have been filed. The residents submitted their responses. The staff completed the survey. The service users attended the focus group. And then, at some point later — sometimes months, sometimes longer — a decision was announced that bore little visible relationship to what anyone said.

Most people who have been consulted by a large organisation have experienced this. Many have stopped expecting otherwise. That learned scepticism is not apathy. It is a rational response to a pattern that repeats across sectors with enough consistency to suggest it is not accidental.

The pattern has acquired a name — consultation fatigue — but the name understates what is actually happening. Fatigue implies a willingness that has been worn down. What most organisations have produced is something more deliberate: a process designed to look like listening without requiring the organisation to change course as a result of what it hears.

The mechanics of being heard without being heard

The structure of most consultations gives the game away.

Questions are written by the organisation, which means the answer categories reflect the organisation’s existing understanding of the problem. Respondents are asked to rate, rank, or select from options that were designed before anyone outside the room was consulted. Free-text fields exist, but are rarely analysed systematically — they are too difficult to process at scale and too unpredictable in what they might surface. The consultation closes, the quantitative results are tabulated, and the report that emerges reflects the organisation’s original position with a margin of adjustment around the edges.

This is not cynicism dressed as analysis. It is the logical consequence of designing a consultation process around the outputs you need to produce rather than the understanding you need to reach. In local government, that output is a mandate — a defensible record that affected parties were involved before a decision was made. In the NHS, it is a compliance requirement — evidence of patient and public involvement in service design. In the private sector, it is an engagement score — something to report to the board or include in a regulatory submission. Each has a legitimate purpose. None of them requires the organisation to be genuinely changed by what it hears.

The people being consulted know this. The communities most affected by decisions are typically the ones with the least trust in the processes designed to involve them — because they have the most experience of those processes. When a community has watched a planning consultation produce a predetermined outcome, or seen a workforce survey followed by a management communications campaign rather than structural change, the rational response is to disengage. Not because the issues don’t matter to them. Because they have learned that engaging produces no different outcome than not engaging.

What the same pattern looks like across sectors

In local government, the planning consultation is the clearest case. Residents are invited to comment on a proposal that is already substantially formed. The statutory requirement is to consult; there is no equivalent requirement to be materially influenced by what is said. Communities that oppose a development learn to flood the process with responses because volume is the only variable that demonstrably affects outcomes — not the quality of the argument, not the specificity of the concern, not the evidence that the proposal conflicts with the community’s stated needs.

In healthcare, service redesign consultations run on similar logic. A trust deciding to reconfigure services will conduct patient engagement as part of the process. The engagement is genuine in the sense that it happens; it is limited in the sense that the financial and operational case for the reconfiguration is typically established before the engagement begins. What the consultation can change is the detail. What it rarely changes is the direction.

In the private sector, the employee consultation — particularly in a restructure or acquisition — operates under legal requirements in the UK that mandate a period of consultation before certain decisions are implemented. The requirement is to consult; it is not to reach agreement, and it is not to change course if the arguments against the proposal are sound. HR teams are well practised in running consultations that satisfy the legal threshold while protecting the operational decision already made.

In the charity sector, the gap is perhaps most uncomfortable to acknowledge. Organisations built on the principle of community voice frequently struggle to implement it in practice. Governance structures centre the board and the executive. Impact measurement centres the funder’s reporting requirements. The beneficiary’s account of their own experience sits at the end of a chain of representation — reported by a caseworker, summarised by a manager, condensed into an impact metric by a communications team — that has progressively removed the original voice by the time it reaches anyone with decision-making authority.

The thing that peer validation changes

The problem is not only that organisations fail to act on what they hear. It is that the processes they use mean they often do not hear clearly in the first place.

A consultation that asks closed questions gets closed answers. A free-text field that sits at the end of a fifteen-minute survey gets responses from the minority who feel strongly enough to write something additional. An engagement that channels responses through a manager or a facilitator introduces the interpretation of the intermediary before the data reaches analysis. By the time findings are presented to the people who will make the decision, the original voice has been translated several times.

The methodological shift that matters here is not just collecting open responses — it is validating them within the community before they reach the analyst. When people can see each other’s responses and indicate what resonates, two things happen that closed surveys cannot produce. First, the synthesis reflects the community’s own sense of what matters, rather than the researcher’s coding scheme. Second, people who contribute to a process and see their contribution acknowledged by peers are less likely to conclude that the exercise was performative. The experience of being heard is partly structural — it depends on whether the process was designed to hear or designed to record.

Gobby is one tool built on this logic — open responses rather than closed questions, peer-powered voting so participants can indicate what resonates, AI-assisted synthesis that reflects the community’s own sense of priority rather than the analyst’s coding scheme. It is still used by a minority of organisations. The majority continue to run processes that produce defensible records rather than genuine intelligence. The technology to do otherwise is not the constraint. The will to act on what might be heard is.

That is the harder problem, and no tool solves it. What better processes do is remove the organisation’s ability to claim it didn’t know. If the findings are specific, community-validated, and clearly stated, the decision to set them aside becomes a conscious choice rather than an administrative oversight. That is a different kind of accountability — and it is the reason some organisations prefer the consultation that changes nothing.

My opinion

I have been in enough of these rooms to know the difference between an organisation that doesn’t know how to listen and one that has decided not to. Most consultation theatre is the second kind. Tools like Gobby do something useful: they remove the claim of ignorance. If the findings are specific and community-validated, the decision to set them aside is no longer an oversight. It is a choice. And the organisations that consistently make it know exactly what they are doing.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan comes on shift at 11pm and the menu is gone. Not crashed — gone. The update that rolled out at 6pm moved it somewhere else, and the section of iOPS that handles electronic file transfer to the Crown Prosecution Service is no longer where it was yesterday. There is a person in the custody suite waiting to be processed. Morgan has a legal time limit. Morgan picks up the paper form.

This is Greater Manchester Police in July 2019. iOPS — the Integrated Operational Policing System — went live sixteen months behind schedule. The custody module, which was supposed to manage custody records and transfer files electronically to the CPS, was not working as specified. The announcement had said it was ready. The go-live had happened. Nobody said out loud that it wasn’t done.

It wasn’t done.

Deployed, not delivered

The workaround that custody sergeants had been using for three months no longer worked. The update had changed the interface without adequate retraining. There was no mechanism to refuse the tool — the option was to make it work, improvise a new workaround, or escalate a complaint that had no clear landing point. Officers improvised.

GMP’s own independent review named the consequences: technical shortcomings, inadequate training, and operational disruption that left vulnerable people without referrals to support services. Referrals for high-risk domestic abuse victims fell by 50%. Referrals to Greater Manchester Victims’ Services fell by 87%. Officers spent time running duplicate paper processes alongside the digital system that was supposed to replace them. The system did not help the workforce do the work — it created a second job beside it.

iOPS had gone live not because it was ready but because the organisation needed it to have gone live. Sixteen months of delay had a contractual cost, a political cost, and a considerable cost in credibility. Ready was redefined as deployed, and the workforce inherited the gap between the two.

Why no one says it isn’t ready

A software product that is finished cannot justify an ongoing roadmap. It cannot retain an engineering team. In a subscription market, completion is a liability — a done product is one that doesn’t need what comes in the next release cycle. Agile methodology and continuous deployment gave the industry a vocabulary for this: not unfinished, but iterating. Not incomplete, but evolving. The language made incompleteness sound like a sign of rigour.

Organisations accepted this because the alternative was more expensive. To say the system was not ready meant saying the procurement decision was wrong. GMP’s contract had already been signed, extended, and renegotiated. Admitting the custody module was unfit meant unpicking an obligation that was difficult to unpick. Deploying and adapting was easier.

There was no real mechanism to refuse. The tool was the tool. The workarounds became part of the job. The cost was absorbed shift by shift by custody sergeants who had no power to put it back where it came from.

The cost that didn’t appear in the contract

Birmingham City Council signed a contract for an Oracle ERP system projected at £19 million. By 2026 the total cost is forecast to reach £216.5 million. Two and a half years after go-live, the system was still not, in the auditors’ own words, safe and compliant. In September 2023 Birmingham declared bankruptcy — the largest local authority insolvency in UK history. The system was never ready either.

NHS IT outages disrupted more than 274,000 patient interactions across six major trusts in a single year. In each of those incidents, clinical staff reverted to paper. Not because paper was preferred, but because the digital system had failed and the paper process was the only thing keeping the work going. More than a quarter of UK central government digital systems are now classified as outdated, at an estimated cost to the public sector of £45 billion annually in foregone productivity.

Office workers endure an average of 3.6 technology interruptions a month — software updates, system failures, workarounds required. Nobody counted this cost when the contracts were agreed, because the cost landed on the worker. The vendor’s contract was satisfied. The product had been delivered. That it didn’t work as specified was the next problem, billable separately.

What it was designed for

The most basic test for any system deployed to workers is whether it actually helps them do the specific work it replaced. iOPS failed that test on the night the update moved the workflow officers had learned. The tool stopped being a tool and became an obstacle with extra steps. The work still needed doing. Staff found a way to do it. The system got the credit for a deployment that happened.

The next test is whether what the vendor calls an improvement actually improves anything for the person using it. iOPS received updates. Each notification used the word “improvement.” Custody sergeants had not requested the features those updates added. The gap between what the vendor was building and what the workforce needed is not a failure of communication. It is the expected result of a development roadmap driven by the vendor’s commercial cycle, not by the legal obligations of a sergeant at 11pm.

The final test is whether the system responds to the specific person in front of it, or expects the person to fit. iOPS updated on the vendor’s schedule. Its release windows did not account for shift patterns, operational hours, or the consequences of breaking a workaround mid-cycle. The specific circumstances of night-shift custody work — legal time limits, vulnerable people waiting — were not part of the specification for when an update went live. The system was not designed around the people using it. They were expected to fit.

My Opinion

I have been in enough procurement processes to know that the people who will actually use the system are rarely in the room when it is chosen. Requirements are gathered, but they are normally vetted by people who do not work with the system day to day — so what gets built reflects what the organisation said it needed, not what the work actually demands. Sometimes a system is deployed not because it is ready but to satisfy a contract: the way a UK train leaves a platform on time even when it isn’t. What is missing is oversight with teeth — people who have enough operational experience to know when something is wrong, and enough authority to stop it going live until it is right.

I’ve seen this done properly where in a hospital the IT Development team deployed a capability that clinical staff had been asking for. Then they got smart - they instigated a digital suggestions box for new features/services/products and the hospital staff embraced this as they got a voice on what was needed and important - not what the organisation thought it needed. The even smarter change was the IT Dev team wrapped this all with an agreed SLA for delivering selected capabilities and they stuck to it. What the users asked for they got when it was promised. The downside if it can be called one was the sheer volume of requests once staff saw it working.

Questions the contract never asked

The following questions are not academic. They are for anyone currently running a process that exists only because a system the organisation bought was never quite finished. The vendor moved on. The contract was satisfied. The workaround remained.

1. What workaround are you currently maintaining that exists only because a system your organisation bought was never quite finished?

2. If you had had to sign off on the cost of that workaround — in time, in error rate, in staff goodwill — when the contract was agreed, would the vendor still have won it?

3. When was the last time someone in your organisation said “this isn’t ready” and it made a difference?

4. What would it take to put “not ready yet” back into the vocabulary of public sector procurement?

The cost of perpetual beta is not paid by the organisations that sign the contracts or the vendors that provide them. It is paid by the person on the shift who picks up the paper form when the menu has moved again. That cost is real. It has just never appeared in a budget line.

Leave a comment

Sources & References

• HMICFRS: Greater Manchester Police Integrated Operational Policing System Inspection, 2020. https://hmicfrs.justiceinspectorates.gov.uk/publications/greater-manchester-police-integrated-operational-policing-system-inspection/

• Computer Weekly: “Police IT system failure creates significant backlogs”, 2020. https://www.computerweekly.com/news/252479783/Police-IT-system-failure-creates-significant-backlogs

• Data Centre Dynamics: “Total cost of Birmingham City’s Oracle system failure to reach £216.5m by 2026”, 2024. https://www.datacenterdynamics.com/en/news/total-cost-of-birmingham-citys-oracle-system-failure-to-reach-2165m-by-2026-report/

• The Register: “Birmingham City’s Oracle ERP system still not ‘legally safe’”, April 2024. https://www.theregister.com/2024/04/17/birmingham_city_councils_oracle_erp/

• The Register: “Birmingham City Council goes under after Oracle disaster”, September 2023. https://www.theregister.com/2023/09/05/birmingham_city_council_oracle/

• Security Brief UK: “NHS IT outages disrupt 274,620 patient interactions”. https://securitybrief.co.uk/story/nhs-it-outages-disrupt-274-620-patient-interactions

• Tech Monitor / DSIT: “Legacy technology costs UK public sector £45bn annually”. https://www.techmonitor.ai/government-computing/legacy-technology-costs-uk-public-sector-45bn-annually/

• Ivanti: 2025 Digital Employee Experience (DEX) Report, 2025. https://www.ivanti.com/company/press-releases/2025/tech-disruptions-cost-companies-millions-of-dollars-in-lost-productivity-annually-according-to-research-from-ivanti

  ---

  You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

An experienced endoscopist spends years learning to see. The skill is not in holding the camera. It is in the eye: the trained attention that catches a flat, easily missed growth at the edge of the screen, the kind that turns into bowel cancer if it is not found and cut out. That attention is built by doing the looking, thousands of times, with no one and nothing pointing the way.

Between 2021 and 2022, four endoscopy centres in Poland brought in an AI tool that draws a box around suspect growths as the camera moves. It works. Switch it on and detection goes up. But researchers also watched what happened to the same doctors when the tool was switched off again. In the months after routine AI use came in, their unaided detection rate fell from 28.4 per cent to 22.4 per cent — a fifth of their skill gone, on the procedures where they were once again looking alone. The Lancet Gastroenterology & Hepatology published the finding in August 2025. The doctors had each performed more than two thousand colonoscopies. They were not novices losing a skill they had half-learned. They were experts, and the looking had quietly stopped being theirs.

No one decided that those doctors should get worse at finding cancer. Each centre adopted the tool for an honest reason: on the day, with the box on the screen, more growths are caught. The erosion was the by-product. It arrived without a meeting and without a vote, and the act that had kept the eye sharp — the looking itself — was the very thing the tool took over.

This piece grew out of a conversation. A few weeks ago Jason reviewed Nick Bostrom’s Deep Utopia for his Book Club.

Jason B Perkins

Deep Utopia: The Automation Paradox

What if the future arrives exactly as the technologists promised? Not a collapse. Not a machine uprising. Not even a slow, disappointing muddle. What if artificial intelligence keeps improving, systems keep coordinating, science keeps accelerating, and the world becomes astonishingly good at solving problems? Disease recedes. Scarcity shrinks. Logistics…

Read more

2 months ago · 3 likes · 3 comments · Jason B Perkins

I left a comment that disagreed with the end of his argument rather than the start of it. We have been chatting it over since. What follows is both sides, written honestly and left unresolved on purpose.

We agree on Bostrom’s distinction. Some of what we do has instrumental value: we do it to reach a result. Some has intrinsic value: the doing is the point. As machines deliver the result faster, the instrumental reasons for human effort weaken, and what survives is the effort we would still choose when we no longer have to. The colonoscopy study makes me want to name a third category the distinction misses. Call it formative difficulty: effort whose result we want, and whose doing builds or sustains the capability to get that result at all. Jason reads Deep Utopia as a reason to protect that kind of difficulty on purpose. I think he is right, and I think the harder question is who gets to decide.

That is the architecture answer, and it holds. But the colonoscopy centres were not careless. They adopted the tool for a real, same-day gain in detection, and the erosion still happened. So the question I could not let go of in Jason’s comments is who decides which difficulty is worth keeping, and on what grounds. The decision is already being made, almost always by default. No one in those rooms asked whether the unaided looking was instrumental or formative, because that is not a question a business case is built to ask. The answer arrived as a tool, and a capability that took two thousand procedures to build began to fade in months.

Bostrom says the scarce goods of an abundant world will be meaning, agency, and the texture of things done by hand. That sounds like liberation. Look again and it can describe a market, because someone always sets the price of a scarce good, and someone is always priced out. We have watched smaller versions of this. The handmade thing does not vanish when the machine arrives; it moves upmarket and becomes how the wealthy signal rather than how ordinary people live. There is no law that protects formative effort from the same gravity. The risk is not that the hard parts disappear. It is that they are kept by whoever can still afford to do things the slow way, and lost by everyone measured on speed.

If we are going to defend any difficulty, the test has to be sharper than nostalgia, because most appeals to the value of struggle are sentimentality about how things used to be done. Plenty of difficulty is worth removing. No one should defend long division by hand, or a three-week wait for a letter, or a disease we can now cure. The difficulty worth keeping is the narrow kind that builds or sustains something in the person doing it: skill, judgement, identity, or a relationship that would not otherwise exist. The endoscopists’ looking sustained an eye for cancer. The tool kept the detection and let the eye go, on the quiet assumption that the eye was a by-product rather than the point.

Jason reaches the same line from the other direction, and calls it virtuous and vicious friction. Vicious friction only diminishes us: the pointless meeting, the layer of bureaucracy, the months spent waiting for access. Virtuous friction defines us: the apprenticeship that builds judgement, the demanding conversation that builds trust, the hard way that leads to mastery. He started from the architecture and I started from the person, and we arrived at the same cut. Not effort against ease. Formative difficulty against incidental difficulty. The trouble is that the two are indistinguishable on a dashboard, where both show up only as time the machine could save, and the people best placed to tell them apart — the doctors who can feel their own eye dulling — are rarely the ones holding the budget or writing the rollout plan.

None of this is an argument against the tools. Those doctors should have the AI; on the day, it finds more cancers, and that is not a small thing to set aside. The argument is that someone has to be watching the difference between the difficulty that was keeping a skill alive and the difficulty that was only costing time, and at the moment almost no one is. The Polish researchers were careful: their study was observational, and they could not yet say what happens to doctors who train with the box from the start and never build the unaided eye at all. That is the more troubling question, and it is still open.

We will not lose the hard parts in a single decision anyone could be held to. We will lose them one defensible efficiency at a time, each removal sensible on its own, until a generation arrives that can check the machine’s work but could never have done it alone. If meaning, judgement, and skill really are the scarce goods of the world we are building, the choice in front of every organisation is not whether to keep some difficulty. It is who gets to decide which difficulty, and whether the person it would have formed is ever in the room when the decision is made.

Over to you. Before you automate the next hard thing, Jason offers five questions worth holding it against:

1. Agency — does it strengthen human control, or quietly remove it?

2. Connection — does it deepen trust and human bonds?

3. Judgement — does it sharpen our ability to decide, or outsource it?

4. Opportunity — does it widen what people can collectively become?

5. Safety — can the outcome be audited or reversed?

So we will put the same question to both our readerships and answer it in the thread: which difficulty in your own work is keeping a skill alive, and which is just slow?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Somewhere in most organisations there is a folder, a shared drive, or an archived system containing several years of things people said. Exit interview notes. Customer complaints in their original form, before they were categorised and counted. Free-text responses from the bottom of annual surveys. Messages sent to a service team that were resolved, closed, and filed. Forum posts from an internal platform that was replaced two years ago and whose contents were migrated to storage rather than deleted.

None of it has been read systematically. Most of it never will be.

This is not a data problem. The data exists. It is a processing problem — or rather, it was. For most of the period in which this material was accumulating, there was no practical way to analyse it at scale without a significant investment in qualitative research that most organisations could not justify for data they had not planned to use. So it was treated as administrative residue: kept because deletion felt risky, ignored because analysis felt impossible.

That calculation no longer holds.

The signal that was never treated as data

The gap between what organisations collect and what they analyse has always been widest in the qualitative register. Numbers get processed. Text gets filed.

This is partly a legacy of the tools available. Survey platforms were built to aggregate ratings, not to synthesise narratives. CRM systems were built to track transactions, not to find patterns in how customers described their experience. HR systems were built to record headcount and performance scores, not to make sense of what people wrote in the comments field at the bottom of the appraisal form. The architecture of most organisational data infrastructure assumes that what matters can be counted.

The consequence is that organisations have spent years collecting a secondary record of their own reality — the version in which people spoke in their own words, described their own experience, and articulated what the numbers could not carry — and treating it as noise.

Exit interviews are the clearest example. Most organisations conduct them. Most collect the responses in some form. Very few analyse them across a period long enough to identify patterns — which managers people were leaving, which structural conditions were driving attrition, which aspects of the culture were named repeatedly by people who had decided to leave but no longer had any reason to be diplomatic. The data exists. The analysis that would make it useful never happens, because exit interview findings are processed as individual cases rather than as a cumulative record.

Customer complaints follow the same logic. The categorised version — number of complaints by type, by channel, by product — reaches a dashboard. The original text, in which customers described in specific terms what had gone wrong and what it had cost them, is archived after the case is closed. Organisations that have been collecting complaints for five years have five years of precise, unsolicited qualitative description of the gap between what they promise and what they deliver. Almost none of them have read it.

What changes when the analysis becomes possible

The shift that is underway is not primarily about new data. It is about the retroactive value of data that already exists, and the prospective design of data collection with analysis in mind from the start.

AI-assisted qualitative analysis has made it practical to process large volumes of unstructured text in ways that were previously either manual and expensive, or automated and crude. The earlier generation of text analytics tools — keyword frequency, sentiment scoring — reduced qualitative data to a thin quantitative proxy. What is becoming possible now is closer to genuine synthesis: identifying themes, finding patterns, distinguishing between types of concern that look similar in aggregate but are structurally different in cause.

The organisations best placed to use this capability are not necessarily the ones with the most sophisticated data infrastructure. They are the ones that recognise they have been sitting on material with analytical value and begin treating it as such. A charity with three years of service user feedback in a case management system. A professional services firm whose client debrief notes have been saved but never reviewed across engagements. A healthcare trust whose patient letters — the ones written by people who felt strongly enough to put something in writing — are archived in a folder that no analyst has opened.

The design question for new data collection is equally important. When an organisation understands that qualitative responses have analytical value, it changes how it structures the collection. Open-response questions get asked more deliberately. Free-text fields get positioned earlier in a survey, not appended as an afterthought. Platforms are chosen with synthesis in mind rather than just with response rates in mind. Tools such as Gobby.io, which combines open response with peer-powered voting — where participants indicate which responses reflect their own experience — produce data that is both richer and more efficiently analysable than free text alone, because the participants have done part of the synthesis work before the data reaches an analyst.

What the evidence was saying all along

The most significant implication of this shift is not operational. It is about what organisations know about themselves, and what they have chosen not to know.

An organisation that has been collecting qualitative signal for years and never analysed it has, in practice, been making decisions without a significant part of the available evidence. That is not a neutral position. It means that the version of reality reaching leadership has been shaped by what the data infrastructure was designed to draw out — typically the quantitative, the recent, and the aggregated — and has excluded everything the data infrastructure was not built to carry.

When that material becomes analysable, organisations sometimes discover that the patterns were visible all along. The attrition problem that presented as a compensation issue in the survey data was described, repeatedly and specifically, in exit interview notes as a management culture problem in a particular division. The customer satisfaction decline that appeared as a score movement in quarterly reporting had been articulated, in precise terms, in complaint correspondence two years earlier. The service redesign that failed to achieve adoption had generated, during its consultation phase, a clear account of the operational reality it failed to account for — an account that was collected, filed, and never read.

None of this means that quantitative data is wrong or that qualitative analysis is always right. It means that organisations have been working with a partial picture and treating it as complete. The data to fill that picture, in many cases, already exists.

The question is whether the organisation is prepared to read it.

My Opinion

I have been through more exit interviews than I can count, and almost all follow the same pattern: questions designed to measure the process, not to capture what the person leaving actually thought. I have asked to read exit interviews from people who left teams I was responsible for and been met, more than once, with a blank look or a direct refusal. A genuine caveat applies: this data is shaped by why someone left, which means it needs to be read in context rather than taken at face value. But there is a difference between data that requires careful reading and data that is never read at all. The true voice of people is the thing most organisations consistently arrange not to hear.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan finds out on a Tuesday evening, scrolling through the news. Not from the company. Not from the bank. Not from anyone who thought they had a duty to say something. From a journalist who found out first.

The article is six paragraphs long. The breach happened seven months ago. Morgan’s name, date of birth, email address, and payment history have been sitting somewhere they were never supposed to be. The notification is still in a queue. The legal team is still reviewing.

By the time Morgan finishes reading, the account has been live in the wrong hands long enough to matter.

The system is working as designed

Companies are not legally required to disclose data breaches immediately. In the UK, under GDPR, the obligation to notify the Information Commissioner’s Office is seventy-two hours. Notification to individuals is required only when there is a likely high risk to their rights and freedoms — and determining that likelihood is left largely to the organisation doing the determining.

The gap between an event happening and a person learning about it is not a failure of the system. It is the system. Seven months is not unusual in documented cases. Twelve is not unheard of. The disclosure process is managed for legal exposure, reputation, and investor sentiment. The person whose data was taken is, structurally, the last consideration.

None of this requires a villain. The process is run by compliance teams doing their jobs inside a framework designed before the scale of digital exposure was fully understood. The problem is not malice. The problem is architecture.

Whose interests the framework serves

The disclosure framework was built around the assumption that a company’s primary obligation in a breach event is to contain it, assess it, and report it in good order. The individual whose data was taken is a downstream consideration — someone to be managed after the institutional response is complete.

This is an understandable way to design a legal framework. It is not an acceptable way to treat a person. The individual whose data was taken has no independent means of knowing what has happened. They cannot check. Cannot act. Cannot protect accounts, change passwords, alert a bank, or do any of the things that might have reduced the damage. The window in which those actions were possible opens and closes while the disclosure is being reviewed.

The data that was taken does not expire. An address does not change. A date of birth does not change. The value to a criminal does not diminish with time. The delay does not reduce the risk — it transfers it entirely to the person.

The company managed the crisis. Morgan just lived inside it.

What the delay actually costs

The visible cost — fraudulent transactions, identity theft, months of remediation — is measurable, if the person can prove causation in time. Most cannot. The link between a breach notification seven months late and the credit application rejected six weeks ago is not the kind of thing an individual can easily trace through a system designed to obscure its own connections.

The invisible cost is something else. Anyone affected comes to understand, in a way that cannot be undone, that there are institutions holding information about them that will manage a catastrophic failure without telling them until the legal minimum has been met. That the decision about when they deserved to know was made by people they have never met, in a process they had no part in, according to criteria they were never shown.

The relationship between a person and the organisations that hold their data is not a relationship between equals. It never was. But the disclosure framework makes the power difference formal, and applies it at the moment a person is most vulnerable.

The notification system, as it currently operates, serves the organisation managing the breach rather than the person affected by it. It assists the company to discharge its legal obligation at a time and in a manner that suits the company’s interests. The individual affected receives a letter when the process is complete, not when the information might have been useful.

A seven-month delay adds nothing to the individual’s position. The argument that thorough investigation improves the quality of notification is technically defensible. In practice, the person receiving the notification has no way to verify the investigation’s quality, no means to challenge the timeline, and no recourse if the delay increased their exposure.

Every person whose data is in a breach has a different threat profile — different accounts, different vulnerabilities, different ability to act quickly. A single delayed disclosure treats all of them identically, at the moment when individual difference matters most. The person who is digitally confident and financially resilient and the person who is neither receive exactly the same letter, at exactly the same time, containing exactly the same limited information.

My Opinion

The architecture of delayed disclosure is not the result of poor drafting. It reflects a consistent choice about whose interests the legal system was designed to protect. The notification framework was not designed and then found to serve institutions better than individuals. It was designed to serve institutions. The conversation about how to change it will not begin with regulators or compliance teams. It will begin when enough individuals decide that a letter arriving seven months late is not a notification. It is an alibi.

The questions no letter asks

If your data has been in a breach — and statistically, it has — the notification you received, if you received one at all, told you what the company was legally required to say. It did not ask what you think about the system that produced that notification. These are the questions the letter didn’t come with.

• Have you ever found out about a breach affecting you through a news story rather than a direct notification?

• If you ran a company and discovered a breach, what would stop you from telling your customers immediately?

• Should notification timelines be standardised and enforced — and if so, by whom?

• What would you want a company to say to you, and when, if your data had been compromised?

These questions sit at the centre of my book The Shadow System — specifically the sections on how companies manage reputational risk against actual risk in the aftermath of a breach.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Every year, organisations across every sector spend significant money finding out how engaged their people are. The answer comes back as a number. Sixty-two percent. Seventy-one. A red, an amber, a green. And then, in most cases, very little changes — because a number, however precisely measured, does not tell you what to do next.

This is not a failure of effort. It is a failure of method. The question being asked — how engaged are people, on a scale of one to ten — is structurally incapable of producing the intelligence needed to act. It tells you the temperature of the room. It does not tell you why the room is cold, which walls are letting in the draught, or whether the people in it have stopped expecting anyone to fix it.

The pattern holds across every sector. An NHS trust running annual staff surveys. A private equity-backed scale-up tracking employee sentiment through quarterly pulse checks. A charity trying to understand what its service users actually need from a redesigned programme. A number where there should be a conversation.

What the number misses

The case for quantitative engagement measurement is not wrong. Trends matter. Comparability matters. Knowing that satisfaction dropped twelve points between January and March, or that one team consistently scores lower than the rest of the business, is genuinely useful. The problem is not that organisations measure; it is that most treat the measurement as the understanding, rather than as a signal that understanding is needed.

Qualitative intelligence — what people actually say, in their own words, about their own experience — produces something different. It surfaces the specific, the unexpected, and the things no survey designer thought to ask about. It also surfaces disagreement within a group that aggregate scoring conceals. A team with an average engagement score of sixty-eight might contain people who are deeply committed and people who are quietly preparing to leave. The number averages them into false coherence.

The difficulty has always been scale. Reading and coding open-text responses from three hundred employees, or three thousand service users, is expensive and slow. The analysis introduces its own biases. And the time between collecting the data and acting on it is long enough that the context has already shifted. So organisations default to the form that is easiest to process — the number — and accept its limitations as the cost of manageability.

That trade-off no longer holds as firmly as it did. The combination of open-response collection, peer voting, and AI-assisted analysis — the approach taken by tools like Gobby, which works in this space — means that qualitative data can now be gathered and synthesised at a speed and scale that was not previously practical. People respond in their own words. Those responses are surfaced to peers, who vote on what resonates. The result is not just a fuller picture; it is a picture that has been validated by the community itself, not just interpreted by the analyst.

The pattern across sectors

In the public sector, the gap between what is measured and what matters is often widest. Patient experience surveys ask whether care was clean, timely, and respectful — categories defined by the regulator, not the patient. What they rarely capture is whether the patient felt heard, whether the treatment made sense in the context of their actual life, or whether the system’s definition of a good outcome matched their own. The number that comes back is real. The experience it describes is partial.

Local government consultation has the same problem at a different scale. A planning consultation that asks residents to rate a proposal on a five-point scale produces a mandate for or against. It does not produce understanding of why, which aspects are acceptable and which are not, or what a modified proposal might need to look like to bring the sceptics with it. The binary produces a decision. It does not produce the intelligence that would make the decision stick.

In the private sector, the equivalent is the employee engagement survey and its close relative, the customer satisfaction score. Both have become so ritualised that the measurement has partially replaced the thing it is supposed to measure. Teams know how to answer the survey. Customers know that the score goes nowhere. The process continues because it produces a defensible number, not because it produces understanding. When engagement drops, the standard response is a communications campaign — because the survey told you sentiment, not cause.

The charity and social enterprise sector is arguably most exposed. Organisations whose entire purpose is to serve and represent specific communities often have the least developed means of understanding whether they are doing so. Impact reporting is dominated by output metrics — number of people served, sessions delivered, referrals made. What is missing is the voice of the person at the end of the service, speaking in their own terms about whether it changed anything for them. Not a satisfaction score. A considered account of what happened.

Understanding you can act on

The value of more detailed engagement data is not primarily analytical. It is operational. When the reasons behind a trend are visible — not inferred, but stated by the people who are living them — the path to action becomes clearer and the likelihood of getting it wrong decreases.

A portfolio company workforce that scores low on engagement is a problem. A portfolio company workforce that is able to articulate, collectively and specifically, that the integration plan removed the three things that made the culture work is a solvable problem — or at least one where the board can make an informed decision about the trade-off. A service user group that can say, in their own words and with their peers’ endorsement, that the new referral pathway is harder to navigate than the one it replaced gives a programme team something to act on rather than something to commission further research about.

My Opinion

Every organisation I’ve worked with runs some version of the same survey — historical questions, coded to a scale, returned as a number. I’ve seen what happens when you replace that with something closer to in-the-moment: a city authority that put real-time word clouds — drawn from live citizen responses — on a screen in the corridor where the chief executive walked every morning. The conversation in that building changed. Not because the data was new, but because it was immediate, in people’s own words, and impossible to walk past. That is the difference between recording sentiment and understanding it.

The shift required is not primarily technological. The tools exist. What is required is a change in what organisations believe engagement measurement is for. If the purpose is compliance — to show that a survey was conducted and a score was recorded — then the number is sufficient. If the purpose is understanding — to know what is actually happening and why, well enough to do something about it — then the number is a starting point, not an answer.

The organisations that close that gap are not the ones that find a better survey platform. They are the ones that decide the distinction matters.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Alex had been on the waiting list for eighteen months before the letter arrived. It said that the authority’s housing needs assessment had been completed and that, based on the information held, Alex’s application had been assigned a priority band. Band C. The letter did not explain what Band C meant in practice. It did not describe how the assessment had reached that conclusion. It mentioned, in the third paragraph, that a decision-support system had been used to process applications, and that appeals could be submitted in writing within twenty-one days.

Alex submitted an appeal. The appeal was considered. The band was confirmed.

At no point in this process did Alex learn what information the system had used to arrive at its assessment, what weight it had placed on different factors, or why Alex’s particular circumstances — a care dependency, a fixed income, a medical condition that made the current address untenable — had resulted in Band C rather than Band B. The letter existed. The process had been followed. The appeal had been considered. The outcome held.

Somewhere in that housing authority, a board had approved the system. The paper had been well-prepared. The numbers were credible — the system processed applications in a fraction of the previous time, reduced inconsistency across manual assessors, and met the procurement criteria the implementation team had set out. The board had asked about cost, about timeline, about how the system would handle exceptions. It had asked about data retention and compliance obligations. Nobody had asked what Band C would mean to Alex.

The gap was not in the preparation

The people in that boardroom were not negligent. They were competent, experienced, and attentive. The chair ran a disciplined meeting. The questions raised were reasonable given the information in front of the room. And the information in front of the room had been assembled by people who understood the system — who had selected it carefully, procured it properly, and built a reasonable implementation plan.

The gap was not in the preparation. It was in the scrutiny.

To ask what Band C would mean to the people who received it required knowing enough about how the system worked to understand that the band was not an administrative category but an algorithmic output — shaped by weighted factors, training data, and configuration decisions made during implementation. To ask whether the appeal was a real avenue of recourse required understanding that an appeal reviewed against the same criteria that produced the original decision is structurally different from an appeal reviewed by an independent assessor with full information. To ask whether the failure modes had been examined required knowing that automated decision systems fail differently from human assessors — not randomly, but systematically, concentrated in the people whose circumstances differ most from the data the system was trained on.

Those questions were not asked. Not because they were unanswerable. Because they were not visible to the people in that room.

The risk that doesn’t show up in a financial model

Governance models were designed for a world in which the primary risks of a large technology procurement were financial and operational. Could the organisation afford it? Would it be delivered on time? Would it integrate with existing infrastructure? Did the vendor have a track record? These are legible questions for a board with deep expertise in finance, law, and organisational management. The methods for evaluating them are established. The signals of risk are familiar.

AI and data-driven decision systems carry a different kind of risk. The failure is not that the system goes over budget, though it may. The failure is that the system makes consequential decisions about real people based on patterns extracted from historical data — and if that data reflects the inequities and assumptions of the processes it was trained on, the system reproduces those inequities at scale, faster, with the additional authority of technological neutrality. That failure does not appear in a financial model. It does not surface in a project risk register. It does not emerge in a compliance review that covers data retention and lawful processing but not the distributional consequences of the decisions being made.

It surfaces in a letter.

The boards approving these systems are not, in the main, composed of people with the literacy to see that risk. That is not a criticism of the individuals — it is a description of how board composition evolved in response to the risks governance processes were built to manage. For most of the history of organisational governance, technology decisions whose consequences were primarily operational could be evaluated through financial and legal lenses without significant loss. That is no longer reliably true.

The structure of most board meetings compounds the problem. Papers are summarised to one page. Technical detail is compressed into reassurance. The people with the deepest knowledge of the system are usually the people most invested in its approval. Independent scrutiny — not from the vendor, not from the implementation team, but from someone whose only brief is to look for what could go wrong — is not a standard feature of most technology governance papers. It is an occasional addition, when the stakes are visible enough to prompt it. The stakes were not visible. They were in Band C.

In the EU, the AI Act — in force since August 2024 — classifies AI systems used to assess housing eligibility as high-risk, requiring transparency, human oversight, and challengeable outputs from August 2026. In the UK, the government’s Algorithmic Transparency Recording Standard is mandatory for central government departments; a bill that would have extended comparable requirements to local housing authorities stalled in 2024 without progressing. For the authority that assessed Alex’s application, and thousands like it, no such requirement currently applies.

Until they do, the room that approves the system and the person who receives its output will remain two entirely separate worlds.

The accountability that disperses

When an AI-assisted decision system produces a wrong outcome, the accountability for it is distributed across enough points in the governance chain — the vendor, the implementation team, the commissioning officers, the board itself — that it disperses without fixing to any of them. The appeal process, if it reviews the original decision against the same criteria that produced it, is not a genuine avenue of recourse. It is a confirmation mechanism.

Alex’s appeal eventually succeeded — Band B assigned, not because the system was reviewed, but because one housing officer with the right information made a different call. The system continued processing the next two thousand applications unchanged.

This is the pattern, not the exception. An authority reviewing its own AI deployment will see a system operating as designed — applications processed, appeals handled, no formal complaints logged against the technology. The harm does not show up in a post-implementation review. It accumulates in the cases where the right advocate was not present, where the first appeal was the last one, and where the person whose circumstances fell outside the training data had no way of knowing that fact.

A revised band and a revised system are two very different outcomes that look, from inside the authority, like the same one.

Three questions that change what gets asked before approval

The questions that were not asked in that boardroom are not technically difficult. They do not require a specialist in machine learning or algorithmic auditing. They require a different starting point — one that begins with the person at the end of the system rather than the system itself.

Before approving a technology that will make consequential decisions about people, a governance body should be able to answer three things.

The first is whether this technology actually helps the people it processes — not faster, but better. Does it make the experience easier for the person submitting an application under stress, with limited time, with a care dependency that doesn’t fit neatly into any field? The paper that came to that board had been prepared by the people implementing the system. The people at the end of it had not been consulted. Whether that consultation is required — as a governance condition, not a supplementary appendix — is a governance choice.

The second is whether the technology adds something real, and to whom. A system that processes applications faster is not automatically one that serves applicants better. It may serve the organisation better — reduced staffing costs, greater throughput, more consistent output — while producing outcomes that are worse for the specific people the organisation exists to help. The board had asked about processing time. It had not asked what the improvement looked like from the side of the person submitting the application.

The third is whether the technology can respond to the individual circumstances of the people it affects — and what happens to the people whose circumstances fall outside the parameters it was trained on. That question, asked before approval, changes the procurement criteria. It requires the vendor to demonstrate that edge cases are handled with appropriate human judgement, not simply processed by the same logic as every other case. It makes an independent appeal process a governance condition of deployment rather than a feature added after complaints arise.

A board that cannot answer those three questions before approving a system has not governed the technology. It has approved a budget line. The governance is what happens in between.

My Opinion

Most organisations are not prepared for what is coming. If they treat AI governance as a variant of the regulatory models they already have — compliant, procedural, box-ticking — it will fail. We have not yet reached agreement on what AI actually is, which makes governing it a peculiar challenge: the responsibility sits with people who cannot reliably describe the thing they are responsible for. The relevant legislation is also not contained within a single jurisdiction — a service consumed by a user in another country may bring obligations the commissioning authority has never considered. That is the gap. It is wider than most boards currently know.

Before the next approval

• Could the people who approved a significant technology decision at the senior level of an organisation you know describe, in plain language, what it does and what its consequences are for the people at the end of it? If not — what does that say about the quality of the oversight it received?

• If you sit on a board or senior governance body, what is your personal level of literacy in the specific technologies your organisation uses or is considering? Is that literacy sufficient to probe the assumptions in the papers you are asked to approve — or are you relying on the competence of the people who prepared them?

• If you are a technology leader who presents to boards, what is your responsibility to ensure genuine understanding rather than informed consent? Are you building presentations that equip boards to scrutinise, or ones that build confidence in a decision already made?

• And the hardest one: is it possible to be accountable for a decision you did not understand? If accountability requires understanding, and the understanding was not present in the room, where does the accountability for what happened to Alex actually sit?

The board never knew Alex’s name. That is, in the end, the clearest description of the problem.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

By the time the information about what is actually happening in an organisation reaches the people who make decisions about it, it has usually been through several hands. A front-line worker describes something to a team leader. The team leader includes a version of it in a report to a manager. The manager summarises the pattern to a director. The director presents a slide to the board. At each stage, something is selected, something is compressed, and something is lost. The decision that eventually gets made rests on a version of the original experience that none of the people who had that experience would fully recognise.

This is not a management failure. It is a structural one. Every organisation beyond a certain size faces the same problem: the people with the most direct knowledge of what is working and what is not are not the people with the authority to change it. Connecting those two groups requires a chain of translation. And translation, by its nature, introduces the translator.

The question is not whether this happens. It does, in every sector, at every scale. The question is how much gets lost, and whether the organisations that depend on that intelligence have understood what the loss is costing them.

What the translation removes

The first thing that translation removes is specificity. A front-line nurse can describe exactly what happens when a discharge pathway breaks down — which step fails, what the patient experience is at that moment, what the workaround looks like and why it is used. By the time that account has been summarised into a patient experience metric, the specific has become a category and the category has become a score. The score moves up or down across reporting periods. What it cannot carry is the texture of the original account: the detail that would allow a decision-maker to understand not just that something is wrong but precisely what, and what a plausible fix would need to address.

The second thing translation removes is dissent. A workforce that contains people with genuinely divergent views on a strategic direction will, in most organisations, produce a management layer that presents a consolidated position. Not because dissent is suppressed — though sometimes it is — but because the function of a management summary is to synthesise, and synthesis flattens disagreement into a dominant view. The minority position, which may be the more accurate one, disappears in the transition from conversation to report.

The third thing translation removes is timing. The experience of a customer, a patient, a service user, or an employee exists in real time. The reporting of that experience is always retrospective. By the time a quarterly engagement survey is analysed and presented, the conditions it describes may have shifted. By the time a pattern identified in a management report reaches a board decision, the people who articulated it may have adapted, left, or stopped reporting. The intelligence arrives after the moment when it could most usefully have been acted on.

The same structure in different rooms

The ward sister who cannot get a staffing concern to the executive team without it being reframed as a rota management issue. The retail team leader whose customer feedback about a pricing change never makes it past the operations layer because it conflicts with a decision already made at the centre. The charity caseworker whose account of a service user’s experience is condensed into an outcome category before it reaches the funder’s impact report. The engineer at a scale-up who knows the architecture cannot support the growth plan but has learned that raising it produces a request for a document rather than a conversation.

These are not aberrations. They are the same structural pattern — the distance between the person with the direct knowledge and the person with the authority — expressed in different rooms, with different consequences. In each case, the translation that happens between those two people is doing something specific: it is making the original account manageable for the organisation’s existing processes. And in doing so, it is making it less true.

The individual framing is not irrational. It is what an existing structure can respond to — a named person, a specific failure, a defined gap that a programme can address. The structural framing asks a harder question: not who failed to pass it up, but how many translations the account had to survive before it reached anyone with the authority to act, and what that number is a consequence of.

The organisations that notice this tend to describe it in individual terms. A particular manager is not passing things up. A specific team has stopped raising concerns. A function is not feeding the right information into the planning process. The response is usually a communications initiative or a culture programme — an attempt to change the behaviour of the individuals in the chain.

The organisations that solve it tend to describe it in structural terms. The chain itself is the problem. The number of translations between the original voice and the decision-maker is the variable that needs to change. Everything else is managing the symptoms.

Reducing the translations

The case for qualitative engagement tools — open response, peer validation, AI-assisted synthesis — is usually made on the grounds of richer data. More nuance than a score. More specificity than a category. That is true, but it understates the more significant structural point.

When people respond in their own words, and when those responses are validated by their peers before they reach an analyst, one layer of translation is removed entirely. The manager’s interpretation does not sit between the worker’s account and the organisation’s understanding of it. The researcher’s coding scheme does not determine which themes are visible and which are not. The community itself has indicated what matters and in what terms — and that signal reaches the people reading it in a form closer to the original than any summary process can produce.

This does not eliminate the distance between front-line experience and senior decision-making. It does not remove the organisation’s ability to ignore what it hears. What it changes is the fidelity of the signal — and with it, the organisation’s ability to claim, in good faith, that it did not know.

In a post-acquisition integration, that fidelity is the difference between understanding what the workforce is experiencing and receiving a management view of what the workforce is experiencing. In a public service redesign, it is the difference between knowing what patients or residents actually think and knowing what the engagement process was designed to find out. In a charity measuring its own impact, it is the difference between the beneficiary’s account of what changed and the output metric the funder asked for.

The voice that does not reach the room does not disappear. It goes somewhere else — into attrition, into disengagement, into the gap between what an organisation believes it is doing and what is actually happening on the ground. It takes a different form, usually a more expensive one, and it surfaces at a time and in a way that is harder to respond to than the original account would have been.

My Opinion

The organisations I have worked with are not, in the main, hiding from what their people know. They are fixated on what can be measured — so they measure it: a sliding scale, an interval survey, a free-text field that usually becomes an afterthought. What gets analysed is the number. What goes unasked is the right question, put to the right person at the right moment and understood in context. That absence is not accidental. It is a consequence of designing for what can be counted rather than what actually needs to be heard.

The room does not have to wait for it to arrive that way.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan is a secondary school teacher in their mid-forties — the kind of person who spends the working day managing other people’s problems and expects, by the time they get home, to be done with crises for a while.

That stopped being true about six months ago.

It is a Tuesday evening and Morgan is at the kitchen table again. There is a folder. It has grown over six months from a single printed email into something that requires a cardboard divider. Bank correspondence in one section. Police reference numbers in another. A third section for the accounts Morgan did not open — the mobile contract in Rotherham, the credit card with the bank they have never used, the subscription to something they cannot name. Each one requires a letter. Each letter requires evidence. Each request for evidence requires a call. Each call begins with a recorded message about high volumes and current wait times.

The original incident took an afternoon. Someone rang, someone convinced, someone transferred. The amount was not catastrophic — significant, but not life-altering. That number appeared on every police form filled in. It appeared in the bank’s assessment of the case. It is the number that gets counted in the statistics that appear in reports that describe the scale of digital fraud in this country. It does not include the folder.

The folder is where the real cost lives. Not the money that went on a Tuesday afternoon six months ago, but the forty-seven hours since — logged, because Morgan started logging them in month two, when it became clear that nobody else was. Forty-seven hours on hold, on calls, writing letters that ask institutions to confirm what happened to a person who already knows what happened to them but must prove it again to each new department in turn. There is also the thing that does not log easily — a low hum, a background awareness, persistent and exhausting, that the folder is never quite finished, that somewhere there is probably another account, that the next letter might require something not yet thought to keep. The fraud was an event. This is a condition.

The cost nobody counts

UK Finance recorded more than £1.17 billion in fraud losses across 2024 — 3.3 million separate incidents. The figure cited in press releases, parliamentary briefings, and annual reports is always the money. What those reports do not contain is the folder. What they do not count is the recovery.

Only around 14 per cent of frauds against individuals are ever reported to Action Fraud or the police. Partly this reflects a belief, often well-founded, that reporting will not lead anywhere. The Office for National Statistics found that 71 per cent of fraud victims reported being emotionally affected. The Police Foundation report Invisible Harms documented physical and psychological impacts less understood and less measured than the financial loss, and found specialist support services to be lacking and early intervention largely absent. The shame, the self-blame, and the erosion of trust that follow an incident appear in no statistic that the banks or the government routinely publish.

The true cost of digital crime is not the original loss. It is everything that comes after.

A system designed for process, not people

The systems that are supposed to help are not designed for the person in front of them. They are designed for process — for case management, for compliance, for due diligence, for the protection of the institution carrying out the review.

The fraud team calls back to confirm what the victim already knows. The police reference number opens a portal that summarises what has already been submitted. The credit agency writes to confirm that flagged accounts have been noted — which requires a written acknowledgement that the confirmation was received. Each exchange adds documentation to someone else’s record. None of it changes the situation for the person who was defrauded.

The letter templates are identical. The call scripts are identical. The processing timelines are fixed. Every victim becomes a case reference moving through queues designed for the average. The average does not spend Tuesday nights logging hours because nobody else is.

When no one can act

A significant proportion of digital fraud targeting UK victims is perpetrated from overseas. This creates a hard stop in the recovery process that victims often reach only after months of effort. They are told, eventually, that the perpetrator is outside UK jurisdiction. That nothing further can be done. That the case is closed on the institutional side.

This is the moment many victims describe as the most damaging — not the original fraud, but the formal confirmation that the system has run its course without resolution. The UK government’s fraud strategy for 2026 to 2029 acknowledges the cross-border dimension, but the gap between published strategy and what an individual victim actually encounters remains substantial.

The crime was an afternoon. The recovery is a year of your life.

My opinion

What I notice, after three decades inside the technology industry, is how consistently the conversation stops at the immediate aftermath. We see the incident — the money transferred, the case number assigned, the bank’s response issued. What is not visible is the secondary and tertiary blast radius: what comes after, which no single organisation owns. Each institution closes its part of the process. The story continues for the person at the kitchen table. Support exists, but it is fragmented and difficult to navigate — and when someone is told that nothing can be done because the perpetrator is in a foreign country, that moment carries a weight that no fraud report measures. The consequences run much further than the original event, and they rarely land on the institution.

Questions the system never asks

Fraud recovery touches more people than the statistics suggest. If any of this is familiar — from your own experience, someone close to you, or the systems you work inside — these are worth engaging with directly.

1. Will any of this be familiar — from your own experience, or someone close to you?

2. If you have been through a recovery process — for fraud, for a breach, for anything like this — what would you want someone to know about what it actually involved?

3. Should the cost of recovery fall on the victim — or on the institutions whose systems were exploited, and whose processes extend the damage?

4. What would you want the person sitting at that kitchen table on a Tuesday night, six months in, to know — that nobody told them?

The true cost of digital crime is not the original loss. It is the months of bureaucracy, cognitive weight, and institutional indifference that nobody counts.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan had been thorough. That was the thing that made it so difficult to explain when the question came back six months later.

The assessment had been careful — a credit risk evaluation for a commercial loan application, supported by an AI-assisted analysis platform the team had been using for the better part of a year. The tool had been performing well. The outputs were coherent, explainable, defensible. The assessment reached a conclusion, the decision was recorded, the file was closed.

Six months later, a query arrived from legal. Could the team reproduce the original assessment? Run the same inputs through the same system and show that the same logic applied?

The query was run. The output was different.

Not marginally different. Not a rounding variation. Different in a way that, had it been the original output, might have changed the conclusion. Different in a way that could not be explained without knowing what had changed in the model between then and now — and that information was not available, because nobody had recorded it, because nobody had been asked to, because when the system went live, nobody had thought to ask.

The problem with the update

AI systems are not static. The models underlying them are updated — sometimes by the vendor, automatically, as part of routine improvement cycles; sometimes in response to identified bias or error; sometimes because the training data has been refreshed. This is, in principle, good. A model that learns and corrects over time is more valuable than one frozen at the point of deployment.

The difficulty is that improvement and consistency are in direct tension. A model producing better outputs today than it did six months ago is, by definition, a model that would produce different outputs given the same inputs from six months ago. That difference is the point of the improvement. In any context where a decision needs to be reconstructed or defended, it is also an unanswered question about what the original decision was actually based on.

Most organisations deploying AI-assisted tools have not resolved this tension. They have ignored it. The model updates. The outputs change. Nobody is notified.

Nobody records the version in use when a specific decision is made. The audit trail — if one exists — captures the conclusion. It does not capture the reasoning engine that produced it.

Morgan’s situation was not unusual. It was the predictable result of deploying a consumption-based AI tool without the governance to track what the tool was doing over time. The barn door had been left open. The question was not whether the horse had bolted. The question was which horse, when, and whether anyone had kept a record of the herd.

The infrastructure that was not built

For an AI-assisted decision to be defensible over time, three things need to be true.

1. The version of the model in use at any given point needs to be recorded.

2. The inputs to any significant decision need to be preserved in a form that can be rerun.

3. The outputs need to be traceable back to the specific model state that produced them.

None of these are technically difficult. Version control is a solved problem. Input logging is standard practice in well-governed systems. Audit trails linking outputs to model versions are routine in regulated software environments. The infrastructure exists.

The decision not to apply it to AI deployment is not a technical gap. It is a governance gap — the same gap that appears in every wave of technology adoption where deployment has consistently moved faster than the thinking that should accompany it.

In financial services, decisions about lending, credit, and investment carry regulatory obligations around explainability and reproducibility. In healthcare, an AI-assisted diagnostic or triage decision needs to be reconstructable if a patient outcome is later questioned. In law, the reasoning underlying a recommendation used in proceedings may face scrutiny that requires reconstruction. The sector changes. The underlying failure is the same.

What makes this particularly hard to catch is that model drift is invisible by design. The system continues to function. Outputs continue to arrive. Nothing signals that the model today is not the model that was running when the previous decision was made.

The degradation of the audit trail happens silently, with every update cycle. By the time someone needs to reconstruct a decision, the model that produced it may no longer exist in any recoverable form.

Organisations that deployed AI-assisted tools before any framework for tracking model versions was in place are already in this position: the original model gone, the update history undocumented, past decisions unreproducible. This is not a future risk. It is the present condition for most organisations that moved quickly to deploy.

When the query comes

The most immediate consequence is legal exposure. In financial services, the FCA’s Consumer Duty requires that AI-assisted decisions be explainable and that firms maintain records sufficient to reconstruct the decision-making — not just the conclusion but the model state active at the time. The Data (Use and Access) Act 2025, which received Royal Assent in June 2025, adds further safeguards around automated decision-making, including the right to information about the reasoning used and the ability to contest decisions. A firm that cannot reproduce the original output cannot satisfy either obligation. The decision may have been correct. Without reproducibility, that cannot be demonstrated.

In healthcare, the consequences sit closer to the individual. An AI-assisted triage or diagnostic support tool that has been updated between a clinical decision and a subsequent review creates a gap in the clinical record. The original output cannot be reproduced. The basis for the original decision becomes opaque in retrospect. That opacity matters most precisely when the outcome was not what was hoped for.

In law, the problem is evidentiary. Any AI-assisted assessment used as part of a legal process — risk scoring, document analysis, due diligence support — may be subject to scrutiny requiring the reasoning to be reconstructed. A system whose model has changed since the original output was produced cannot provide that reconstruction. The question of what the AI actually considered, and on what basis, becomes unanswerable.

Beneath all of these is a more general corrosion. Organisations deploy AI tools to improve consistency — more reliable, more defensible, more uniform outputs than human judgement alone. Model drift undermines that promise at its foundation. The consistency was real at the point of deployment. Whether it persisted through subsequent updates is unknown, because nobody was watching. Every benefit of the deployment becomes contingent on the governance infrastructure most organisations have not built.

Closing the door

The work required to manage model drift has to be done before the problem appears, not after. Once a decision has been made, the model has been updated, and the version history has not been kept, the gap cannot be closed retrospectively. The barn door cannot be rehung from inside the field.

Version pinning is the starting point: every AI-assisted decision of significance should record the model version, the input data, and the output at the point it is made. This is not different in principle from the version control already applied to regulated software in financial services, healthcare, and legal environments. The principle is established. The application to AI tools has not been made.

Vendor contracts are the second lever. Most organisations deploying AI platforms under SaaS or consumption-based models have no contractual requirement for model update notification, version documentation, or backward reproducibility. This is a procurement failure as much as a governance one. The capability to reproduce historical outputs may exist technically with the vendor. Without a contractual obligation to preserve and provide it, that capability is at the vendor’s discretion.

The AI Business Office — the governing function responsible for AI consumption and risk across the enterprise — is where both of these responsibilities sit. Not in compliance, not in IT. In a function with genuine authority to set standards and enforce them — not merely to advise. That function is absent in most organisations deploying AI today. Its absence is the systemic condition that makes Morgan’s situation not exceptional but typical.

The deployment decision is the easy part. The governance that makes it defensible over time is slower, less visible, and less likely to feature in the business case. It does not appear in a vendor demonstration. It is the work done — or not done — in the months after the system goes live, once the attention has moved on and the question of what comes next has not yet been asked.

By the time it is asked, in many cases, the answer is already too late for the decisions already made.

The problem predates the AI

Model drift is not new. Statistical and machine learning models have drifted as long as they have been deployed. Any model trained on data and used over time will drift as conditions change, as data is refreshed, as the model is retrained — actuarial models, credit scoring systems, fraud detection engines. The problem is older than the current generation of AI tools.

What AI has changed is the speed, the scale, and the opacity of what changed. Traditional software updates come with version numbers, change logs, release notes. AI model updates change patterns in parameters, not lines of code — what changed is often not just undocumented but undocumentable in the same form. The update cycle is faster. The number of decisions affected is larger. The answer to “what exactly is different, and in which direction?” is harder to state even when someone thinks to ask.

This creates a provenance problem that existing governance frameworks were not built to handle. The question is not only whether the model drifted. It is whether anyone can say what it drifted from, when, and with what consequence for the decisions made in between. Most organisations cannot answer that. Not because they are careless. Because the infrastructure to track it was never put in place.

Organisations moving into AI-assisted decision-making in regulated contexts are taking on an obligation that most have not costed and many have not identified. The question is not whether model drift will affect their decisions. For most organisations that have been running AI-assisted tools for more than a year, it already has. The question is whether they will know.

What the question from legal would reveal

• If your organisation has deployed an AI tool supporting significant decisions — in lending, clinical assessment, legal analysis, or any regulated context — can you reproduce the output of a decision made twelve months ago? Not approximately. Exactly, from the same inputs, using the same model.

• If your AI platform vendor released an update in the last six months, were you told? Is the update version recorded anywhere against the decisions made before and after it? Does anyone in your organisation own that responsibility?

• When you consider the decisions your organisation has made with AI assistance over the last year, which would be most difficult to defend if questioned today? What would you need to reconstruct the reasoning — and is that information still available?

• The work that makes AI deployment defensible over time costs money and requires deliberate decisions to build. The absence of it also costs money — in legal exposure, regulatory risk, and decisions that cannot be explained. Which investment is your organisation actually making?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The room goes quiet. Not uncomfortable quiet, a different quality entirely. The kind that happens when a group of capable professionals simultaneously understands that it does not have an answer to a question it should have been asking all along.

The question came in the middle of a design review that had been going well. The wireframes were done. The architecture was settled. The team had been professional throughout and the design was technically sound. Someone near the back of the room looked up from the screen and said four words that were not on the agenda: who is this hardest for?

Nobody answered immediately. Alex looked at the wireframes. The design had a user in it, of course it did, but that user was implicit, unexamined, averaged into the decisions without ever being named. The silence in the room was the collective recognition that the person who would find the design hardest to use was probably not the person the design had been built around.

What the room had been reviewing

Before the question arrived, the review had been evaluating what design reviews are structured to evaluate. Does it meet the technical requirements? Is it consistent with the component library? Does it follow the agreed interaction patterns? Is it on schedule? All of it legitimate. None of it the wrong question. And none of it asking who the design would fail.

A review process evaluates what it is structured to evaluate. The structure of this review did not include a question about the hardest user. And so that question had not been asked — not through negligence, not through indifference, but because the agenda did not have a line for it.

The design had an implicit user at its centre: someone who could read the instructions without difficulty, navigate the steps without support, understand the error messages without translation, and complete the verification without a second attempt. That user was not wrong to design for. They were just not the only person who would have to use the product.

The review had been thorough and the design was ready. The question changed what ready meant.

The structural fix is a single line

The absence the silence named is not expensive to address. It does not require a new team, a revised budget, or a different process. It requires one question added to the structure — to the design brief, to the review agenda, to the definition of done — in a form that is not optional and not supplementary.

Who is this hardest for, and what have we done about it?

Asked early and written into the process, that question is the intervention. Not at the end of the project, when the architecture is fixed and the cost of change is high. At the beginning, when a question in a brief costs nothing to answer and the answer shapes what is built. In the middle, when a question in a review can still redirect a flow, a content decision, an error state, a test plan. Before release, as a criterion that must be met before the product ships.

The reason the question is not already standard is not that it is expensive. It is that the processes governing design were built to evaluate what their authors cared about, and the hardest user was not in the room when those processes were written.

What made the question work in Alex’s review was that it arrived without defensiveness, a genuine inquiry that assumed the team capable, not a challenge to what they had produced. It was followed by something specific: not “users with lower digital literacy” as an abstract category, but a named-enough description of a real kind of person the team could treat as a design constraint. From there they knew what to do.

That specificity is what turns a question from a philosophical concern into a piece of work. Abstract categories — accessibility, inclusion, the underserved — are easy to acknowledge and difficult to act on. A specific person in specific circumstances is a brief. The question that produces a brief is the question that produces a change.

What the question produced

The first change was to the onboarding instructions. Written for someone who already understood the purpose of the system, they assumed knowledge that the hardest user would not have. Rewritten at a lower reading level, with the purpose of each step stated plainly before the step itself, they carried the same information to a wider range of people. The change took two hours. It had been available from the start of the project.

The second change was to the verification flow. Designed as a four-step process with each step branching on the user’s input, it required a correct first attempt to avoid a dead end that offered no clear route forward. The hardest user — the person the team named after the question was asked — would have reached that dead end reliably. The flow was restructured: two steps, linear, with a recovery path at every stage. Completion rates in user testing improved across all participants, not only the hardest case.

The third change was to the error messages. “Invalid input” had appeared throughout the design. The hardest user would not know from those two words what was invalid, why, or what to do about it. The messages were rewritten to name the error specifically and state the fix plainly. It took an afternoon.

The fourth change was a user test. Before the question, the test plan had included participants matched to the central profile, engaged users who knew what they were doing. After the question, a round of testing was added with participants who represented the hardest case. They revealed a fifth issue the team had not anticipated: a step considered self-explanatory that three of the four hardest-case participants read differently from how it was intended. The step was reworded before the product shipped.

None of these changes were large. Together they constituted the difference between a product that served most of its users and one that served a wider range. The only cost was the question that had not been on the agenda until someone put it there.

Designing for the outlier

The six-word question contains more than it appears to. Asking who the design is hardest for forces the team to ask something they had not been asked before: does this product genuinely help the person who needs it most, or does it add to the capability of people who were already capable without it?

The pattern is not unique to design reviews. Netflix has reported that 80 per cent of its subscribers use captions at least monthly, a feature built for people who are hard of hearing, now used daily by people watching in a second language, in rooms where the sound cannot be loud, putting children to sleep. Microsoft’s Inclusive Design programme names the principle: solve for one, extend to many. Designing for the person who finds the product hardest to use does not produce a narrower product. It produces a more useful one.

That is what the redesign of Alex’s verification flow demonstrated. A flow restructured to be navigable for the person who would have found it hardest did not improve completion rates only for that person. It improved them across every participant in the test. Designing for the outlier is the strongest test a design can face. It reveals what designing for the average conceals.

The design processes that do not have this question built in are not negligent. They evaluate what they were built to evaluate, and the people who built them made decisions about what mattered. The person who would find the product hardest to use was not part of those decisions. That has not been corrected, not because the correction is expensive, but because the people who govern design practice have not decided that it is required.

Four questions worth asking

1. Think of the last design review you participated in as a designer, a stakeholder, a commissioner, or a user tester. Was the question asked: who is this hardest for? If it was, what happened? If it was not, what do you think would have been different if it had been?

2. If you run or participate in design reviews, what would it take to add the human question as a standard agenda item not optional, not supplementary, but required? What is the obstacle, and is the obstacle proportionate to what it is protecting?

3. If you commission technology, does your brief require the supplier to answer the question: who is this hardest for, and what have you done about it? If not should it?

4. If you had one question to add permanently to your design process — not as guidance, but as a required criterion — who in your organisation would need to decide that? Have you ever been in the room where those decisions are made?

The silence in Alex’s design review did not last long. The team had a description within a few minutes grounded enough in the work to treat as a constraint. And once it was a constraint, they knew what to do with it. The question that was not on the agenda was the most productive thing the review produced.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The afternoon my family member nearly died began without warning. A hornet sting — the kind of thing you’d complain about briefly and then forget — and within minutes the immune system had declared war on itself. Blood pressure dropping. Airway closing. A blue-light run to a resuscitation bay where a crisis team did what crisis teams do, which is chaotic, brutal, and clinical, and not something that leaves you easily.

They survived. That part, at least, went the right way.

What nobody told us — what we never understood until we were living it — is that survival is not the same as resolution. The body that came home from that hospital was not the same body that had left that morning. The immune system had been permanently recalibrated. What had been a minor irritant was now a potential death sentence. And the list of things that could trigger a response had begun to grow.

Twenty million people are living this

This is not a rare condition being dramatised for effect. The UK has some of the highest allergy rates in the world. Approximately 20 million people, nearly a third of the UK population, are living with allergic conditions of some form. At the severe end, the trajectory is sharply upward. Hospital admissions for anaphylaxis have more than doubled over the past two decades, now exceeding 25,000 a year. Food-related anaphylaxis admissions more than tripled between 1998 and 2018. The cohort at highest risk for fatal outcomes is adolescents and young adults.

The number of deaths from food-induced anaphylaxis, around ten a year in England and Wales, has fallen over the past two decades, a direct result of improved emergency medicine. The number of people experiencing the crisis has not. It is rising.

The clinical picture is well understood. A sting, or a misplaced ingredient, triggers an immune cascade. Inflammatory mediators flood the body. The vascular system collapses. Blood pressure falls. The airway constricts.

Without high-dose adrenaline, the outcome is not certain. With it, the patient is stabilised, discharged, and returned to an environment that looks exactly as it did before — except that their relationship with physical safety has been permanently severed.

What the clinical picture does not fully capture is what comes after.

The whiteboard on the kitchen wall

In our home, the central artefact of daily life is a whiteboard in the kitchen. It is titled “No Foods.” It is not a list of standard allergens or personal preferences. It is a living document — updated when a product changes its formulation, when a supplier is switched without announcement, when something that was safe last week turns out not to be safe this week, even which vendors own products are not safe. This week the addition - biscuits from one company that were fine 6 months ago.

This is one of the less visible consequences of a severe allergic episode: the initial trauma can destabilise the baseline immune response, making previously safe ingredients suddenly hostile. The immune system, having learned that the world is dangerous, begins to treat a wider range of inputs as threats. A minor recipe alteration by a manufacturer, the kind of change that would go entirely unnoticed by most people, becomes a potential crisis. Every meal is an exercise in risk assessment. Spontaneity, for ordinary food and ordinary eating, is simply gone. Imagine that you can never again go out for a meal, or order a take-away, or grab a sandwich for lunch - that’s not imagination it’s reality.

The psychological cost of this has been measured. Research consistently shows significantly lower quality of life scores and higher rates of severe anxiety in adults with a history of anaphylaxis compared to healthy populations. The correlation between a severe allergic episode and PTSD is well established, a 2020 study found more than 40 per cent of anaphylaxis patients developed the disorder.

The sound of a buzzing insect, something you or I would register briefly and ignore, produces an immediate, involuntary panic response. The brain has matched the sound to the memory of resuscitation. The cortisol floods before the conscious mind has time to assess whether there is actually a hornet in the room. This is not anxiety in the colloquial sense. It is a physiological response to a physiological memory, and it does not switch off.

There’s no cure — just be careful

The social experience of living with severe anaphylaxis has its own particular texture, and it is worth naming plainly.

When people learn about a condition like this, the instinct — a kind instinct, genuinely — is to troubleshoot. Have you tried an antihistamine first? Surely a little bit would be fine? What about when you’re on holiday — do you just not eat out? There’s a new treatment I read about, have you looked into that?

Just the other week a local market stall where I buy coffee from offered something else from the stall - I had to explain thanks but no thanks. The offer was genuine as a thank you, but when I said no to one thing they offered another, then another, then final some honey to which my response was - that will kill. It wasn’t ignorance just someone who was being genuine and kind but who didn’t understand or have the experience to relate.

These questions come from goodwill. They are also exhausting in a way that is hard to explain without sounding ungrateful. Because what they share — what they all have in common — is the assumption that the condition is a problem to be solved, a boundary to be tested, an inconvenience to be worked around with the right combination of effort and information. They place the burden of managing other people’s discomfort with the situation onto the person already carrying the weight of the condition itself. Imagine having to relive the experience every single time you have to explain to someone why you can’t eat something.

The medical answer to “can’t you just be careful?” is: yes, that is precisely what is happening, every hour of every day. Careful is the entire structure of the life. The whiteboard is careful. The pre-reading of every menu is careful. The carrying of two auto-injectors at all times is careful. The hypervigilance about insects in summer, about shared preparation surfaces, about products that changed without warning — all of it is careful. There is no relaxing of careful. There is no holiday from it.

What the well-meaning questioner is actually asking, without quite realising it, is whether the person could do a little less of the work that is keeping them alive.

This is not a criticism of individuals asking those questions. It is an observation about what we, collectively, have never been asked to learn. Severe allergy management is invisible work — constant, precise, and carried almost entirely by the person affected and the small circle around them. The system that should support it — healthcare, food labelling, public understanding, employer awareness — provides partial answers at best. The gap between what the condition demands and what the environment provides is filled by the individual, every day.

Twenty million people in this country are managing allergic conditions. Tens of thousands are in the severe range, where a miscalculation is not an inconvenience but a medical emergency. The numbers are rising.

The more useful question is not what those individuals should do differently. It is what a society that understood this — genuinely understood it, not just acknowledged it — would look like. What would food labelling require? What would employer sick leave and workplace risk awareness cover? What would public education in schools actually teach?

And whether we are, in any meaningful sense, building towards that — or whether we are still, quietly, expecting the person with the whiteboard to carry it alone.

The other side of the whiteboard

What the phrase “carrying it alone” means in practice — for the people in the circle around the person affected is a reality of everyday life.

I know the exact words to say when I call emergency services. Not approximately — exactly. The order, what to lead with, what comes second, what must not be left out. I know this not because I’ve thought about it, but because I’ve had to do it more than once. I am hyper-vigilant about what goes into our food, about whether something has changed on a label, about whether the auto-injectors are where they should be. I challenge. I say no. There are situations I assess before being asked, places we do not go, things we do not order, risks I have absorbed into the background noise of daily life without announcing them.

None of this is something I name as a burden. It is just how we live.

The hardest thing to explain — to a friend, or to an employer — is that there is nothing to point to. After an episode, recovery is not only physical and it is not quick, what direct dose adrenaline does to the body is brutal. It can take days, sometimes weeks, before things settle. The nervous system takes time to stand down. During that time, what is needed is proximity — being close, being home, being available. Not a formal diagnosis. Not sick leave as conventionally defined. Just: present. Employers have no real frame for this. There is no obvious illness to present. There is only the aftermath of something serious, which looks like nothing from the outside.

When I hear the words “I don’t feel well” — for any reason, including something as ordinary as a cold — my heart drops before I notice anything else. The possibility announces itself before the likelihood can be assessed. That response was learned. It does not un-learn. It is not dramatic. It is just there, every time, a fraction of a second ahead of rational thought.

The condition is carried by the person who has it. It is also carried, differently, by the people close to them. The whiteboard is in our kitchen.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The teacher had been watching it happen for three years before they put it into words. Children arriving in class who couldn’t reach boredom. Not restless, not disruptive — just unable to get to the still place where imagination begins. The pause before invention. The quiet gap that used to be slightly uncomfortable and was, in that discomfort, useful.

It wasn’t every child. But it was enough children, consistently enough, across enough different homes and backgrounds, that the teacher had stopped explaining it as a parenting problem or a personality type. Something structural had changed. The doorway was being blocked before anyone arrived at it.

Boredom, the teacher said, used to be the beginning of something. Now it’s never allowed to start.

Children’s brains are not small adult brains. This sounds obvious. The implications are less often followed to their conclusion.

The prefrontal cortex — the part of the brain responsible for impulse control, the ability to defer gratification, the capacity to say “not now” to a stimulus and return to a chosen task — continues developing into the mid-twenties. The systems being handed to children in their primary years were designed to defeat exactly those capacities in adults who have them fully formed. Variable reward schedules, infinite scroll, notifications engineered to break the moment attention begins to settle.

Applied to a brain that hasn’t yet built its filtering mechanisms, these systems are considerably more powerful. The architecture was never redesigned for the people it was being given to.

Age verification, as it currently exists, is largely a legal gesture. A tick-box. A date of birth entered by a nine-year-old that says they were born in a different decade.

The commercial incentive to grow the user base is structurally incompatible with genuine exclusion of minors. Younger users are not an unfortunate byproduct of a system aimed elsewhere. They are an attractive demographic — highly engaged, inexperienced enough not to recognise what is being done to them, and with decades of usage ahead. The systems were built for adults, applied to children without redesign, and the companies that built them have consistently treated the question of harm as a reputational problem to be managed rather than a design problem to be solved.

The long-term consequences of early, sustained exposure to attention-capture systems on developing cognition are not yet fully known. This is not a reassuring sentence. It means we are running an experiment on a generation — on the formation of their attention and their capacity for sustained thought — without their consent, without their parents’ full understanding of what consent would even mean, and without any agreed mechanism for knowing when the harm has been large enough to act on.

A 2024 meta-analysis published in JAMA Pediatrics — 143 studies, more than a million adolescents — found a significant positive correlation between social media use and anxiety and depression in young people. These are not correlations the industry disputes any more. They are correlations the industry has largely decided to absorb.

The test it fails on every count

Attention-capture technology applied to a developing brain does not help the child — it interrupts the developmental processes that build the capacities they will need for the rest of their lives. You cannot augment a brain you are interrupting at a critical stage. The claim that these systems offer connection and creative expression cannot hold when the underlying architecture is designed to deepen dependency. What is added to the platform is not what is added to the child.

These systems do not adapt to the child. They apply adult exploitation mechanics to children because it is cheaper than designing differently. A system that genuinely responded to who was in front of it would treat children as a protected category rather than a category to be captured.

Before the next app is handed over

Four questions.

1. Think about what unstructured time felt like as a child — the mild discomfort of it, and the invention that eventually followed. Do the children in your life have access to that? Has anyone asked them?

2. If a toy was found to reliably reduce children’s attention spans and increase anxiety, it would be removed from sale. What would it take to hold digital products to the same standard — and who would need to demand it?

3. What age — genuinely, not the legal minimum — would you consider appropriate for unsupervised access to algorithmically-fed social media? And if that age is higher than the current norm, what do you do with that gap?

4. Is this something you have had an honest conversation with the children in your life about — not a rules conversation, but the actual mechanics of what these systems are doing and why? If not, what has made that conversation hard to start?

The next generation’s relationship with attention is being formed right now, in systems designed by people who were not thinking about them.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam spent three evenings on the application. Not on the covering letter — on the research. The company’s recent work, where it had been, where it was going, the names of people who might read what Sam wrote. The application itself took one more evening. By Friday afternoon it was in.

Then nothing. Not a receipt. Not a holding message. Eventually not even a rejection. Just the particular silence that follows something you cared about and cannot un-send.

Three weeks later, Sam checked the job board. The listing was gone. The role had been filled.

What the filter decided

Sam’s application was read. Just not by a person.

An applicant tracking system — software designed to process high volumes of applications before any recruiter makes contact — compared Sam’s CV against a pattern. The pattern was built from historical data: previous successful hires, their job titles, their prior employers, the particular density of certain keywords in certain fields. Sam’s CV did not match. The system moved on.

There was no decision Sam could appeal. No criterion to review, reconsider, or present counter-evidence against. No one in the organisation knew the application had been received, let alone assessed. The system did not tell Sam a judgement had been made. It simply stopped.

The silence was the answer.

The recruiter who left the room

This is not a new failure. It is the endpoint of a process that has been under way since the mid-1990s.

When specialist recruitment agencies dominated the market, the relationship between candidate and employer was mediated by a person with knowledge of both sides. That person knew what a candidate’s career trajectory meant — why they had moved, what they had built, what they were capable of next. The relationship was imperfect, and expensive, and slow. But it was a relationship.

Mass-market job boards made applying easier and made that relationship rarer. Then platforms introduced Quick Apply — a feature that lets a candidate submit to a role in two clicks using a profile already on file. The frictionlessness was sold as convenience. What it produced was volume. Hundreds of applications per role became the norm, and the only rational response to hundreds of applications per role, with no additional resource, is to filter automatically.

AI-powered filtering is the logical outcome of that chain. And it has now reached the point where AI-generated CVs — assembled by tools that know what ATS systems favour — are gaming AI-powered filters in turn. The signal is indistinguishable from the noise.

No human being in the system made a wrong decision. No individual is responsible for what Sam experienced. Which is precisely what makes it so difficult to fix.

I have sat on the other side of this. As a hiring manager I have opened a shortlist and known — not suspected, known — that the candidates in front of me were not the best people who applied. The best candidates were somewhere in the stack the system never surfaced. There was no mechanism to go back and look.

The scale of the silence

Sam’s experience is not unusual. A 2025 analysis by Jobscan found an ATS system detectable at 97.8% of Fortune 500 companies. A Harvard Business School survey of more than 2,000 executives found that 94% of employers with middle-skills roles and 88% of those with high-skills roles believed that automated filtering was removing qualified candidates using criteria such as job title, credentials, and years of experience rather than actual ability to do the work. They continue to use the systems.

The most visible legal consequence is currently making its way through the American courts. In Mobley v. Workday, Inc., five people over the age of forty applied for hundreds of jobs using Workday’s platform and received almost no interviews. In May 2025, a federal court in California certified a collective action — a case that may now cover hundreds of millions of job seekers over forty. The claim is that an automated system made systematic decisions about people based on protected characteristics, without any individual in the process being aware of or accountable for what was happening.

In November 2024, the UK Information Commissioner’s Office published a report raising concerns about AI recruitment software operating in Britain — software found to be filtering candidates on the basis of gender, race, and sexual orientation, invisibly, with no individual in the chain accountable for what it decided.

For Sam, none of that is the point. The point is that Sam spent four evenings preparing something that was never read. And will never know why.

A wall with a letterbox

Sam could not use the system to present themselves clearly. There was no mechanism to ask a question, add context, or correct a misreading. The system did not help Sam do what they were trying to do — it removed them from the process without their knowledge.

A competent recruiter who knew the role and spent twenty minutes with Sam’s application would have surfaced things no pattern-match can — the reasoning behind a career move, the range of a role that looks narrow on paper, the capability that does not translate cleanly into keywords. The automated system did not supplement that judgement. It replaced it, at lower quality.

The system applied the same pattern-match that it applied to everyone. That pattern was trained on past hires which means it was designed to reproduce the past. If the past encoded bias — by employer, by educational institution, by the names and formatting conventions of particular social groups — the system reproduced that bias at scale, invisibly, with no one accountable.

A system that cannot help the candidate, cannot add to the process, and cannot distinguish one person from another is not a recruitment tool. It is a wall with a letterbox. Applications go in. Nothing comes back out. And no one inside knows what the wall contains.

If you own any part of this

Most people reading this have sat somewhere in this system. Has this happened to you, or to someone you know — the care, the preparation, the submission, the silence? Not a rejection. Just nothing.

If you work in technology or HR, what would you change tomorrow — not after the next procurement cycle, not after the regulatory guidance arrives, but tomorrow?

If you are currently using these systems to hire, do you know what they are filtering out on your behalf? Not what they are selecting. What they are removing.

Is optimising for volume the right goal — or have we confused efficiency with quality at precisely the moment quality matters most?

My opinion

The harm in recruitment is not a technical malfunction. It is a design choice made upstream, long before the system was deployed, invisible to everyone downstream — including the organisation that bought it and the candidate it rejected.

Recruitment is a human activity. We hire people, not machines — so why are decisions about who gets considered being made by software with no understanding of context, trajectory, or circumstance? The outcome is already visible: candidates submitting fictitious CVs engineered to pass an automated filter, because they know that what they have actually done matters less than whether they can predict the pattern.

Until someone in the process is accountable for what the system decided, Sam’s experience will keep happening, at scale, in silence.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Riley submitted the application on a Tuesday morning. The Universal Credit advance, emergency money designed to arrive within 24 hours, was there to bridge a gap that Riley had not expected and could not have planned for. Retirement, a short-term shortfall between pension arrangements, a position they had never been in before. The form took two hours. The supporting documents were gathered and uploaded. The submit button was pressed.

By Wednesday afternoon the money had not arrived. Instead, a message: additional documentation required. No explanation of what had triggered the request. No indication of how long the review would take. No suggestion that anything was wrong, but no money either, and no path through that Riley could see.

Riley had not done anything wrong, they hadn’t committed fraud or attempted to, they had simply applied for emergency money and been flagged by a machine learning model that had read their age and their nationality and produced an output that looked, from the outside, like due diligence.

What the system saw

The DWP’s AI model for Universal Credit advance claims has been in operation since May 2022. Its stated purpose is to flag potentially fraudulent applications before payment, a reasonable thing for a public body to want to do. The model was trained on historical data: years of records from the fraud investigation process, cases that had been referred for review, patterns extracted from that referral history.

The problem is what that data actually contained. The investigation records did not capture who committed fraud. They captured who was referred for investigation, a related but distinct category, shaped not by the behaviour of claimants but by the behaviour of caseworkers. Who the investigators looked at. Which patterns they had been trained to notice. Which demographic characteristics, accumulated over years of investigation practice, had become associated in the institutional record with the word “suspicious.”

The model was trained on the output of that process. It learned the pattern of the investigators, not the pattern of the crime.

Riley’s age, late sixties, and nationality were among the signals it read. The flag was not an error in the conventional sense. The model did exactly what it was designed to do. It identified a pattern in the training data and applied it. The pattern it identified was the product of how human investigators had behaved across years of decisions that nobody had examined before the data was fed into the model.

From Riley’s position, the process was illegible. The notification gave no reason. The documentation requested was not connected, in any explanation Riley received, to the specific concern the model had raised — because the model’s reasoning was not disclosed. The request could have been for any reason. Riley gathered the documents, submitted them, and waited in the particular uncertainty of someone who has done nothing wrong and does not know how to prove it to a process that hasn’t told them what it suspects.

What the data actually captured

The DWP published a fairness analysis in July 2025, the first such analysis of the model’s outputs. Its finding for age was specific: claimants aged over 66 are referred at 49 times the rate of those aged 35 to 44, a number the DWP’s own analysis asks to be read with caution, on account of sample size. Non-British nationals face a separate, statistically significant disparity. The DWP’s own conclusion was that these figures present minimal concerns of discrimination or unfair treatment, a judgement made in aggregate, against the model’s overall fraud-detection performance.

A person is not an aggregate. A system that fails one person has failed that person by a hundred per cent.

The model had been running for three years before those numbers were officially produced. Three years of decisions, made at machine speed, built on a pattern that the people deploying the model had not interrogated.

The finding is damaging not because the DWP is unusual in this respect, but because it is not. The problem of training AI on data that was collected for a different purpose, under different assumptions, by people who are no longer available to explain what it captured is not a DWP-specific failure. A Gartner survey of 248 data management leaders in 2024 found that 63% of organisations either did not have or were unsure whether they have the right data management practices for AI. A 2025 S&P Global survey of over 1,000 IT professionals across North America and Europe found that 42% of organisations had abandoned most of their AI initiatives that year up from 17% the year before. Gartner projected that 60% of AI projects would be abandoned through 2026 for the same reason.

The problem is the default. The question that should be asked before a model is trained — what does this data actually capture, and is it fit for this purpose? — is not in the procurement brief, not rewarded in the governance process that approved the deployment. An AI rollout has a sponsor, a business case, and a delivery deadline. A data audit has none of these things. The person who raises the prior question is seen as slowing the project down.

What happens instead is that the uncertainty in the data, the gap between what the data claims to capture and what it actually captured, is not addressed. It is absorbed. The AI model processes the data and produces outputs. Those outputs look different from the raw data they were trained on: formatted with a confidence that the underlying material did not have. A spreadsheet formula error looks like a spreadsheet formula error. An AI-generated flag looks like an AI-generated flag, processed, weighted, produced by a system that has reviewed thousands of cases. The uncertainty has been cosmetically removed. What was always imprecise is now presented as authoritative.

The speed is the problem. When human caseworkers made decisions based on biased investigative patterns, the bias was slow, inconsistent, and visible to anyone who looked at the records. Individual decisions could be challenged. Patterns could be identified over time by people paying attention. The bias was real and it caused harm, but it was legible.

When those same patterns are encoded in a model and applied at machine speed to every application that passes through the system, the bias becomes fast, consistent, and invisible to anyone without access to the training data and the analytical capacity to trace the pattern back through the outputs. The fairness analysis that revealed the disparity was published three years after the model began operating. Three years of Riley-shaped decisions, each one arriving with the authority of a system that had been approved, deployed, and described as an improvement.

This is what happens when you put dirty data into a fast lorry. The bin hasn’t changed. Only the lorry got faster.

What the system did not do for Riley

The system added friction and delay to an urgent process at the moment the process existed specifically to avoid delay. The machine learning model was positioned as a fraud-detection improvement, an addition to the system’s capability. What it added, in Riley’s case, was the appearance of due diligence applied to a demographic pattern that nobody had verified was a fraud pattern.

Riley was a first-time claimant with no fraud history, in a short-term financial gap, applying for emergency money. None of that changed the flag. The model read age and nationality and produced an output. Riley’s individual circumstances, the specific facts of this specific person’s specific situation, were invisible to the process. What was visible was the pattern. The pattern was wrong.

The right of appeal exists in theory. In practice, Riley received no explanation of why the flag was triggered and therefore had nothing specific to contest. An appeal process that cannot name what it is correcting cannot correct it. The accountability exists on paper. It does not exist at the level of the individual case, for the individual person, at the moment they need it.

Opacity and urgency compound each other in ways they do not in most other AI applications. Riley needed the money quickly. The delay imposed by the flag was not a minor inconvenience, it was the failure of the system to do the thing it existed to do, at the moment it was supposed to do it.

The request could not be contested because the reason for it was not given. The time taken to gather the documentation and wait for review was time Riley could not afford. The system had been designed to detect fraud faster. It had not been designed to protect the person it flagged from the cost of being wrong.

What your organisation has not asked

The last time your organisation deployed or fine-tuned an AI model, did anyone trace the training data back to its original purpose and ask whether it was collected under assumptions that still apply? Or did the procurement process treat the data as given, a condition of the project rather than a subject of scrutiny?

If your AI is working from data that came from a legacy system, a different team, or a previous era of policy, does anyone in your organisation know what was baked into it at the start? Not what the documentation says it captures. What it actually captured, given who produced it and under what instructions.

When an AI output is wrong, does your organisation have a process for tracing the error back to the data or does the inquiry stop at “the model needs retraining”? If it stops there, the same data produces the same pattern. The lorry gets faster. The bin stays the same.

How many decisions your organisation makes on AI-generated outputs have been tested against this question: would this output look different if the training data had been gathered differently? If nobody knows, that is the answer.

Riley eventually received the money. The process took eleven days. The explanation never came.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam is presenting the quarterly review. The slide on screen is mostly green. Transaction volume is up. Response times are within target. Cost per interaction has fallen for the third consecutive quarter. The team has worked hard and the operational data confirms it.

Then Sam shares the second slide.

It shows the results from the human outcome survey, a question the organisation introduced fourteen months ago and now treats as a primary metric alongside the operational data. The question is this: did this make your life easier? The percentage who answered yes is lower than the dashboard would suggest. No one says anything for a moment.

The gap the dashboard couldn’t see

One finding sits at the centre of the silence. The account management flow, the one the team had spent a quarter building, the one that had a 94% completion rate in the transaction logs, was completing. It was just not helping. People were finishing the interaction and then calling the contact centre to do the same thing again, because the digital process had technically resolved their request without giving them the information they needed to act on it.

The operational data had no way to see that. Completion was completion. A completed transaction looked identical whether the person on the other side was better off or not.

What made this particular finding difficult was not what it revealed about the technology. The technology had worked as designed. What it revealed was the distance between the organisation’s design assumption and the person’s actual need, a gap that the operational metrics had been measuring around, consistently and accurately, for six months without once pointing at it.

The human outcome survey had pointed at it in its first cycle.

Two things that look like measurement

Most technology measurement is built around what is easy to measure: transaction volume, uptime, response time, cost per interaction, a satisfaction score. These are genuine and necessary. They tell you whether the system is working. They do not tell you whether the person is better off.

That distinction matters because the two things can diverge. A system can process every request correctly, within time, below budget, and within the satisfaction score threshold, while the person using it is not meaningfully helped. When system performance and human outcome are aligned, the operational data is sufficient. When they diverge, the operational data will always miss it because it was never designed to see it.

The investment Sam’s organisation made was in the measurement that sits between those two things: a regular, person-centred question asked of a consistent cohort and treated as a primary metric, not a supplementary one. The question was short, the cohort was manageable, and the review cycle was quarterly. What it required was the willingness to know the answer and to treat an uncomfortable answer as information rather than as a problem to manage.

That willingness is the harder investment. And it is harder to make than it sounds when the comfortable measurement is already available, already accurate, and already green.

What distinguished Sam’s organisation from those that have attempted something similar and retreated was not the sophistication of the method. It was three conditions applied simultaneously. The question was written from the person’s perspective, not the organisation’s, not did the interaction resolve according to our process, but did this make your life easier. It was treated as a primary metric, not a supplementary one, not a footnote to the operational dashboard but a standing item in the same governance conversation as the cost and completion data. And the findings were acted on: the gaps the measurement revealed led to specific changes in the next sprint, not to a report that sat in a folder and was referenced in the following year’s strategy document.

Remove any one of the three and the measurement produces knowledge without consequence which, over time, is indistinguishable from not measuring at all.

What the measurement asked them to do

The account management flow was redesigned. The team removed three steps, rewrote the final confirmation screen to include the specific information people had been calling to ask for, and added a direct link to the next action in the process. The completion rate held. The follow-up contact-centre calls for the same request fell by more than a third over two quarters. The human outcome score for that interaction moved above the organisational threshold.

A second finding produced a different kind of change. The notification system, the automated message that told people their request had been received, was generating confusion rather than confidence. People could not tell from the notification whether anything was being done, or when. It was technically accurate. It was not useful. The content was rewritten, not the system. Three sentences became one. The call rate for status queries dropped sharply in the following quarter.

A third finding was harder to act on. A feature the development team had invested six weeks in was being used by fewer than eight per cent of the people it had been built for. The human outcome data showed it was not making their lives easier. The feature was not removed, the minority who used it found it helpful, but it was deprioritised in the roadmap, and the equivalent capacity was allocated to a flow the measurement had identified as a real source of difficulty.

None of these changes were dramatic. What was significant was the cumulative direction: a product moving, over time, closer to the experience the people who used it actually needed, because the organisation was measuring the distance and choosing to close it.

The questions operational data cannot answer

Each of the three changes answered a different question and all three are questions that operational data cannot answer.

The account management flow had been answering the wrong one. The system was processing. The person was not better off. Those are different things, and a completion rate measures only the first. An organisation that restricts itself to that question will build technology that processes correctly and never quite serves. Sam’s organisation chose to ask the other one, did the person’s situation improve?, and to invest in knowing the answer rather than assuming it.

The notification redesign revealed something else. The notification was not wrong; it was simply not adding anything. It created work rather than removing it. An organisation that measures whether its technology adds something for the person, not whether it generates an output, will make different decisions from one that measures output alone. The redesign took a day. The measurement that revealed its necessity took fourteen months of consistent data collection to surface.

The deprioritised feature answered the hardest question: does the technology respond to the people who are actually using it, rather than the population the team imagined when they designed it? It had been built for a user group that turned out to be smaller than the design assumption. The measurement revealed the gap. What closed it was not a rebuild but a reallocation — one that the operational data had no mechanism to surface, and the human outcome data made visible in its second review cycle.

This last question is in some ways the most consequential. The first two reveal whether the technology is working for the person as designed. The third reveals whether the design itself was right. An organisation that measures human outcomes consistently will eventually discover that the system is not reaching the range of people who actually use it, because no system does, at first. What it does with that discovery is what separates an organisation that builds trust from one that builds a persuasive story about its metrics.

The measure your organisation has not made

The case Sam’s organisation makes is not that human outcome measurement is complicated. It is that it requires a decision most organisations have not made: to know the answer when the answer is not flattering. The gap that decision would close is specific. So are the conditions that keep it open.

• The starting point is the primary metrics your organisation uses to assess the success of its technology. Do any of them measure whether the technology made someone’s life easier from their perspective, not the organisation’s? If not, what would the closest available measure be? And if that measure exists but is not in the primary review, that absence is worth explaining.

• If you are responsible for a technology measurement approach: when did you last present a finding to leadership that was uncomfortable, that revealed a gap between system performance and the person’s experience? What the culture does with that kind of finding is the real test of whether the measurement matters.

• If you could add one human outcome measure to your organisation’s technology dashboard, something that would tell you whether the technology is working for the people it serves, what would it be? The answer to that question usually already exists somewhere in the organisation. The question is whether it is being counted.

• The organisations that measure what matters build more trust with the people they serve than those that optimise the metric. Is that trust worth the investment in harder measurement? If your organisation is not making that investment, the reason is worth examining because it is usually not that the measurement is impossible.

The gap between system performance and human outcome is not a technical problem. It is a measurement decision. Most organisations choose the comfortable measurement. The gap stays.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.