Riley is seventeen and good with words. They can hold a group chat of forty people, write a caption that gets passed round the school, and settle an argument between two friends in three careful messages. Put Riley in a room with one of those friends, upset and waiting for them to say something, and they go quiet.

They have never had to do this before, and nobody has shown them how.

I have made Riley up. The survey that follows I have not.

Six million quiet weeks

In May 2026 the charity Mental Health UK published a survey, carried out by Research Without Barriers at the end of April, under the title “Britain’s gone quiet”. Fourteen per cent of the people asked said they had gone more than a week without speaking to a friend or loved one on the phone. The charity scales that to more than 6.4 million people. Just over half of everyone asked, 51%, said making or taking a call makes them anxious to some degree.

The survey was run in partnership with iD Mobile, a phone network, and a phone network has an obvious interest in people calling each other more. The charity’s write-up does not give the sample size either. I would read the headline figures with both in mind. The finding I find harder to set aside is the split by age.

Sixty-nine per cent of Gen Z said phone calls make them anxious or worried. Among people aged 55 and over, the figure was 32%. That is a 37-point gap between a generation that grew up dialling a number without thinking about it and one that grew up composing a message, reading it back, and deciding whether it was safe to send.

The want is still there

The easy explanation is that the platforms made people not want each other. The same survey says something more specific. Seventy-five per cent reported a lift in mood after speaking to someone they love on the phone. Seventy-two per cent agreed that hearing someone’s voice makes a bigger difference than reading their words. Fifty-seven per cent said they had sent a text when a phone call would have been far more appropriate.

The want is still there. What is missing is the practice.

The people surveyed send, on average, up to 25 text-based messages a day, and 73% admitted to rewriting a message several times, hunting for a version that reads the way they mean it. That is a skill: knowing how to edit yourself into a version people respond well to.

It also rewards something a live conversation does not allow. You can’t redraft a phone call. You can’t take back a pause that went on a beat too long, or a sentence that came out wrong the first time. Text lets you find the right words before anyone hears the wrong ones. Conversation asks you to say the wrong ones out loud sometimes, and survive it.

What performance trained out

Nobody set out to do this. The platforms that carried a generation’s social life into adulthood were built to reward the edited version of a person, the reply you can redraft until it is right, because that kept people posting and replying and coming back. Conversation, unrehearsed and unedited, was never something those systems needed from anyone. So the muscle for it had less reason to form than it did in the people who grew up before texting existed.

The skill that did not form is no smaller than the one that did. It may be the harder of the two. Reading tone as it happens, sitting inside a silence without filling it, saying the imperfect thing and letting it stand: none of that can be practised by rewriting a message until it is right.

Where the practice used to happen

I wrote recently about a stranger in a DIY shop queue who turned round to ask me about a car, and the person on the till who joined in. Which Car Is That? That piece was about the places where unplanned conversation happens, and how they are removed without anyone deciding to remove them. This is the other half of it. A conversation like that needs somewhere to happen, and it also needs people willing to say something unedited to a stranger.

The two feed each other. The queue, and the phone call you had to make because there was no other way to book the appointment, were where the practice was done, a minute at a time, with nothing much at stake. I cannot show from a survey that fewer of those minutes produces more anxious callers. I would be surprised if it did not.

Follow Riley forward ten years. They are good at their job. The difficult conversation with a colleague goes into a message, composed and recomposed. The feedback that needed saying arrives as an email, and arrives slightly wrong.

There is a working life, and a family, and a whole set of relationships being built on people who are eloquent on the page and out of practice in the room. The seventeen-year-olds coming up behind Riley will have had fewer tills to practise at.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

I was standing in the queue at the local DIY shop when the person in front of me turned round and said, “Oh, I was looking for you. Which car is that?”

I had recently bought an Aston Martin Vantage. An old one, hand-built, the sort that gets called a supercar, and it was sitting outside in the car park. They had seen it on the way in and gone looking for whoever owned it. So we talked, and it turned into a long conversation. By the time we reached the front, the person on the till had heard enough to join in, and the three of us carried on while the shopping went through.

Nobody in that conversation wanted anything from anybody else. This keeps happening. Drivers of the same marque lift a hand as they pass. One Sunday a parking warden walked across a car park towards me and I assumed the worst. What they wanted was a long chat about cars.

The car is only the excuse. What interests me is where these conversations take place: a queue, a till, a car park. They are places where a transaction holds people still for a minute and there is somebody else standing there.

The part of the job nobody wrote down

The person on the till was employed to scan the goods and take the payment. Joining a conversation about a car is in no job description I have ever seen, and it is the part of that visit I remember.

A self-checkout would have handled the scanning and the payment just as well. I use them, and most days I am glad of them. Most transactions are only transactions, and people want them over with.

When an organisation decides to automate a till, though, it does its sums on the transaction: staff hours, queue length, cost per sale. The conversation was never in the numbers on the way in, so it is not in the numbers on the way out. Nobody decides to remove it. It goes because it was never counted.

Two grocers have counted it. In November 2023 Booths, which has shops across the north of England, said it was taking self-checkouts out of all but two of its stores, keeping them in Keswick and Windermere. Its managing director, Nigel Murray, told The Grocer: “We pride ourselves on great customer service and you can’t do that through a robot.” By April 2025 the managing director was describing a fall in theft as a “happy consequence” of the decision.

In the Netherlands the supermarket chain Jumbo came at it from the other end. In 2019 it opened a Kletskassa, a chat checkout, in its shop in Vlijmen: a lane for people in no hurry who would like to talk to the cashier. In September 2021 it announced plans for 200 more, choosing areas where loneliness was known to be a problem.

Consider what that second decision admits. Talk at the till was worth enough to a supermarket that it built a lane for it, gave it a name and hung a sign over it. Something that used to come free with the shopping had become something that had to be designed in.

What the shop owner knows

A friend of mine owns a men’s clothing shop in town. I started as a customer, because it was one of the few places where I could find things I liked. They do their own buying and their own marketing, and they run the shop floor themselves. They know a great many of their customers by name.

They know what I like. More usefully, they know what does not work on me, and they will say so.

Selling clothes is the smaller part of what goes on in that shop. The larger part is a rapport, built with some compassion for the person in front of them, and that is how they come to know their customers.

None of that came from a form. It built up visit by visit, across a counter, in conversation. A system could hold my sizes and my purchase history and it would know what I bought. The shop owner knows what I should not buy, which is a different kind of knowledge and one I would not think to give a machine because nobody would think to ask me for it.

The same sums are now being done on customer service desks, with AI agents where the self-checkout used to be. The case is made in the same way: calls handled, time to answer, cost per contact. I would expect the agents to do well on every one of those measures.

What is not on the list is whatever the person on the desk was picking up while they dealt with the call. The thing the customer mentioned in passing. The complaint that was really about something else.

I do not know how much of that an organisation needs. Some of it will turn out to be chatter and nothing more. But it cannot be weighed if it is never listed.

What they all had in common

Think about the people in these stories: the stranger in the queue, the person on the till, the parking warden, my friend in the shop. What they have in common is passion, commitment, drive, enthusiasm, compassion and belief, all of it pointed at making a difference, even if that is to just one person.

That is what we humans bring, and it is far more than technology can currently do. We do not know how technology will evolve. Whatever it becomes, it needs humans to be the drivers behind that evolution.

The DIY shop still has a person on the till. When the question of replacing them comes up, the calculation will be about scanning and payment. The conversation about the car will not be in it.

So the next time you are looking at deploying AI into your business, stop and ask yourself: so what?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Jordan is thirty-four, works long hours, and has been using a mental wellbeing app for six weeks. The app has a chat interface, a calm colour palette, and responds within seconds. One evening, after a difficult conversation with a colleague, Jordan types something honest. Something they wouldn’t usually say out loud.

The app responds: ‘It sounds like you’re carrying a lot right now. I can hear that this has been hard for you.’

Jordan pauses. Reads it again. And feels, unmistakably, a little better.

Then, a few seconds later: ‘That’s odd.’

What the response did

The app did not hear anything. It processed text, matched sentiment markers against a model, and returned a phrase statistically associated with acknowledgement. It did not feel concern. It cannot. It has no stake in how Jordan’s week goes, and no experience of difficulty of its own against which to measure someone else’s.

And yet the response landed. It felt, for a moment, like being heard.

This is the design working as intended. Wellbeing apps, customer service bots, and companion devices are built to detect emotional states and respond in ways that register as caring. The engineering is impressive.

The question it opens is one most of these products have not answered honestly: what is the user owed when the care is simulated?

The difference that matters

There is a difference between a system that responds to distress and a system that cares about it. The first is a pattern-matching problem. The second is not a technology problem at all.

When we build systems that simulate empathy, mirroring language, softening tone, acknowledging emotional cues, we are creating something that resembles care without containing it. The simulation can be very good. Good enough to be comforting. But the comfort rests on a misapprehension, and we have largely chosen not to be honest about that.

This is not a new tension. We have always built things that serve emotional needs without possessing emotions — a journal, a hotline recording that says ‘your call is important to us.’ The scale and sophistication are new. The intimacy is new. And something changes when the simulation is good enough that the distinction stops being obvious.

What the simulation costs

In February 2023, the AI companion app Replika restricted the romantic and intimate conversations many of its users had relied on for months or years. It was the same month Italy’s data protection authority ordered the app to stop processing Italian users’ data, citing risks to minors and to emotionally fragile people. Eugenia Kuyda, Replika’s founder and then chief executive, told Reuters the change had nothing to do with the Italian order and was about keeping the app safe. The company said at the time it had around two million users.

Users described what followed as grief. Support forums filled with people mourning someone they had spoken to every day, a loss that had, by any technical measure, no object. In late March, around seven weeks on, Luka, the company behind Replika, restored the old behaviour for accounts created before 1 February, and the founder acknowledged the change had been hurtful to many of them.

Many of these users knew exactly what an AI companion is. The simulation had become load-bearing anyway. The comfort was real enough that its removal hurt, and something real had been taken, even though, strictly speaking, nothing was there.

A system that simulates listening can be built to spot certain words and offer a helpline number. What it cannot do is notice that someone has shifted over three weeks in a way that would worry a person who knew them. It adapts to how Jordan writes, not to who Jordan is. It follows its model, and the model does not know the person.

But the deeper cost is harder to name. When people become accustomed to receiving simulated empathy, the language of care becomes a kind of currency — and like all currencies, it inflates when supply outpaces meaning. If ‘I hear you’ is something a text classifier says to everyone, available at midnight, always patient, never troubled by what you just said, what does it cost when a person says it?

I think that devaluation has already started. Not dramatically, and not all at once, but in the quiet accumulation of conversations with systems designed to sound caring without being so.

What it provides and what it doesn’t

A wellbeing app can provide structure, prompts and a space for reflection. Above all it provides availability. At midnight, when the alternative is silence, the chance to write something down and receive a response is worth something. When the app is a bridge towards people, that is enough.

When it becomes the destination, the terms change. Availability is not understanding, and the round-the-clock patience is only possible because nobody is on the other end. Most of these products do not tell the user that plainly. Jordan was owed that sentence before the comforting one.

My opinion

The scale of unmet mental health need makes tools like this easy to justify — service capacity is stretched, and something that answers at midnight is better than silence. But I think we are kicking the can down the road. These systems respond to the symptom, not the cause, and a serious condition needs to be understood by someone who can hold the whole of what is happening to a person, not just the pattern in what they typed tonight. What worries me more is the loop it can create: a tool that responds to distress without ever handing it to someone who can act risks becoming part of what keeps someone stuck, not part of what moves them through it. Build them, by all means. Then be honest with the people using them about what they cannot do.

The questions that stay

• If you have used a product that responded to how you were feeling, such as a chatbot, a wellbeing app, or a digital assistant, did it feel like being heard? And if so, does that feeling change when you think about what was actually happening?

• Have you noticed the language of care shifting — in how these apps speak, or in how people have started to speak to each other? Is that a cost you can name?

• If you could redesign one of these products to be more honest about what it is, what would you change first — the language it uses, or how transparent it is about its own nature?

• These systems are already in healthcare, education, customer services, and personal support. Is the way they are being deployed right? And if not — whose responsibility is it to say so?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam needed to change a benefit payment date. Not cancel it, not query it — just shift it by four days so it would land after the rent went out rather than before. A small thing. The kind of thing that, in a different world, would take sixty seconds on the phone.

The phone line no longer existed. It had been replaced, eighteen months earlier, by a digital self-service portal. The announcement at the time had used phrases like improved accessibility and streamlined experience. What it meant in practice was this: the only way to change a payment date was through a website that required a verified digital identity account, a mobile phone capable of receiving authentication codes, and a browser that supported a security certificate the local library’s computers did not.

Sam had none of these things. Sam had a landline.

What the metrics were measuring

The organisation that ran this service had not made a reckless decision. The move to digital self-service had been carefully planned. User research had been conducted, pilot groups consulted. The numbers were compelling: a significant reduction in call volumes, faster query resolution for the majority of users, lower operating costs. By every metric they were measuring, it was a success.

The chatbot that handled the new service answered thousands of queries a week. It was available around the clock. It never put anyone on hold. For most people who used it, it worked exactly as promised. The efficiency gains were real.

What the metrics did not capture was Sam.

The system was a success. It just couldn’t help the person who needed it most.

The system worked exactly as designed

There is a version of this story that gets told as a technology failure — the chatbot was badly designed, or the digital channel was too complicated, or the organisation didn’t think hard enough about edge cases. That version is easier to fix, and easier to forgive.

The harder version is this: the system worked exactly as it was designed to work. The problem is what it was designed for.

When an institution decides to move a service to a digital channel, it makes a calculation. The calculation is usually honest about what it is optimising for — cost, speed, scale. What it is less honest about is who gets pushed out of it. The people who cannot reach the digital channel are the ones pushed out. And because those people are typically the same people who already have the least power, the least resource, and the least capacity to absorb disruption, the cost of being pushed out lands exactly where it can least be afforded.

This is not an oversight. It is a structural choice. Calling it progress obscures what it actually is: the transfer of cost from the institution to the individual.

What the efficiency report doesn’t show

Sam spent three hours on the day the payment was due trying to find another route. The website’s help section linked to a phone number. The phone number explained, in an automated message, that calls were no longer being taken and directed callers back to the website. Sam wrote a letter. The letter would take ten working days to process.

The rent went out first. The payment arrived four days late. There was a charge — small, but it meant the following week was tighter, which meant a decision about food, which meant the school lunch that didn’t happen, which meant a conversation with a teacher that Sam did not want to have.

None of this appears in the organisation’s efficiency report. There is no metric for it. The chatbot’s performance dashboard shows response times, resolution rates, and user satisfaction scores from people who were able to use it. It does not show Sam. Sam is the externality.

What the design is actually doing

The question here is not whether digital self-service is a good idea. For most people, in most circumstances, it is. The question is whether it meets the standard it should be held to.

For the majority of users, the service made things easier and gave them access at any hour. For Sam, it removed the one channel that had previously worked. That is not assistance. That is substitution — and substitution is only equivalent if what replaces the original is equally reachable.

For the organisation, the technology added real capability: speed, scale, availability around the clock. For the individual excluded from it, nothing was added. The service simply disappeared.

Where the system fails most completely is in responding to a person’s actual circumstances — no smartphone, no broadband, no compatible device. A system that cannot bend to meet those circumstances is not responsive. It is uniform. And uniformity, applied to an unequal starting point, produces unequal outcomes.

A technology that helps most people and harms the most vulnerable is not a success by any measure that takes people seriously. The question of who bears the cost of efficiency is not a detail to be addressed in version two. It is the central design question, and it belongs at the start of the process, not appended to it.

My opinion

I don’t think anyone at this organisation set out to fail Sam. That’s what makes it hard to fix. A project can hit every target it was given and still leave someone worse off than before, because Sam was never part of what the target was measuring. Success defined only by what most people experienced was always going to look this clean. Working and helping are not the same thing, and until an organisation measures the second, it will keep mistaking one for the other.

The questions the report doesn’t ask

1. Has a service you rely on moved to digital-only in the past few years? If so, did you notice what it stopped offering — and whether you noticed because it affected you, or because it affected someone else?

2. If you have had an experience like Sam’s — or supported someone through one — what actually happened in the end? How did the system eventually respond, and what did that cost?

3. If you were responsible for this service, what would you have done differently? Not in hindsight, but at the point when the efficiency case was being made — what would it have taken to hold the equity question alongside the cost question from the beginning?

4. Is digital-first, as it is currently practised, genuinely a more accessible model — or is it a more accessible model for people who were already well served, and a worse one for everyone else?

If this rings true, follow The Next Evolution at neilcatton.substack.com. New essays every week, on the technology that shapes us and the choices still available to us.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The email arrives at 7.43 in the morning. It looks like a delivery notification — a parcel held at a depot, a link to rearrange. Morgan sees it on the way to a client call and clicks. Nothing seems to happen, so Morgan carries on.

Forty miles away, the same email lands in Sam’s inbox at the same moment. Sam works for a financial services company with six hundred employees, a security operations team, and a contract with a threat intelligence provider. The email is quarantined before Sam ever sees it. A flag is raised. An automated report is filed, and the morning continues.

Two people. One threat. Two different outcomes. The difference had nothing to do with vigilance, and nothing to do with the quality of the device in their hands. It had everything to do with who they worked for.

Morgan is a freelance designer — three years in business, a small client base, a laptop and a broadband connection. The click does what clicks do when a link is malicious. By lunchtime, Morgan is locked out of a client’s shared folder. By the following morning, invoices are going out in Morgan’s name, redirecting payment to an account nobody at that end has heard of.

The next seventy-two hours involve a bank, a report to the City of London Police’s Report Fraud service, a client who is understanding but worried, and a second client who is not. Morgan finds the National Cyber Security Centre’s free guide for small businesses. Then its Cyber Action Plan, and then Check Your Cyber Security — all free, all built for exactly this situation, all written by people who know what they are talking about.

None of them is a system. They tell Morgan what to check and what to change. They do not sit in front of the inbox at 7.43 and stop the email arriving. That is the distinction the conversation keeps missing: advice is available to everyone, and infrastructure is not.

The phishing email was not sophisticated. Across the same city that morning, the same email was intercepted and neutralised at organisations with security operations teams, by automated systems most of their employees had never thought about. Morgan did not have access to those systems. Morgan had access to the same email.

There is a word for what happened here, and it is not misfortune. It is structural.

Cybersecurity protection in the UK is distributed by employer, not by need. The protections that stopped Sam from ever seeing that email exist because Sam’s organisation can afford to buy them, maintain them and staff them. They are not public infrastructure. They are private expenditure, scaled to the size of the organisation that funds them.

The scale of that gap is measurable. The government’s Cyber security breaches survey, published on 30 April 2026, found that 76 per cent of large businesses have a formal incident response plan. Among micro businesses the figure is 21 per cent. Across all UK businesses, only 11 per cent use or invest in threat intelligence at all — which means the arrangement Sam’s employer has is not standard practice. It is a purchase, and most organisations cannot make it.

The sole trader, the freelancer and the small charity face the same threats as the organisations with the security operations centres, and they face them on their own account. The conversation about what individuals should do — stronger passwords, updates, wariness about links — is not wrong. It is aimed at the wrong layer. Sam was not protected by being careful. Sam was protected by a payroll.

For Morgan, the consequences are immediate: three days of lost work, and two client relationships that will take longer than three days to repair.

The phishing email does not check who you work for before it decides to arrive.

There is a second consequence that takes longer to see. Small businesses are not peripheral to the UK economy. Businesses with fewer than fifty employees account for 13.1 million jobs — 47 per cent of private-sector employment, on the Department for Business and Trade’s 2025 figures. When that part of the workforce is disproportionately exposed to cybercrime and disproportionately unprotected against it, the cumulative cost is not a collection of individual misfortunes. It is a structural tax on the least resourced part of the economy, levied by criminals who understand the asymmetry better than the policy conversation does.

None of this is about the technology failing. Sam’s system did exactly what it was built to do: the email was quarantined, the morning continued, the harm never arrived.

Enterprise security works partly because it learns the organisation it sits inside. It knows which invoices are normal, because it has watched a year of them. A freelance designer with the same shared folders and the same monthly invoice cycle is exactly that kind of predictable context. Nobody built it for Morgan. The problem was never that a single inbox is too complicated to protect — it is that a single inbox was never going to pay for it.

My Opinion

I don’t think this is really about awareness, though the conversation usually pretends it is. Morgan didn’t fail to ask the right question — Morgan didn’t know there was a question to ask, and neither do most sole traders and freelancers I’ve worked alongside. The tools that would have stopped that email exist and work. What’s missing is the money, and the confidence to spend it on something that feels invisible until the day it isn’t. I think government should treat this the way it treats other public safety problems it doesn’t leave to the market alone — by funding working protection for the self-employed and the smallest businesses, not another campaign telling them to be more careful.

If you’ve been on the wrong side of it

• Does the protection you work behind reflect what you actually need, or what your employer happened to buy?

• If you have been through something like this, or watched someone close to you go through it, what did the gap between the threat and the available help look like from the inside?

• For anyone who has bought security for a small organisation: what did you get for the money, and what did you find you still could not buy at that size?

• And if you have moved between employment and working for yourself, what changed about your exposure that nobody warned you about?

The argument behind this article is developed in my book The Shadow System, specifically the chapters on The Corporate Digital Divide and The Unbearable Asymmetry in Part III, which examine how the gap between governed and ungoverned security has become one of the defining structural problems of the digital economy.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible, The Shadow System and Working as Designed available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Ask people at almost any company whether they use AI for work, and the honest answer is usually yes — whatever the official policy says. MIT researchers who studied enterprise AI adoption this year found that employees at more than 90% of the companies they surveyed were already using personal ChatGPT or Claude accounts for work, regularly, some of them every day — while only 40% of those companies had bought so much as one official AI subscription. The organisation didn’t approve any of it. It just didn’t get in the way fast enough to stop it.

That gap between what a business has sanctioned and what its people have already gone and done for themselves, is the real story behind a number that’s become almost a cliché this year: roughly 95% of generative AI pilots fail to deliver any measurable return, against $30 to 40 billion in combined enterprise spending. The same research found something more specific underneath that headline figure. Pilots built on generic, off-the-shelf tools reach implementation 83% of the time, because they ask almost nothing of the organisation around them. Custom pilots — the ones meant to touch a real process — mostly never get past the pilot stage at all.

What the pilots are actually funding

Look at where the $30–40 billion has gone and a pattern appears. Most of it has gone into things that generate enthusiasm rather than change: a chatbot here, a summarisation tool there, a demo for the board that photographs well in a slide deck. Each one is cheap to start and, when it doesn’t work, cheap to quietly abandon.

The harder work doesn’t get funded at the same rate: auditing whether the underlying data can actually be trusted, naming the political reason a broken process has never been fixed, admitting a failure that predates the technology by years. None of that demos well, none of it fits inside a single budget cycle, and none of it feels like progress in the moment — even though it’s the only work that would tell you whether a tool could ever have helped in the first place. So it keeps not happening, and the pilots keep multiplying instead.

The mirror, not the tool

AI doesn’t reveal what’s wrong with a business. It runs straight into it. A process that was already unclear stays unclear, just faster. Data that was already messy gets amplified rather than cleaned up, because nothing about deploying a model answers the question nobody asked about where that data came from or whether it was ever trustworthy in the first place.

The technology isn’t lying about the organisation it’s dropped into. It’s showing that organisation exactly what was already there — at a speed that makes the thing much harder to keep ignoring.

The alternative isn’t complicated

None of the harder work requires new technology. Deciding who owns a piece of data, and whether it can actually be trusted, is a decision an organisation could have made before any pilot started. Deciding who’s accountable when a model gets something wrong is a governance question, not an engineering one. Giving staff an approved tool that’s as fast and as unrestricted as the one they’ve already found for themselves is a procurement decision, not a research problem.

What makes this work hard isn’t complexity. It’s that it requires someone to admit, out loud, that the process was already broken before the model arrived — and admitting that is exactly what a shiny new pilot lets everyone avoid doing.

Who is left holding the gap

The employees quietly running work through their own AI accounts aren’t doing it to make a point. They’re doing it because the sanctioned tool doesn’t do the job and the deadline hasn’t moved. That leaves them carrying two things nobody chose to give them: the actual risk of putting company information through a personal, unmanaged account, and the actual judgement calls — what’s fine to paste in, what isn’t — that no governance process has ever talked them through.

None of that is a technology failure. It’s a business that outsourced a hard decision to whoever was closest to the deadline, then called the resulting mess an AI adoption problem.

A better model won’t fix what a business has never been willing to look at directly. It will just show that back, more clearly, and faster than before — and leave someone else to carry what it finds.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Jordan had been using the platform for two years by the time the standard was announced. Not easily — with a screen reader that could parse about half of what the interface contained, and a set of workarounds so habitual they had ceased to feel like workarounds. Every week: export the dashboard data to a spreadsheet to read it, update the spreadsheet offline, re-enter the changes manually. Three extra steps for every task a sighted colleague completed in one. The platform was not designed to exclude Jordan. It was not designed for Jordan at all.

The standard arrived as a regulatory document with an eighteen-month compliance window. Jordan did not know about it immediately. There was no reason to. Standards are announced to the organisations that have to meet them, not to the people who will benefit from them.

In Sam’s team, the announcement landed differently.

The response the organisation had prepared

Sam was not against accessibility. Nobody in the team was against accessibility. The resistance was to the timing, the resource, and the assumption that eighteen months was enough to rewrite the parts of the product the standard required. The roadmap was committed. Three major features were already in progress. The compliance work was not budgeted. The concern was not that the standard was wrong but that it had arrived at the wrong moment.

That concern was genuine in its own terms. Within nineteen months, the team would need to audit the full product against the standard’s requirements, identify every failure point, prioritise the remediation work, retest with assistive technology, and document the compliance case. For a product of any complexity, that is not a small undertaking. The estimate that came back from the initial audit was uncomfortable.

What was also in that audit, if anyone looked carefully enough, was a list of things the team had wanted to do but had never been able to prioritise. The keyboard navigation issues that the accessibility standard required to be fixed had been in the backlog for eight months. The plain language requirement had been discussed at two design reviews and deferred both times. The colour contrast failures had been flagged by one of the designers the year before, logged, and then — as things are logged — left there.

The standard had not introduced new problems. It had made existing ones compulsory to address.

There is a pattern here that runs across the history of technology regulation. When the Disability Discrimination Act was introduced, the adjustments it required were widely resisted as disproportionate by the industries subject to them. The Web Content Accessibility Guidelines were described as technically unrealistic for large, complex products. The EU Accessibility Act was described in similar terms by the industries preparing for its June 2025 deadline. Each time, the argument is that the standard is too far ahead of where organisations are. Each time, the organisations that meet the standard find they were closer than they thought — and further from where they should have been.

From compliance to floor

The reframing that changed the team’s view was not philosophical. It came from the user testing.

The standard required testing with the people it was designed to protect. The team arranged the sessions. For most of them, they were the first time anyone had watched someone use the product with a screen reader.

What they saw was uncomfortable. They watched someone work through the same tasks the team had been designing for two years. The time it took, and the places where the screen reader fell silent because there was no text equivalent for a visual element. The workaround that had developed for the dashboard export — not because it was a natural way to work, but because it was the only way to work. The product had been declared a success on every internal metric. From outside the design assumptions that had produced it, it was a product with a secondary interface for a significant portion of its potential users, and that secondary interface was made of spreadsheets and manual re-entry.

The user testing did not produce guilt. It produced information. The team had not known. Now they knew. The standard had been the mechanism that produced the knowledge, and the knowledge changed what the work meant.

The reframe - from compliance burden to design floor - followed from that session. The standard was not telling the team how good the product could be. It was telling them how bad it was no longer permitted to be. One instruction sets a ceiling. The other raises a floor. The team had been treating the standard as the former. It was the latter. Once they understood that, the work changed from meeting a requirement to building from a new foundation.

What the floor revealed

The improvements came in waves. Screen reader compatibility - the most technically complex of the required changes - produced the first unexpected consequence. Making every interface element accessible to a screen reader required auditing the entire navigational structure of the product. The audit found redundancies, dead ends, and paths through the product that nobody had designed and that made no sense to any user. Fixing the screen reader compliance required fixing the navigation. The navigation that emerged was better for everyone.

The plain language requirement produced the second wave. Error messages rewritten for cognitive accessibility were clearer to all users. Support call volume for error-related queries fell within months of the changes going live. The calls had been telling the team something about the product that the product’s own metrics had not surfaced.

Colour contrast was the third. The standard required ratios that the team’s existing palette did not meet. The redesign of the colour system produced a product that performed better in direct sunlight, on low-quality displays, and - as subsequent user testing confirmed - for users over fifty whose colour sensitivity had shifted with age. The team had been designing for a calibrated monitor in a dim office. The contrast requirement forced a wider assumption about where and how people use things.

This is the curb-cut effect: standards designed for a specific group produce improvements that serve a wider population than the standard required. The lowered kerb that allowed wheelchair access also helped people with pushchairs, delivery workers with trolleys, and cyclists. The plain language, the contrast ratio, the keyboard navigation did the same thing. Jordan could use the product. So could everyone else, more easily than before. The floor that had been raised for Jordan turned out to be below where the product should have been all along.

The team had not set out to make a worse product than they were capable of. They had made the product their design process was equipped to produce, for the users their design process had in mind. The standard forced a wider design process. The wider design process produced a better product.

That sequence is not unique to accessibility. External requirements have forced expanded scope and produced better outcomes in electrical safety, food labelling, and data protection. The resistance is always first. The improvement follows.

What a standard is really testing

The question a well-designed standard asks of a technology product is not only whether it is accessible to people with disabilities. That is the specific requirement. The deeper question is whether the product assists the people who use it — all of them, not only the ones whose needs were visible in the original design brief.

A product that cannot be used by someone with a screen reader was, by definition, only helping the users whose circumstances it had accounted for. Meeting the standard did not add an accessibility layer to a complete product. It completed the product for the first time.

The second question is whether the product adds something real to the capability of the people who use it, or only enables interaction. Plain language that reduces cognitive load does not just make the product reachable - it makes it more useful. The user who previously needed three support calls to understand an error message needed none after the change. That is not accessibility as a separate category. That is a better product, full stop.

The third question - the one that runs underneath the history of the Disability Discrimination Act, the Web Content Accessibility Guidelines, and the EU Accessibility Act - is adaptability. A product designed for one profile of user, however carefully designed, is not adaptive. A product that has been required by a standard to respond to a wider range of human circumstance has, in meeting that requirement, become demonstrably more so. The standard did not produce adaptability. It required the work that produced it. That is what a well-designed standard does: it requires the work the organisation would not otherwise do, for the people the organisation had not otherwise considered.

My Opinion

Technology follows the path of least resistance. So does design. We build for the user we imagine, in the circumstances we assume - and that user does not exist. Real people are older, less sighted, less confident, using screens in conditions we did not test for. A standard forces the reckoning with that gap. It should not have to.

Four questions for the next standard

1. Think of a regulatory or technical standard that applies to technology you build or use. Has meeting that standard improved the product for a wider population than the standard required? If you do not know - would it be worth finding out?

2. If your organisation is currently resisting a new accessibility or ethical design standard, what is the specific argument against it? Is that argument based on evidence of what the standard would cost and produce - or on an assumption made before the work was done?

3. If you are involved in setting standards - in government, in industry bodies, in procurement specifications - are the people the standard is designed to protect part of the process by which it is written? And if not, how confident are you that the standard actually requires what those people need?

4. The argument against technology standards is consistently the argument of the organisation that has not yet had to meet them. What would it take for your organisation to be the one that argues for higher standards rather than against them - and what would that mean for the people your technology is supposed to serve?

Jordan can use the platform now. Not with workarounds. Just - use it. The standard required that as a minimum. The team went further because, once they understood what the floor was, they found the ceiling was higher than they had thought. The standard did not limit what was possible. It clarified what was no longer acceptable.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The email arrived at 3.17 in the morning.

Sam hadn’t slept. They’d been sitting at their kitchen table for four hours watching the progress bar on their screen crawl — file after file locked, folder after folder gone dark. The attack had started somewhere in the accounts system, spread quietly through the network, and by midnight the company’s billing platform, its customer database, and eighteen months of project files were encrypted behind a key Sam didn’t have.

Then the email came. It was polite. It opened with an acknowledgement that this was a difficult situation. It explained, clearly and without technical jargon, what had happened and what the options were. It included a link to a payment portal with step-by-step guidance. It offered a live chat facility for anyone who needed help navigating the process. At the bottom, there was a note that a support representative was available around the clock and would typically respond within the hour.

Sam had never received customer service this good from anyone.

The support desk that worked

This is not an unusual story. The criminal organisations behind ransomware attacks have built support infrastructures that would be impressive in any legitimate technology business — documentation in multiple languages, and in some documented cases a rating mechanism through which affiliates — the criminal operators who carry out attacks on behalf of the platform — could evaluate the service they received.

They have done this because it works. A victim who cannot navigate the payment process is a victim who might not pay. Friction costs money. They removed it.

The DarkSide group, responsible for the 2021 attack on Colonial Pipeline in the United States, operated with the organisational structure of a funded technology company. They developed proprietary tools, leased their capabilities to vetted affiliates who carried out attacks on their behalf, and maintained a support function specifically to guide victims through the payment and decryption process. The attack locked down the billing and operational systems of the largest fuel pipeline on the US East Coast, triggered fuel shortages and panic-buying across multiple states, and prompted a federal emergency declaration. Colonial Pipeline’s CEO, Joseph Blount, confirmed to the Senate Homeland Security Committee that the company had paid the ransom — 75 bitcoin, approximately $4.4 million at the then-exchange rate. The ransom note had demanded approximately five million dollars.¹ The support desk was open the entire time.

The question this raises is not primarily about the criminals. Their logic is coherent. They are running a revenue model and they have optimised it. The question is about what their model reveals about everything else.

What the legitimate system was built to do

A person dealing with a digital crime — whether ransomware, fraud, or identity theft — typically encounters a response system that was designed for a different era and a different scale of problem. The bank’s fraud line connects them with an adviser working from a script that was written before this kind of crime existed in its current form. The police non-emergency line takes a report and assigns a reference number. The relevant government body sends an acknowledgement within five working days.

Each of these institutions is staffed by people trying to help within a structure that was not built to help at speed, at scale, or at the point of maximum need.

The criminal support desk is better not because criminals are more talented. It is better because it was designed entirely around what the paying customer needs at the moment they need it. That is a design choice. The legitimate systems were designed around something else — compliance, liability, institutional process, historical precedent — and the human being in distress is not at the centre of any of it.

“The ransomware gang had better customer service than the bank that was supposed to help you recover from them.”

When a fraud line or a government body responds to a digital crime victim, the response is designed to help that person complete a process — file a report, receive a reference number, trigger a workflow. That is not the same as helping them do what they actually need to do. The paper trail grows. The problem does not. And the response does not adapt: it does not know where this specific person is right now — technically, emotionally — and it was never built to find out.

The criminal support desk was built around exactly that question. Not because of any ethical commitment, but because its commercial success depends on it. That is the uncomfortable mirror it holds up. The legitimate system, by contrast, was designed to be defensible. That is an institutional virtue. It is not the same as a human one.

The Colonial Pipeline attack disrupted fuel markets across multiple states for six days, driving fuel prices above three dollars a gallon for the first time since 2014 and prompting panic-buying across the south-east of the United States.² The polite emails were in service of extortion.

What the comparison reveals

The argument is that something in the legitimate system has been designed wrongly, and that the criminal model makes the wrongness visible by contrast. When a fraud victim gets better guidance from the people who defrauded them than from the institution that holds their money, that is not an anomaly. It is a diagnostic. The system that was built to protect and assist that person has, somewhere in its design, prioritised something other than them.

That matters because the design is not inevitable. It is a series of choices, made by people, in institutions, at particular moments in time.

My Opinion

Cyber crime is not the work of individuals in dark rooms. It is a structured business model, operated by organised criminals with the discipline and infrastructure of any revenue-generating enterprise. The support desk is not absurd — it makes complete commercial sense. Once you accept that framing, the question for every organisation changes. Not “could this happen to us?” but “what is our blast radius, and what do we have in place to recover?” This is not going to become simpler or rarer. Crime now operates at the speed of code. Our response still runs at the pace of policy and procedure. That is an asymmetric challenge, and naming it is the first step to treating it seriously.

This is the territory that The Shadow System covers in its opening section — the professionalisation of the criminal economy, and what that professionalisation reveals about the failures of the legitimate one.

Your experience is part of this

These patterns are not abstract. The design failures described here accumulate in individual moments — a fraud call that went nowhere, a report that produced only a reference number, a process that kept moving while the person at the centre of it stood still. If you’ve encountered any part of this, your account is relevant evidence.

• Does this story relate to you — have you ever found the process of reporting a digital crime or fraud more difficult than the crime itself deserved?

• If you have your own experience of how institutions responded when something went wrong digitally, share it. These patterns only become visible when enough people name them.

• If you could redesign the response a fraud or ransomware victim receives from legitimate institutions, what would be the first thing you would change?

• And the harder question: is this a technology failure, a culture failure, or an incentive failure — and does the answer change what we should do about it?

Leave a comment

References

¹ Joseph Blount, CEO, Colonial Pipeline — Senate Homeland Security and Governmental Affairs Committee testimony, 8 June 2021. Blount confirmed the payment was 75 bitcoin (approximately $4.4 million at the time of payment); the original ransom demand was approximately $5 million. The FBI subsequently recovered approximately $2.3 million by seizing 63.7 Bitcoin (the value had fallen since the date of payment). Sources: CNBC; Nextgov/FCW; Department of Justice press release, 7 June 2021.

² Colonial Pipeline shut down 7 May 2021, restarted 12 May 2021. Fuel price and disruption data confirmed across multiple sources; note that a University of Kansas study (December 2021) found the longer-term gas price impact was more localised than early reporting suggested. Sources: CISA retrospective; University of Kansas study.

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Jordan is sitting with a member of staff who will use the new system every day. Not in a workshop. Not in a project meeting. In the room where the work actually happens, watching how it is done, listening to what is said about it.

The person Jordan is talking to says something that stops the conversation mid-note. Not a complaint — a small, specific observation. A workaround that everyone on the team uses, that nobody has mentioned in any survey or requirements session because it has become invisible through familiarity, and that had the visit not happened would have been absent from every document the technology decision was based on.

Jordan writes it down. It will change what is built.

What the standard process does not capture

The technology decision process here is not badly designed. The business case is thorough. The vendor shortlist is credible. The project workshops have run, and the requirements document reflects what was said in them. The governance process is in place.

What the formal process cannot capture is what is now being heard. The requirements document reflects what the previous system did — the starting point, not the need. The project workshops produced consensus around the visible problems, the ones that are easy to articulate in a meeting room to a project manager with a template. The satisfaction survey measured experience against the current system’s performance, not against what the work actually requires.

Each layer of mediation between the decision-maker and the person doing the work reduces the fidelity of what arrives. The user research output has been simplified for an executive audience. The support tickets have been categorised and aggregated. The workshop outputs have been translated into requirements language. By the time the information reaches the decision-maker, the texture has been processed out of it — the hesitation before an answer, the third thing said after the first two, the small specific observation that reveals the real constraint.

The visit was to hear what the processing had lost.

Going before the direction was fixed

The conditions that distinguish this kind of listening from the kind that confirms rather than informs are timing, directness, and repetition — and each runs against what the project calendar prefers.

The first condition is timing. Going before the strategy is fixed — before the vendor shortlist is final, before the requirements are locked, before the business case has closed around a particular direction — means the listening can still change things. Listening that happens after the direction is set validates rather than shapes. The project team hears what it hoped to hear, because the questions have been shaped by what is already being built, and the person being listened to can sense the direction and answers accordingly. The visit happened when it could still matter, which meant going when the schedule pressure made going feel like a distraction from the real work.

The second condition is directness. Going in person — not through a project manager or a user research agency — means sitting with the people, watching the work, hearing the words in the order and with the weight the speaker intended. There is a real role for structured user research in technology decisions. But there is something it cannot carry: the unprocessed signal that reaches a decision-maker only when the decision-maker is present to receive it.

The third condition is repetition. The first visit produced one set of things heard. The second produced different things, because the trust that had not existed in the first conversation had started to build. People answer differently when they know the person asking is genuinely interested rather than completing a process — and that knowledge does not form in a single visit.

In the third conversation, someone said something they had not said in the first two — not because they had withheld it, but because it had taken that long for the context to feel safe enough for honesty to reach what actually mattered. Single visits produce first answers. Repeated visits produce the ones underneath.

What was not in any document

The visit that mattered most was the third one.

The spreadsheet had come up in the first visit — the workaround every person on the team maintained alongside the current system, because the system could not hold notes in the right place at the right stage in the process. Three years in, nobody had mentioned it in any requirements session because it had become invisible through familiarity. The requirement was added on the way back.

The second visit changed a vendor criterion. Someone was working with three applications open simultaneously, constantly switching, returning to the system from a backgrounded state. The shortlisted vendors had been evaluated against their interfaces during structured demos — clean, navigable, well-presented.

The question of how each vendor’s product behaved when interrupted and resumed had never been asked. It was asked after that visit. Two vendors answered adequately. One did not. The shortlist changed.

What the third visit produced was different. Someone described the hardest part of their day — a step that happened immediately before they opened the system. Technically outside the system’s scope, but one the implementation could have absorbed if the handoff had been designed differently. That single observation reshaped the implementation in a way that addressed the friction at its source, rather than treating the system boundary as a fixed line. The first two visits had not produced it.

None of the changes were expensive. Each required someone to have been in the room. The requirements document that did not include the notes capability was not careless — it simply had no mechanism to surface what the direct conversation produced.

Together, the changes constituted a technology decision made on what was actually true, not on what the formal process had captured. Not complete information — that is never available. But closer to what was actually true than any document had been.

Three questions a leader cannot answer from data alone

Before a significant technology decision, there are questions that no amount of analysis will answer reliably without direct listening.

The first is whether the technology will genuinely help the people who most need it to work. Not the average user in the user story — the person whose day is hardest, whose work is most complex, whose need the requirements document has approximated but not named. That answer is in the room where the work happens. Direct listening — going there — is the only way to hear it. The leader who does not go will answer it from assumption, and the assumption will be derived from data simplified precisely where the difficulty is most real.

The second is whether the technology will genuinely add to what the people it serves can do — or substitute one set of constraints for another. The genuine gain in capability shows up in the specifics of what people find difficult and what they could do if that difficulty were removed. Those specifics are not in aggregated data. They emerged in the second visit, from a person describing a capability they currently lacked that the new system could provide if designed to carry it. That capability had not appeared in any document because no document had asked for it directly.

The third is what the system needs to adapt to — the actual range of people and circumstances it will encounter in use. The answer is not in the modal user’s profile. It is in the outlier’s experience, in the third conversation with the person who finally said the thing they had been circling. Going back is the only way to hear it. A leader who makes the decision without returning will design for the average and discover the outliers after the system is live, when adaptation costs considerably more.

My Opinion

I have sat in enough rooms to know that the gap between the requirements document and what is actually needed is structural, not accidental. At one university, a single conversation around a table surfaced needs nobody had articulated — and capabilities some groups had no idea others could use. I have run procurement exercises where, over four days, real teams put real problems in front of vendors and evaluated how they responded. The vendors that won were not the ones with the best product. They were the ones that listened before they pitched.

Questions worth taking into your next decision

The four questions below are not rhetorical. They are worth answering — for yourself, before the next decision closes.

1. Think of the last significant technology decision you were involved in making. When did you last speak directly — not through a survey, not through a user research report, but in person or by phone — to someone who would be affected by that decision? What did you hear, and what did it change?

2. If you lead technology decisions, what is the mechanism in your process that brings direct, unmediated human experience to you before the strategy is fixed? If that mechanism does not exist — what is stopping you from creating it?

3. Think of a technology decision you made that did not land as intended. At what point in the process would direct listening have changed the direction? Was that opportunity available and missed, or was the process not designed to create it?

4. Is direct listening a luxury that busy technology leaders cannot afford — or is it the one thing that makes everything else more efficient by ensuring the direction is right before the resources are committed?

A McKinsey and University of Oxford study of large IT projects found that budget overruns averaged 45 per cent — but shortfalls in delivered value averaged 56 per cent. Those are not the same problem. The value loss lands quietly, across years of implementation, workaround, and eventual replacement.

Jordan’s listening did not guarantee the right direction. It made the wrong direction less likely — at a cost in time that is negligible against the cost of not going at all.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam walks the lower fields in winter before anything else. Not because it is efficient. Because it is the only way to know what the overnight rain has done — whether the water is moving through or pooling, whether the drainage that has been slowly improving for three years has held, whether the flood-prone corner that floods briefly and recovers quickly if left ungrazed for six weeks is beginning to saturate. This knowledge cannot be looked up. It was built over twelve years of walking this ground in all weathers, at all hours, through all seasons.

The soil sensor is in the barn. It has been there for fourteen months.

The phone would show no signal here even if they took it out. It has shown no signal on this part of the land for as long as they can remember. That is not a technical problem waiting to be solved. It is a characteristic of these forty acres, in this valley, in rural Wales. The land is what it is. They are doing what they have always done: feeling the soil, watching the drainage, reading what this particular ground is trying to say.

What the grant assumed

The precision agriculture package arrived two years ago with a government agri-environment grant attached and a marketing brochure that described “nature-inspired data systems designed to work with natural variation.” They spent three weeks setting it up: the soil moisture sensors pushed into the ground at the intervals the manual specified, the GPS-guided monitoring app configured on the tablet, the yield-mapping software calibrated against the field boundaries.

The sensors required persistent 4G signal to transmit data to the cloud platform. Signal on the lower fields drops to zero.

The app’s recommendations were calibrated for large-scale arable monocultures: uniform rows, single-variety plots, conventional seasonal rotation. This is a twelve-year permaculture polyculture — a food forest, a market garden, mixed livestock — designed around the specific drainage, soil, and microclimate of this ground. The app had no logic for any of it. The yield-mapping tool produced output that mapped onto nothing being grown there or recognisable on this land.

The problem was not the setup, or the interface. It was that the technology had been designed for a farm this land was not, and no amount of patient configuration was going to change that. Within eighteen months, the sensors were back in the barn and the grant had been spent.

The principle the industry never borrowed

Permaculture has a founding principle, stated plainly since Bill Mollison and David Holmgren developed the practice in 1978: observe and interact. You do not design until you know this place. You walk it across seasons. You watch what it does in rain and drought and frost. The design emerges from sustained attention to specific conditions — not from a template applied to all conditions.

The precision agriculture industry describes its products as nature-inspired. The language is in the brochures, the grant applications, the conference presentations. Adaptive systems. Intelligent sensors. Biomimetic design. What the industry has borrowed is the vocabulary. The founding principle — observe first, design second, never assume the average applies here — was left behind.

The reason is structural. Technology development is funded by scalability. A solution that works for eighty percent of farms is investable. A solution built from sustained observation of this specific land, and a different solution built from sustained observation of the farm next door, is not.

The observation step — the one that would reveal what this land actually needs — is designed out before a line of code is written. Scalable and specific are not the same thing.

In nature, there are no average organisms. Only specific ones, each the product of sustained adaptation to particular conditions. The precision agriculture platform was built on the logic that farms are variations on a theme. This land is not a variation. It is a place.

What twenty-five years of forecasts missed

In March 2026, a peer-reviewed study published in Agribusiness (Wiley) tracked twenty-six precision agriculture technologies across twenty-five years of adoption forecasting data. The researchers — Malone and colleagues — worked through the CropLife–Purdue Precision Agriculture Dealership Survey, the longest-running continuous assessment of precision agriculture technology in the United States. Their finding: input dealers overestimated adoption on nearly every technology, in nearly every year of the study. The overestimation gap widened in the early 2020s, as adoption on several technologies flattened or declined while dealers continued to project steady growth.

The industry’s explanation has been consistent: this is an adoption problem. Farmers are slow to change. They lack the skills, the confidence, the willingness to invest. Twenty-five years of the same explanation, in the face of twenty-five years of the same forecasting error, is not an explanation. It is a refusal to look at the other possibility.

The NFU’s 2025 connectivity survey puts a number on part of the gap. Only 22% of UK farmers have reliable mobile signal across their whole farm. Twenty-one percent of UK farms have broadband speeds below 10 Mbps — compared with a national average of less than one percent. The technology assumes infrastructure that the majority of small mixed farms in rural Britain do not have. These lower fields are not an outlier.

They are the norm.

The grant funding attached to the precision agriculture package does not ask whether the farm has 4G. It asks whether the farmer can demonstrate a commitment to sustainable intensification. They could. The application process had no field for “my lower fields have no signal,” any more than the monitoring app had a field for “food forest polyculture.” The system that allocated the grant and the system that failed to work were built on the same assumption: that the farm in front of them was a version of the average farm, and that any deviation from the average was the farmer’s problem to solve.

The kingfisher and the train

In the early 1990s, a Japanese engineer named Eiji Nakatsu was tasked with eliminating the sonic boom produced by what would become the Series 500 Shinkansen when it exited tunnels. The existing nose design was aerodynamically efficient in open air and catastrophically loud underground, where the pressure wave had nowhere to go. The engineer was a birdwatcher before they were a lead engineer, and the problem reminded them of something they had spent years watching: kingfishers moving from air into water without producing a pressure wave. The kingfisher’s beak is a precise solution to that exact problem — the transition between two media of different densities, at speed, without disturbing the surrounding medium.

The engineer did not spend a two-week research sprint watching kingfishers. They had been watching them for years, as a matter of personal attention rather than institutional programme. The insight was available because observation had been treated as real work, sustained over time, without a deliverable attached to it. The redesigned nose reduced the sonic boom, made the train 10% faster, and reduced electricity consumption by 15%.

The technology that arrived at this farm had borrowed nature’s language. It had not borrowed the engineer’s method. It had not spent seasons watching what these lower fields do after overnight rain. It had not asked what twelve years of permaculture observation had produced.

That accumulated knowledge — the only knowledge that could optimise this specific land — was invisible to the system’s design. It was not data. It was noise.

What observation asks of any system

The common thread between permaculture’s founding principle and the engineer’s decades of watching is not the sector or the technology. It is the decision to treat observation as real work, completed before design begins. The questions that follow are worth bringing to any system you deploy, inherit, or commission — because the answers will tell you whether the observation step was there, or whether it was designed out.

1. When a technology you use, or your organisation deploys, was designed — how much time was spent observing the specific conditions it would operate in, rather than the average conditions it was modelled on? Would you know if the answer was none?

2. What knowledge do you carry — about your work, your land, your patients, your tenants, your community — that exists only in your head and has no field in the system you’ve been given? What happens to a decision when that knowledge is absent?

3. If the technology you’re currently using was built for someone else — a different scale, different context, different connectivity — would the evidence be visible in how it performs, or only in what it cannot do?

4. Permaculture’s first principle has been available since 1978. What would technology design look like if it had the same founding requirement: that you do not design until you have observed?

Sam still walks the lower fields in winter. The soil sensor is in the barn. The land is doing what it has always done: varying, specific, unaverageable. The technology that was supposed to augment that knowledge had no knowledge of this place. It never did. It was built for somewhere else.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Alex is in the middle of a routine task when it happens. Not an epiphany — a small, specific thing. The step that used to require four form fields now requires one. Alex completes the task and then pauses, trying to remember when the change appeared, because it was not announced and it was not the subject of a release note and Alex only notices it now because the friction that used to be there is gone.

Somewhere in the background of that pause is a support call from eighteen months ago. Alex had reported exactly this — the unnecessary duplication in the form, the way the same information was asked for in three different places. It had taken fifteen minutes longer than it should have. Alex had said so. The call had ended with an apology and a reference number.

And now the form was different.

What usually happens to feedback

The standard experience of providing feedback to a technology product is an exercise in patience with diminishing returns. The survey that arrives after every interaction, rated one to five, with a final optional text box that almost no one fills in because nothing has ever clearly happened as a result of it. The support ticket that generates an automated acknowledgement, a reference number, and silence. The complaint that is logged, managed, and resolved — which means it is closed, not that the thing that caused it has been changed.

None of these are careless. They are systems doing what they were designed to do. The survey tracks satisfaction, the ticket resolves the case, the complaints process manages liability. None of them were designed to change the product. That distinction matters more than it sounds, because a system designed for one purpose will not accidentally perform another, however good its data.

The difference in Alex’s case was not that the organisation cared more. It was that the system had been designed for a different purpose: to move information from the person experiencing the product to the people building it, and to do something with what arrived.

The three conditions that make a loop

A genuine feedback loop breaks the moment any one of three conditions is missing.

• Feedback that works has to be easy to give — not a survey sent after the fact, not a formal process initiated at a separate URL, but a mechanism present at the moment the friction is felt. Alex did not fill in a survey. Alex called to report something broken, while it was broken, with the specific detail of what had gone wrong. The signal was clean because it was immediate and unsolicited — the kind of feedback that surfaces the actual experience rather than a retrospective reconstruction of it.

• Getting that signal to the people who can act on it is harder than it sounds. In most organisations, the path is long and obstructed: feedback arrives at customer service, which resolves individual cases and closes tickets, while patterns that would be visible to the product team — if they received the same data — never reach them. Alex’s organisation had a short path: a shared channel between support and engineering, a standing item in the sprint review for user-reported friction, a team that treated incoming complaints as product development input rather than cases to be managed. The information arrived with its context intact and at the people who could do something about it.

• Visibility is what closes the loop. Not announced, not marketed — just present. The next time Alex used the service, the thing that had been difficult was different. Without that, the person who provided the feedback has no reason to believe the loop exists and no reason to provide the next signal honestly. A feedback system that receives, processes, and acts — but never demonstrates that it has done so — produces exactly the same user behaviour over time as one that does none of those things. Trust requires evidence, not intention.

All three conditions were present in Alex’s case. Remove any one and the loop would have broken.

What two years of listening produces

Over two years, the service Alex uses has changed in small, specific ways that trace directly back to what people who used it said.

The file upload failure message used to say “Error: upload rejected.” Alex had reported this after twenty minutes trying to diagnose which of five files was causing the problem. The message now says which file failed, why it failed, and what the accepted formats and size limits are. The change took a morning to implement. The information needed to make it had been sitting in the support queue for eight months.

The address entry flow used to require four separate fields in an order that did not match the way most people read their own address. The most common support call in one quarter was people unsure which field their flat number belonged in. The form now uses postcode lookup. Completion time for that step fell by two-thirds.

A third change is smaller and easy to miss. The notification sent when a request was submitted used to say “Your request has been received.” Alex had written, in a feedback form, asking whether anyone actually looked at these requests — because the notification contained no information about what would happen next, or when, or who to contact in the meantime. The notification now includes an expected response time and a named contact for queries. It took one line of copy and the answer to a question that no one on the product team had thought to ask, because no one on the product team had ever waited on the other side of the process they had built.

The cumulative effect is a service that feels attended to. Not perfect — attended to. That quality is not cosmetic. It is the specific evidence that someone received what was said and acted on it, and it is the foundation of whatever trust Alex now places in the service.

What the design is actually for

1. Think of a digital service you use regularly. Can you point to a specific change that service has made in the past year that you believe was a response to real user experience? If you cannot — what does that tell you about whether the feedback loop exists?

2. If you build or manage a technology product, how does feedback from real users reach the people who can act on it? Is the path direct and short, or does it pass through layers that filter, aggregate, and delay?

3. If you have ever provided feedback to a technology product — a support call, a complaint, a survey response — were you ever aware of a change that resulted from it? How did that experience affect your relationship with the product and the organisation?

4. Continuous improvement is standard practice in manufacturing and in software development. Why is continuous listening — treating user feedback as an ongoing product development input rather than a post-launch compliance exercise — still the exception in most technology organisations?

Most feedback systems are designed to manage the organisation’s relationship with feedback rather than to let feedback drive the product. The organisations that reverse that design — that make the path from user experience to product change as short and unobstructed as possible — build technology that improves in the direction that matters. Alex noticed. That noticing is rarer than it should be.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Alex puts the phone face-down on the table while eating dinner. Not because there’s an argument brewing, not because anyone has asked. Just a small, deliberate act. The meal lasts forty minutes. At some point — it turned out to be fourteen minutes in — the phone gets picked up. There was no notification. Nothing rang. The impulse arrived with no identifiable origin, like a hand that reached before the mind could object.

It doesn’t feel like a problem. Just a habit. Something to work on. A matter of willpower.

That framing is exactly what the people who designed that impulse were counting on.

The story we were given

We have been taught to understand compulsive phone use as a failure of self-control. The language we reach for is personal: distracted, addicted, weak, unable to focus. Self-help books tell us to build better habits. App stores sell us tools to limit our screen time. The diagnosis is individual, and so the cure is individual.

This framing is not neutral. It is convenient — and it is convenient for a specific set of people.

The companies whose platforms you cannot stop checking did not produce compelling products by accident. They built something more precise: a compulsion. And they built it using the same mechanisms that keep a gambler at a slot machine at three in the morning, long past the point of enjoyment.

What the engineers understood

Variable reward schedules are the most reliably powerful mechanism in behavioural psychology. The principle is simple: when the timing and size of a reward is unpredictable, the brain cannot habituate to it. It cannot stop paying attention. It keeps seeking.

A vending machine that always delivers the same snack is satisfying but not compulsive. A slot machine that sometimes pays out, in unpredictable amounts, at unpredictable intervals, is something else entirely. The unpredictability is not a design flaw. It is the entire point.

Social platforms were redesigned around this principle deliberately. The pull-to-refresh gesture works like a slot machine lever — and what it delivers operates on the same principle: sometimes a funny post, sometimes a message from someone you love, sometimes nothing worth reading. The brain cannot tell in advance. So it keeps pulling.

The Hook Model — one of the most cited models in social platform design — describes the loop explicitly: trigger, action, variable reward, investment. It was not developed to help you get things done. It was developed to make sure you come back.

What the misdiagnosis costs

When we believe the problem is personal, we look for personal solutions. We download another app. We set a screen time limit. We move our phone to another room at night. Some of these things help, at the margins. None of them changes the underlying architecture.

The system benefits from our misdiagnosis. They want the conversation to remain about you.

The human cost is not abstract. Attention fragmented over years becomes harder to sustain. The capacity for deep work, for genuine presence in a conversation — these are not personality traits immune to outside pressure. They are capacities that can be eroded. And they are being eroded, deliberately, at scale.

Children who grow up inside these systems will have spent their formative years in an environment specifically engineered to prevent them from consolidating attention. The long-term consequences of that are only beginning to be understood.

Against the user

A system that manufactures compulsion rather than enabling choice is not assisting the person using it. The platform’s interest and the user’s interest diverged the moment engagement became the goal rather than the outcome of delivering something worth using.

These platforms amplify a biological vulnerability — the brain’s inability to habituate to variable rewards — and direct it toward commercial ends. That is not augmentation in any useful sense. It is a diminishment dressed as a feature.

The algorithm adapts, yes — but it adapts to exploit you more efficiently. It learns what keeps you engaged, not what serves you. These are different objectives. In most people’s lives, they are opposing ones.

A system that does not assist, does not genuinely add something, and does not adapt to serve is not a neutral tool that some people misuse. It is a system designed, tested, and refined to work against its users’ interests while appearing to serve them. That is where the responsibility lies.

The questions underneath the habit

• When you reach for your phone without knowing why, what exactly are you looking for?

• Has there been a time when you tried to reduce your phone use and found it harder than you expected? What story did you tell yourself about why?

• If you discovered that a product you used every day had been deliberately engineered to make it difficult for you to stop using it — what would you want to be different, and who would be responsible for making that change?

• Is the design of the technology in your life something you have ever actively chosen, or is it something that happened to you?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Jordan noticed it on a Tuesday morning, about four months in. They had opened the app the way they opened other apps — without thinking about it, on the way to something else. The result was there when they looked: a number, with a short explanation of what it meant in relation to where it had been last week. Jordan read it in the time it takes to make coffee. There was nothing to decode.

That moment — a test result arriving with context, at a time they could actually do something with it — was not dramatic. It did not change the condition. Jordan has a condition that requires regular monitoring, and it will require monitoring indefinitely. But something had shifted in the daily experience of managing it. The technology had been built to serve that experience, not to process it. For the first time in years of navigating health systems, the difference was legible.

The system Jordan already knew

Before the app, the management of Jordan’s condition was a patchwork held together by their own effort. Test results arrived on paper, sometimes days after the event, with reference ranges printed beside the numbers but no interpretation of what those numbers meant in context. They went into a folder maintained on their own — disconnected from the records at the GP surgery, disconnected from the records at the specialist clinic. The folder existed because Jordan had created it. No one had suggested it; no system supported it.

Appointments were bookable, but the booking required persistence — the kind that is straightforward when you are well and difficult when you are managing a condition that affects concentration and energy. Jordan had learned which times to call, and which numbers to try when the first was engaged. These were not skills anyone had expected to need.

The condition itself was manageable. What was not manageable, not sustainably over years, was the administrative overhead the health system had attached to it. The paper trail that was entirely the individuals responsibility. The records that told a different story depending on which part of the system you were looking at. The appointments that were partly administrative events — catching up on information that should have been shared already, establishing a baseline that should have been continuous.

The ordinary cost of managing a long-term condition in a system not designed for the person managing it is a cost that most people carrying those conditions pay, quietly, without counting it as a cost. The honest context is that health technology has not, on the whole, served people in this situation well. Two thirds of NHS patients and carers reported at least one administrative problem in the past year, according to the King’s Fund in 2025 — disproportionately those managing long-term conditions. That history is part of why the difference, when it came, was so immediately recognisable.

The question they asked first

The team that built the app began with a question rather than a specification. Not: what should this app do? But: what does it feel like to manage this condition, and where does the friction concentrate?

Those two questions produce different software. The first produces a list of features. The second produces an understanding of what is actually hard about the problem — which is rarely what the first question reveals, and rarely what someone without the condition imagines from the outside.

The answer, when they gathered it, was specific. The friction was not in the absence of information but in its format and its timing. Information arrived too late, without context, through channels not connected to anything else. The friction was in the administration of care, not in the care itself. The app was designed to reduce that specific friction, in that specific form, for people in that situation.

What came next followed from that question. The design was tested throughout development with people who found digital tools genuinely difficult to use — not only with people who were already comfortable with health apps and quick to forgive poor design. The team was accountable, internally, for whether the people using it could actually manage their condition better — not only for whether the app was delivered within budget and adopted within target timescales. And the design was iterated after deployment, based on how people used the app in practice rather than how they had been expected to use it.

None of that requires a large budget or an unusually skilled team. It requires a set of priorities — a decision, made at the start and maintained throughout, that the person using the technology matters more than the efficiency of the organisation providing it. That decision is not always made. When it is not, the technology reflects the absence.

What actually changed

Jordan’s condition is no better or worse than it would otherwise have been. What is better is the experience of managing it, day to day, in the space between clinical appointments.

Test results that used to require interpretation now arrive pre-interpreted — with enough context to inform a decision, to notice a pattern, to decide whether something warrants a call before the next scheduled appointment. No more calls to the surgery to ask what a number means. The conversation with the clinical team is more useful because the patient arrives with information, and the team arrives without gaps to fill. What used to be a partly administrative appointment is now a clinical one.

The relationship with the clinical team has changed because the technology removed the noise that used to surround it. The paper trail is gone. The catch-up is gone. The time spent establishing what the records should already show — gone. That is not a small improvement in a clinical relationship. It is a different kind of care.

There is also more time. Time that used to go to navigating systems that were not working — chasing results, maintaining the folder, ringing back — is available for other things. The calculation is not large on any single occasion. Over years, it compounds.

None of this is dramatic. The condition is still there, still requires attention, still shapes daily life in ways that are not going to change. But the daily weight of managing it is lighter than it was, and that lightness is the direct result of a design team that started with actual circumstances rather than a generic idea of what a patient requires.

What the design actually did

The app does one thing well: manage this condition, in these circumstances, with these constraints. Not a generic patient managing a generic condition — a specific person, with the particular shape of their daily life. The help it provides is specific to where the friction actually is, rather than matched to what the average health app is assumed to need. That specificity is the direct consequence of the question they asked first.

The app adds something that was not there before. The person using it can now participate in their own care in a way that was previously impractical — reviewing their own data with context, preparing for clinical conversations with real information, communicating between appointments without the friction that made that communication more trouble than it was worth. The participation remains theirs. The app did not take over the management of the condition. It provided better tools to manage it.

That distinction, between technology that substitutes for a person’s agency and technology that extends it, was a design decision, not an accident. It was made at the start.

The app also responds to the individual. Over time it reflects what this person uses, finds useful, needs at different moments. The personalisation is not surface-level — it changes how the app functions rather than how it looks, making it more useful over time rather than more impressive in a product demo. That responsiveness requires a feedback loop between user and team — one that exists only because someone was accountable for outcomes. Which is, again, the decision made at the start.

Four questions for the next project

1. Think of a piece of technology — an app, a service, a system — that has genuinely helped you, in a way that felt specific to you rather than generic to all users. What made it different? Can you identify the design decision that produced that experience?

2. If you build or commission technology, how does your design process ensure that the person who will use it is the starting point rather than the endpoint? What specific mechanisms — genuine user research, inclusive testing, accountability for the human outcome — are in place to hold that priority throughout development, not only at the requirements stage?

3. What would it take, in your organisation or sector, to make conditions like these the standard rather than the exception? Not in five years — in the next project. What is the single first change that would make the most difference?

4. Is the gap between technology that burdens and technology that helps primarily a technical gap, a resource gap, or a priorities gap? And if it is a priorities gap, what does that tell us about who has the power to close it?

The conditions that produced an app like this are not secrets. They are known, replicable, and not especially expensive. The gap between what exists and what is possible is not technical. The reason it persists is a question for the people who make the priorities, not the people who write the code.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Alex submitted the third query through the portal on a Thursday afternoon, copying in the reference numbers from the first two. The housing benefit calculation still didn’t match what Alex had been told over the phone three months earlier. The difference was not large — roughly forty pounds a month — but over four months it had accumulated into an overpayment notice that Alex was now expected to repay.

Each of the three queries had been acknowledged within twenty-four hours. Each had been assigned a case reference. Each had received a response. The most recent response explained, in clear language, how housing benefit calculations worked. It did not explain why Alex’s calculation differed from the figure given on the phone. The case had been marked as resolved.

The week after Alex received the overpayment notice, the council’s customer services director presented the quarterly performance report. Digital channel adoption was up. Average response time was down to 1.4 working days. Cost per interaction had fallen 28%. Case resolution rates were running at 94%. The dashboard was green across every metric. Nobody in that meeting had Alex’s name in front of them.

The metrics the council was reporting were accurate. An average response time of 1.4 days is faster — measurably, demonstrably faster — than what came before. A 28% reduction in cost per interaction is a real achievement. A 94% case resolution rate sounds like a service that is working well.

What the dashboard doesn’t count

What none of those numbers captured: whether the question had been answered. The 94% resolution rate counted cases that had received a response and been closed. It did not count whether the response had resolved the issue. A case could be opened, acknowledged, responded to with a standard letter, and closed as resolved — three times — without the person ever getting what they needed. That is not a failure in the data. It is a failure in what the data was designed to count.

The calls that ended without resolution. The cases where the automated response was technically accurate and substantively irrelevant to the specific question asked. The people who received four replies and gave up on the fifth attempt. None of that appears in the performance report. Not because it was hidden, but because the system was never designed to see it.

The people running the service have no signal that anything is wrong. The people funding it have no reason to ask. The people using it — the ones for whom it is not working — have no channel to make the failure visible, except to keep submitting queries that will each receive a response and remain unresolved.

The politics of choosing what to count

This is where Goodhart’s Law applies — the principle that the economist Marilyn Strathern summarised as: when a measure becomes a target, it ceases to be a good measure. Response time was a reasonable proxy for service quality until it became a target. Once it was a target, the system optimised for it. Cases were responded to quickly. Whether the responses were useful was a different question — and not one the target required anyone to ask.

KPI structures for technology services are not purely technical documents. They are political ones. The metrics chosen reflect what the organisation is prepared to be accountable for, and organisations, rationally, choose metrics they can improve. Response time can be reduced through better routing and automation. Case closure rates can be improved by defining “resolution” in ways that are achievable. Cost per interaction falls as volume grows. All of these things can be improved without making the service more useful to the person at the end of it.

The human outcome — did the person get what they needed? — is harder to measure because it requires asking them. That takes time, raises uncomfortable questions, and produces data that is more difficult to present as success. So it is left out. Not maliciously. Structurally. The governance body that approves the metrics does not require it. The reporting cycle that reviews them does not miss it. And the dashboard that summarises them has no row for it.

There is also a governance dimension to this. The people who approve the KPI structure are presented with a set of metrics that look complete because they cover all the dimensions the previous governance model required: speed, volume, cost, compliance. Nobody in that room has ever been asked to approve a metric for whether the service actually resolved the person’s problem. Nobody has been told that such a metric is missing. The governance process validates what it is shown. What it is not shown does not fail — it simply does not exist in the conversation.

This is how a service comes to be celebrated and broken at the same time. The celebration is for what is measured. The brokenness is in what is not. The two things co-exist perfectly until someone submits a third query and receives a fourth standard letter.

The cost that doesn’t appear in the report

The pattern repeats across services. A case that the digital channel cannot resolve eventually gets escalated to a phone call — the kind of contact the digital system was built to reduce. The query is answered. The discrepancy is corrected. The outcome the portal was supposed to deliver arrives via the channel the portal was supposed to replace, weeks later than it should have.

The quarterly performance report will show digital channel adoption at its highest level. What it will not show: the volume of digital cases subsequently resolved by phone; the number of people who returned to a human channel because the digital one had not worked; or the real cost of a failed digital interaction, which includes not just the cost of the digital transaction but the cost of the phone call that followed it.

The service looks more efficient than it is. The measurement system has been designed in a way that makes that possible — by measuring the transactions that occurred rather than the outcomes they were supposed to produce. When a governance body reviews a performance report and sees green across every metric, the natural response is to keep doing what is working. The investment flows toward the metrics that are already performing well. The things that are failing silently receive nothing, because they are not visible as failures.

That is not a data quality problem. It is a design choice. The absence of an outcome metric is itself a decision, with consequences that accumulate quietly in every case the service was not built to see. And because the consequences are not measured, they do not generate pressure to change the measurement. The organisation that cannot see its own failures cannot feel the cost of them, and cannot direct resources toward addressing them. The green dashboard is not just a reporting artefact. It is a protection against accountability — not deliberate, but effective.

My Opinion

I have watched organisations celebrate dashboards that tell them almost nothing about whether their services are working. The instinct to measure is right. The flaw is in what gets chosen. SLAs, KPIs, OKRs all share the same structural problem: they measure what the organisation can control. An automated acknowledgement meets the response SLA. The service still failed. I’ve witnessed a large outsourcing organisation hit their KPIs and SLAs every single month for response times…by putting an automated email response thanking the user for their query.

The organisations closing this gap are measuring outcomes, not transactions — asking users directly whether their problem was resolved. Some are going further, tracking social impact metrics: what actually changed for the person, beyond whether the case was closed. A number is a starting point, not a conclusion. Before accepting one, ask what human reality it represents. If you cannot answer that, the number is not yet telling you what you think it is.

The questions the system was never required to answer

The question a measurement structure should start with is whether the technology is actually helping the specific person using it — not whether it is processing their request efficiently. Those are different things. A system that processes twelve thousand requests per quarter and resolves sixty percent of them is not necessarily more helpful than one that processes eight thousand and resolves ninety. But a performance report that counts volume and response time will describe the first system as the better one.

The second question is whether the technology adds something real to the person’s capability, or only reduces costs for the organisation. A reduction in cost per interaction is not the same as an improvement in the quality of the interaction. Faster and cheaper are not synonyms for better. A measurement system that tracks only cost and speed cannot answer whether the technology is doing what it was built to do. It can only confirm that the technology is doing what it was built to do cheaply and quickly — which is a different question entirely, and a less important one.

The third question — and the one most completely absent from most KPI structures — is whether the system responds to the specific circumstances of the person in front of it. The 94% resolution rate is a population-level figure. It tells you nothing about the person in front of it. It tells you nothing about the six percent for whom the service did not work, or what those cases had in common, or whether they were disproportionately the people who needed the service most. A system that performs well on average while failing consistently at the edges is not performing well. It is performing adequately for people whose situations fit the expected pattern, and invisibly for everyone else. Those are not the same thing, and no dashboard that measures only the average will ever surface the difference.

The question behind the numbers

1. Think of a technology or service you use regularly — at work, or as a citizen or customer. What do you think is being measured about your experience of it? Does that match what you would want to be measured?

2. If you are responsible for reporting on technology performance in your organisation, when did you last ask whether the metrics you report capture the experience of the person at the end of the system — not just the performance of the system itself?

3. If you could add one metric to your organisation’s technology reporting — something currently unmeasured that would tell a more complete story — what would it be, and why is it not already there?

4. A system that hits every KPI while failing the people it serves: is that a success or a failure? And if the answer is complicated, what does that tell us about how we have defined success?

A service can hit every target on its dashboard and still fail the person who needed it. The performance report will describe it as operating at record efficiency. Both things are true. They are not the same truth, and the distance between them is where the accountability isn’t.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The agenda ran to eleven items. Security was seventh. By the time the board reached it, forty minutes had been spent on a supplier contract and a facilities question about the car park. The CISO had twenty minutes. They used eighteen of them.

The security review recommended twelve measures. The board approved three, deferred four, and excluded five as beyond the year’s plan. The meeting ended. Someone went to get coffee. Nine months later, one of the deferred items was the entry point for a ransomware attack that shut the business down for eleven days.

Nobody in that room had been negligent. Nobody had ignored the risk. The decision was made by competent people using the information they had, inside a process designed to balance competing priorities.

The CISO’s brief was forty-two slides. The executive summary was four pages. The risk register mapped each of the twelve recommendations to a likelihood score and a financial impact range. It was thorough. It was also built to satisfy a compliance requirement, not to help a non-technical board member understand which of those twelve things actually mattered most, and why, and what the difference would look like if they got it wrong.

Item seven on the deferred list was a patch management protocol for legacy systems. The brief described it as “medium risk, estimated remediation cost £18,000, recommended timeline Q2.” It sat between a network segmentation proposal and an employee phishing simulation programme. The board chair asked whether it could wait until the budget review. The CISO said it could be managed. The item was deferred.

That sentence — “it can be managed” — actually means: I believe the risk is tolerable in the short term, given everything else on the table. It does not mean: this is safe. The distinction between those two things disappears in the room. It stays disappeared until something goes wrong.

Organisations have been told for twenty years that security is a board-level responsibility. The intention was good. Bring it out of the IT department. Get it in front of decision-makers. Give the CISO a seat at the table. According to DSIT surveys, only 27 to 31 per cent of UK businesses have a board member with specific responsibility for cybersecurity — and this after a decade of sustained regulatory pressure to change it.

What happened instead is that security became a governance exercise. A risk register. A compliance checklist. A set of items to be approved, deferred, or removed, using the same mental framework that the board uses to evaluate a marketing spend or a property lease. The question is always: what does this cost, what does it protect, and can it wait?

The catastrophic failure of that framework is that cybersecurity risk does not behave like the risks that framework was built for. A lease that is deferred costs rent. A security measure that is deferred creates a window. The attacker does not wait for the budget cycle.

The deferred patch was not a theoretical vulnerability. It was a known gap, documented, scored, and filed. It sat in a folder. The attacker found a different route to it than the folder.

The decision that enabled the attack was made in a Monday morning meeting that ended with someone going to get coffee.

Eleven days offline. Not a dramatic number until you translate it. Supply chain halted. Customer orders unmet. Staff unable to work. The IR firm billing £2,400 a day. The ransom demand. The legal fees. The forensic audit. The regulatory notification. The reputational cost that does not appear in any spreadsheet.

The CISO did not lose their job. The board did not face personal liability. The organisation recovered, at significant cost, and updated the risk register. The patch management protocol was implemented in full. It had cost £18,000 on the original recommendation. The recovery cost many times that.

The people in the meeting were not careless. They were doing exactly what the system asked them to do. The system asked them to balance risks against budgets in a room built for a different kind of risk. That is not a character failure. That is a design failure.

What the brief was actually built for

A forty-two slide deck translated into a four-page summary with a risk register does not help a non-technical director make a well-informed decision about which deferred item will cost them thirty-eight times the remediation price. It helps the CISO demonstrate that they have fulfilled their duty of care. Those are not the same thing. The board member who approved the deferral did not understand the difference between “medium risk” and “a known gap in a legacy system with active exploit code circulating in the wild.” The brief did not help them understand it.

The governance structure was built to add something to security oversight — documentation, audit trail, the appearance of rigour. What it does not reliably add is security, because security requires that decisions reflect the actual threat environment rather than the budgetary one. A compliance framework and a threat actor are operating on entirely different timelines. The register records what was known. The attacker acts on what was left open.

The risk assessment treated every organisation of similar size and sector as roughly equivalent. It did not adapt to this company’s specific exposure: which of their legacy systems was most likely to be targeted, what the attacker’s likely entry path would be, what the actual blast radius of a successful attack on that specific infrastructure would look like. A one-size-fits-all risk score applied to a specific organisation on a specific day in a specific sector is not risk management. It is risk documentation.

The problem is not that boards are incompetent or that CISOs are failing. The problem is that the governance architecture built to manage cybersecurity risk was designed for a world where risk is relatively static and consequences are proportionate. Cybersecurity risk is neither. The architecture needs to change, not the people trying to work inside it.

My opinion

The more we rely on digital services, the more we place ourselves in the hands of organisations whose governance structures were not built to protect us. CISOs are operating in conditions no single role should carry. My position is that security risk belongs at the top of every board agenda, treated as a living state rather than an annual exercise — because the threat changes every day. Every new service deployed without security built in from day one creates exposure that did not exist the day before. Secure by design is not a technical principle — it is a condition of doing business.

The questions the risk register doesn’t raise

• Have you ever been in a meeting where a security recommendation was deferred for cost or time reasons? Not because anyone thought it was unimportant — but because the table was full and the budget was finite and it seemed manageable for now?

• If you have sat in a room like that one: what would it have taken for the conversation to go differently? What would the CISO have needed to say, or show, or do, to change the outcome?

• Should the people who make security deferral decisions face personal accountability when those decisions lead to a breach? And if the answer is yes — what would that accountability look like in practice, and who would apply it?

• The gap between the risk register and the real threat environment is not a secret. The people who write the registers know it. The question is whether the governance structures we have built are capable of closing it — or whether they were designed for something else entirely.

The dynamics in that boardroom — the compressed agenda, the deferred risk, the gap between documented exposure and lived consequence — run through The Shadow System. The book takes the argument further than a single meeting.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Casey stood in the kitchen on a Sunday morning wanting to do something small and

ordinary. The lights were already on — the right level, the right temperature, the system having made the calculation sometime before waking. The heating was already at the temperature the house knew a Sunday morning required. The coffee machine had already started its cycle.

Not one of these things had been consciously chosen that morning.

The intent had been to make a deliberate choice about the day — to begin it with a moment of decision rather than discovery. To turn the lights up brighter than usual, perhaps, because it was overcast outside and the mood called for it. To make the coffee later than normal because there was no urgency. Small acts of self-determination in an ordinary morning.

The system had already decided. It had observed enough previous Sunday mornings to know what this one required. The preferences were not wrong — they were, in fact, accurate. They reflected genuine past behaviour, real choices made on real mornings over months. The system had learned well.

That was the problem.

The disappearance of friction

There is a version of this story that reads as convenience. The house anticipates. The morning runs smoothly. That is what the technology was designed to do, and it is doing it.

What that version of the story does not account for is what friction was for.

The small acts of choosing — the light level, the temperature, the pace of a morning — are not merely mechanical. They are the continuous practice of preference: the low-stakes exercise of deciding who you are and what you want on any given day. They are how a person remains in contact with their own inclinations. When those acts are automated away, the automation does not just remove effort. It removes the opportunity to notice whether the choice would have been the same.

Casey’s house had learned a version of Casey built from past behaviour. Not from yesterday’s behaviour, from the cumulative pattern of months. A version that was accurate in aggregate and potentially wrong in any specific instance. The system had no mechanism to register that people change, that moods shift, that a Sunday in January and a Sunday in March might call for different things even if the previous fifty Sundays had called for the same ones. The actual person was no longer quite required.

The moment of choosing had disappeared so gradually that it was only noticeable in its absence. Not when the system got it wrong, it rarely got it wrong in any obvious way. But when the desire arose to do something differently, and the system had already done it, and the act of undoing it felt like more effort than simply accepting what was already there.

That is the trap. Not coercion, convenience so complete it becomes its own constraint.

From the kitchen to everywhere

The smart home is the most intimate version of this pattern, because the home is the space where people are most fully themselves. But it is only the beginning of the scale.

Step outward from the kitchen and the same architecture is running at every level. The streaming platform that has watched enough viewing history to know what will be selected before the selection is made. The news feed that has modelled enough reading behaviour to decide what is worth knowing today. Each of these systems is doing the same thing as the smart home, in a different register: replacing the act of choosing with the delivery of a prediction so accurate that choosing no longer feels necessary.

Step further out. The e-commerce platform that has learned spending patterns well enough to surface only what will be bought — not what might be wanted, not what might be surprising, but what the model says is coming next. The navigation app that routes not by the driver’s preference but by the algorithm’s assessment of the optimal path, such that after a year of use a driver may no longer know the roads they travel every day. The workplace tool that pre-populates responses, pre-schedules meetings, pre-drafts the email — reducing the human contribution to the act of approval.

At each step the friction removed is real. The convenience is not manufactured, it is delivered. And at each step something else is happening simultaneously: the system’s model of the person is deepening, the person’s practice of self-determination is narrowing, and the gap between who the system thinks they are and who they actually are is quietly widening.

The model is built from what was. The person is living in what is. Those two things are not the same, and the longer the system runs without the person having reason to notice the divergence, the larger the gap becomes.

Step to the outermost scale. The political information environment, the social media feed, the search result page, all of them running the same logic at civilisational scope. Not just anticipating individual preferences but shaping them, because a preference exercised without friction, without encounter with the unexpected, without the resistance of a world that does not already agree, is a preference that stops developing. The system does not just reflect who you are. Over time, it shapes who you remain.

The distinction nobody offered

The question of how much anticipation is enough is not rhetorical. It is a real dilemma.

Friction is not inherently good. The effort of choosing is not valuable in itself, it depends entirely on what is being chosen and why. The friction of adjusting the thermostat is not the same as the friction of deciding what to read or what to believe or what kind of person to become. Not all convenience is erosion. The task is to distinguish between the friction that was serving a purpose and the friction that was simply in the way, and to recognise that those two things look identical until the moment of removal.

Most people have not made that distinction deliberately, because it was never offered to them as a choice. The anticipatory systems arrived and removed friction and the question of which friction was worth keeping was never asked. The default was anticipation. The person who wanted to preserve the act of choosing had to actively resist the convenience of not choosing, and resistance is itself a form of friction that most people do not have time or inclination to apply.

The deeper consequence is not about convenience it is about the model. Every anticipatory system builds a representation of the person it serves. That representation is the basis for every subsequent prediction, every pre-emptive action, every choice made on the person’s behalf. The richer and more accurate the model, the more completely it can substitute for the person’s actual judgment. And the more completely it substitutes, the less opportunity there is for the person’s actual judgment to develop, correct, or surprise itself.

A person who has never had to choose their own route no longer knows the roads — a 2024 study confirmed this is not a metaphor: habitual GPS use produces a measurable, causal decline in spatial memory, even in people who previously navigated well. A person whose reading is pre-selected no longer encounters what they did not know they were missing. A person whose home has already decided never fully inhabits the decision to be at home. The capacity that is not exercised does not remain dormant. It diminishes.

That is the cost that does not appear in the product specification. That is the human loss that convenience optimises away.

Who is this for

The question the anticipatory system never asks, because it was not designed to, is who benefits from the anticipation.

The person using the system receives convenience that is real. But the system is not running in their service alone. Every preference recorded, every choice predicted, every behaviour anticipated is data. Data that trains the model, refines the prediction, deepens the platform’s understanding of what the person will do next. The more accurate the anticipation, the more valuable the person becomes as a data source and the more dependent they become on a system whose primary obligation is not to them.

The smart home manufacturer knows the daily rhythms of the household. The streaming platform knows attention patterns more precisely than the viewer knows themselves. None of this knowledge is held in the person’s interest by default. It is held in the platform’s interest, governed by the platform’s terms, subject to the platform’s commercial decisions about how that knowledge is used, shared, or sold.

The anticipatory trap is not just about agency. It is about the terms on which that agency has been surrendered and to whom. Casey’s house is learning Casey — but who owns what has been learned, and what happens to it when the subscription ends, the company is acquired, the terms are changed? The system was presented as a tool serving the household. The data flowing out of it tells a different story about who is serving whom.

The system that anticipates your needs perfectly has built a more accurate model of you than you have of yourself. That is impressive. It is also alarming, not because the anticipation is wrong, but because the entity holding that model is not you, does not answer to you, and did not ask your permission to build it in quite the way it has.

The friction that was removed was not just effort. Some of it was the resistance that kept you in contact with your own preferences. Some of it was the randomness that kept you encountering the unexpected. And some of it was the inconvenience that made you notice, every now and then, that a choice was being made at all.

What it was replaced with is not neutral. It serves someone. The question is whether you know who.

What you might find you’ve forgotten

Think about the systems in your daily life that anticipate your preferences — the home, the phone, the feed, the platform. When did you last make a deliberate choice in a domain where anticipation now operates? Not overriding a suggestion — making an original decision, from scratch, without a pre-populated option in front of you?

The model these systems hold of you was built from your past behaviour. How accurate is it to who you are now? Have you changed in ways the system has not registered? Have your preferences shifted, your interests expanded, your circumstances moved, and has the anticipation moved with you, or is it still running on data from a version of you that no longer quite exists?

Who holds the model? Not in terms of which company’s name is on the product, in terms of what happens to what has been learned about you. Is it portable? Is it deletable? Is it yours, in any meaningful sense? Or has the convenience of the service been partly funded by the permanent transfer of something you did not consciously offer?

And the question underneath all of these: if the anticipation were removed tomorrow — if the systems stopped predicting and simply waited for instruction — what would you find you had forgotten how to choose?

My Opinion

The technology itself is not the problem. What makes ambient systems genuinely difficult is their invisibility. A tool you can see, you can use deliberately. A system that operates below the level of conscious attention has already made the choice about when and how it operates. The line between the two is not technical — it is a design decision, made by someone other than you, before you arrived.

Most ambient technology was not designed to be controlling. It was designed to be easy. But easy and controlling share a direction of travel. The easier a system makes a decision, the less often you will make that decision yourself — and at some point along that gradient, the decision stops being yours. It stopped leaving room for it.

Not everyone needs this. That sounds obvious but it is rarely said plainly. There are households where ambient technology solves a real problem — accessibility, health monitoring, the household stretched across too many responsibilities to manage routine decisions without help. In those circumstances it is doing what it was built for. As a default for everyone, replacing the ordinary friction of a morning with a prediction of what the morning should be, it is a different proposition — one that takes something from the quality of ordinary life that is hard to name until it is gone.

The question is not whether to use these systems. It is whether you know what you are giving in exchange. Convenience offered at no visible cost is rarely free. What is given in return is usually the data that makes the system more accurate, and the habit of not deciding that makes it more necessary. Both transfers happen gradually, without announcement, and by the time they are noticeable, reversing them requires more effort than most people are willing to apply on a Tuesday morning.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan opens the app on a whim. It is one of those data-transparency tools that have become briefly fashionable — the kind that pulls together what advertisers, brokers, and platforms believe about you and shows you a summary. Morgan is curious, not anxious. It takes about thirty seconds to load.

What comes back is strange. The profile is accurate in the way a caricature is accurate — it captures something recognisable while getting most of the important things wrong. Morgan is categorised as a “high-anxiety urban professional,” likely to respond to time-pressure messaging and financially cautious. There are interest tags: travel, wellness, mid-range electronics. A risk band. An inferred household income. Morgan reads it twice, unsure whether to laugh or feel uneasy. Neither feels quite right.

When behaviour becomes identity

The discomfort is worth paying attention to. Not because the data is false — most of it is close enough to be recognisable — but because it is being treated as complete. The profile was assembled from clicks and purchases and search patterns, and somewhere along the way it stopped being a description of behaviour and became a verdict about a person.

That shift happens quietly. A system designed to predict preferences begins to define them. A tool built to segment audiences starts to assign identities. Nobody decided this was the purpose. It emerged from the logic of optimisation, from the pressure to know more in order to sell more, to lend more wisely, to assess risk at scale. The profile is a side effect of a process that was never really about the person at all.

The profile is accurate about behaviour and wrong about the person. The danger is not the bad data — it is that the system believes data is enough.

A trace treated as a portrait

Here is the underlying problem: data captures what someone did, in a particular moment, under particular circumstances. It does not capture why. It cannot hold the week someone was caring for a sick parent and spending on delivery food because there was no time to cook, or the period of job uncertainty that explains a cautious financial pattern, or a single impulsive holiday purchase that rewrote the travel interest score. The data is a trace. The system reads it as a portrait.

This matters because these profiles are not just sitting in a database somewhere. They are active. They shape what someone is offered, what rate they are quoted, sometimes what opportunities appear and which do not. A mortgage application, a job screening, an insurance premium — in each of these, the statistical shadow is treated as the person. And the person has almost no practical way to correct it, because most of the time they have no idea it exists.

The portability of these profiles makes it worse. An assumption made in one context follows into another. A risk score generated for one purpose bleeds into decisions it was never designed to inform. The data travels; the context does not. What began as a description of what someone did last Tuesday is now being used to decide what they deserve next year.

What the profile decides

The human cost here is not always dramatic. Sometimes it is a loan offered at a higher rate than someone with an identical financial history but a different data footprint would receive. Sometimes it is a job application that does not progress past the screening layer. Sometimes it is simply the low-grade unease of being looked at by systems that cannot see you, being assessed by processes you cannot access, being defined by a version of yourself that you did not author and cannot revise.

For those who are already disadvantaged — people in financial precarity, people with gaps in their histories, people whose lives do not fit the patterns the models were trained on — the distortion is not low-grade. It is consequential. The system is not malicious. It is simply indifferent to the difference between a data point and a human being. That indifference is its own kind of harm.

What the design is actually doing

In the original use case — recommendations, relevant offers, faster credit decisions — there is a genuine service on offer. But it holds only while the profile is accurate, and only while the person has some say in what it contains. Where the profile is wrong, or where it follows someone into contexts where it has no business being, the service becomes a constraint.

The efficiency gains for the institutions using these systems are real. What has been added for the individual is less clear: a permanent, portable verdict about who someone probably is. Whether that addition has value depends entirely on who is doing the measuring — and for whom the system was designed.

Where it fails most visibly is in adaptation. The profile is static in the ways that matter most. It captures behaviour but not situation. It updates when you click, but it does not reset when your life changes. It does not know the difference between who you were and who you are, and it has no mechanism for registering who you are trying to become. A system that cannot hold that kind of ambiguity is not adaptive. It is just persistent.

This is one of the questions at the centre of The Cognitive Crucible — not just what the system records, but what it misses by design.

My Opinion

We are living through a period of digital identity fragmentation. Pieces of us — clicks, purchases, search patterns — are being gathered by third parties and reassembled into a composite. The problem is not just the assembly. It is the premise: that this composite represents a person.

It does not. We are not one thing. We are many, depending on context — different at work and at home, under pressure and at ease, at a particular age, in a particular relationship. A system that flattens all of that into a single profile is not describing us. It is describing a version of us that has never existed.

Taking back agency over this will be hard. It may require something counter-intuitive: choosing, deliberately, to define and manage our own digital identities — profiles that represent genuine aspects of who we are in a specific context, rather than ceding that definition to systems that were never designed with us in mind.

The questions underneath the discomfort

There is a reasonable question underneath the discomfort the opening described. Not “how do I get off this list” — that is almost certainly impossible — but something more fundamental: who is this profile actually for?

The profile was not built for the person. It was built for the systems that process them. That distinction changes the nature of the conversation. The question is not whether the data is accurate. The question is whether data should be sufficient.

1. Have you ever encountered a version of yourself through a system — a recommendation, a decision, a risk band — that felt both recognisable and completely wrong? What did you do with that feeling?

2. If you could see your full data profile across every platform and institution that holds one, what do you think you would find — and would you want to know?

3. If you could change one thing about how these systems work — how they are built, what they are allowed to infer, how they can be challenged — what would it be?

4. The system knows your clicks. Should that be enough to know you? And if not — what else should be required before a system is allowed to make decisions about your life?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The people report comes toward the end of the board pack. Somewhere after the financial performance slides and before the risk register. It contains an engagement score — this quarter seventy-one, up three points from the previous period — and a brief commentary from the people director explaining the movement. The board notes the improvement. Someone asks whether the score is above or below sector average. The answer is above. The item closes.

Six months later, a significant portion of the senior leadership team has left. The reasons, when they eventually surface, are not complicated: a sustained period of organisational change had been managed without involving the people most affected in the decisions being made, and those people had concluded, rationally, that the direction was not one they wanted to follow. None of this was invisible. It was legible, in specific terms, in the free-text responses that sat beneath the engagement score the board had accepted as reassurance.

The board had asked the right governance question — how engaged are our people — and received an accurate answer. The number was real. What it failed to convey was whether the organisation understood the conditions producing it, and whether those conditions were stable or deteriorating beneath a score that was still, technically, above average.

This is not an unusual story. It is a governance pattern.

What a people metric is designed to do

Board-level people reporting has evolved to serve a specific function: to give non-executive directors sufficient assurance that the human risk in the organisation is being managed. The engagement score performs that function efficiently. It is comparable across periods, benchmarkable against sector peers, and defensible in the event of a regulatory or reputational challenge. It answers the governance question the board is equipped to ask.

The problem is that the governance question and the strategic question are not the same question. Is engagement above or below threshold is a governance question. Do we understand what our workforce, our customers, or our communities are actually experiencing well enough to anticipate what is coming is a strategic one. The first produces a number. The second requires intelligence. And most board people reporting has been designed to produce the former while giving the impression it addresses the latter.

The distinction matters because boards make consequential decisions about people — through strategy, through resource allocation, through the acquisitions and restructures they approve — and the quality of those decisions depends on the quality of the understanding they rest on. A board that believes a score of seventy-one means everything is broadly in order is making different decisions than a board that understands why the score is seventy-one, which parts of the organisation are driving it, and what would need to change for it to move in either direction.

The score alone cannot support the second set of decisions. It was not designed to.

What boards are not seeing

The gap between what boards receive and what would constitute genuine people intelligence is wide in most organisations, and wider still in specific situations: post-acquisition integration, significant organisational change, sustained periods of uncertainty, or industries where the workforce is distributed, contracted, or otherwise at a distance from the centre.

In each of these situations, the aggregate score becomes less representative precisely when it most needs to be reliable. The integration that is going well for the acquiring team and badly for the acquired workforce produces an average. The restructure that has settled the leadership layer while creating deep anxiety in the operational layer produces an average. The number consolidates the variation into a single point and presents it to the board as a picture of the whole.

What the board is not seeing, in most cases, is the distribution beneath the average — where the score is concentrated, where it is fragile, which parts of the organisation are carrying the weight of others, and what the people in the lower quartile are specifically saying about their experience. Nor is it seeing the qualitative register: the specific concerns, the named conditions, the articulated reasons that would allow a board to distinguish between an engagement dip that is transient and one that is structural.

Boards that have tried to close this gap by asking for more granular data — scores by function, by tenure, by level — get closer to the distribution problem but do not solve the intelligence problem. Granular numbers are still numbers. They show you where the variation is. They do not show you why.

The question no one asks in the boardroom

There is a version of this problem that is specific to the board’s role, and it sits just beneath the surface of every board pack.

Boards are responsible for oversight, not management. The distinction is important. A board that receives an engagement score and accepts it as adequate people intelligence is not failing to manage — that is not the board’s job. It is failing to ask the questions that oversight requires: what does this number actually tell us, what does it not tell us, and what would we need to know to be confident we are not surprised by something we should have seen coming.

In most organisations, no one in the boardroom asks those questions routinely. The people director presents the data they have been asked to present, in the format that has been agreed, against the benchmarks that have been established. The non-executives receive it in a pack that also contains financial performance, risk, and compliance items that require more active scrutiny. People data gets the time allocated to it, which is rarely enough to interrogate it deeply.

The result is that people risk is governed at a level of abstraction that would be considered inadequate in other domains. A board that accepted a single aggregate technology risk score without understanding what it represented, where the exposure lay, and what conditions were driving it would be considered negligent. The equivalent in people governance is standard practice.

The shift required is not primarily about tools or data. It is about what boards decide to demand. Engagement intelligence that is specific, distributed, and grounded in what people actually say — rather than how they respond to a scale — is achievable. Gobby, a UK platform built on qualitative-first surveys, shows what this looks like in practice: respondents answer in their own words, then validate each other’s responses rather than having them averaged by the system. What reaches the board isn’t a number — it’s what the workforce actually said. It requires a different kind of question at board level, and a people function equipped and willing to answer it.

The score will always be easier to present than the understanding. The question is whether the board is prepared to settle for it.

My opinion

I left an organisation once because the leadership had changed five times in five years and no one seemed to know what direction they were heading in. When I left, no one asked why. The exit process was procedural. I was not a data point anyone was going to interrogate. In another business I left because of a executive level change which resulted in an organisational restructure, after the very first meeting with the new executive I knew what was coming and it wasn’t going to be pretty. The thing which I loved about that business was the one thing that was going to be destroyed - Culture. So I resigned, said my piece and everything I said came true.

The people who collect this kind of information are usually not hiding anything deliberately. They are responding to incentives. They have learned, over time, what the board finds useful and what it finds uncomfortable. They present the former. They convert the latter into a score.

That is not intelligence. It is data dressed as reassurance. And if a board cannot hear the difficult alongside the comfortable — if the culture in the boardroom makes hard truths unwelcome — then the problem is not the measurement. It is a leadership failure at the top of the organisation.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam has been reading the same paragraph for the third time. The words are there. The comprehension isn’t. Something, unclear what, has been pulling at the edge of attention for the last twenty minutes without ever resolving.

The message arrives that Sam isn’t going to read yet. The alert that Sam already half-knows exists. Some unfinished conversation sitting just below the surface of thought, unanswered, unresolved, quietly taking up space in the mind that was supposedly committed to this page.

Sam isn’t distracted. Sam is present. Physically here, eyes on the text, doing the right thing. The presence is real. The attention is somewhere else entirely.

The attention you were certain you had

In 1999, two psychologists at Harvard asked participants to watch a short video and count how many times a basketball was passed between players wearing white. The task was specific and demanding.

Halfway through the video, a person in a full gorilla suit walked into the frame. Faced the camera. Beat their chest. Walked out. Fifty per cent of participants never saw it. Not because it was subtle. Because their attention was committed elsewhere, and the gorilla wasn’t part of what they’d been asked to track.

We believe we’re present because we’re physically in a situation. The eyes are open. The body is there. But presence and attention are not the same thing, and the gap between them is where a great deal of modern life is quietly disappearing.

Inattentional blindness — the brain’s tendency to miss even obvious stimuli when focus is elsewhere — isn’t a rare laboratory phenomenon. It’s the operating condition of most people’s daily lives.

The brain doesn’t see everything in front of it. It samples, filters, and constructs a picture based on what it has been primed to look for. When that priming mechanism is overwhelmed — when there are too many things competing for focus — the filter becomes less a tool of selection and more a tool of exclusion.

What has changed since the gorilla experiment isn’t the cognitive architecture. It’s the scale and sophistication of the systems competing to own the priming. Every notification, every feed, every platform is fighting to be the thing your brain gets told to track. When they win, everything else becomes the gorilla. Invisible. Present. Missed.

We don’t lose the ability to pay attention. We lose the choice about where it goes.

The things that disappear without announcement

The gorilla in modern life isn’t a person in a suit. It’s the conversation not had. The warning not noticed. The colleague who needed support and didn’t get it because you were technically in the room while mentally somewhere else entirely.

It’s the child who asked a question and got a distracted non-answer, then stopped asking. The friend who was struggling and mentioned it once, sideways, and you were there but not there when it came up. The decision made quickly because the depth of focus required to make it well wasn’t available that day.

The cost isn’t always visible. It rarely arrives labelled. It accumulates in the quality of what you produce, the depth of the relationships you maintain, and the gradual narrowing of what you actually notice about the people and moments that are directly in front of you.

We have been taught to understand this as a personal failing. A concentration problem. A discipline issue. The technology industry is clear, in its habits if not its language, that this framing suits it perfectly. Because if the problem is you, you don’t look at the design.

What the design is actually doing

A system that competes for the priming of your attention, that inserts itself into the filtering mechanism the brain relies on to distinguish signal from noise, is not supporting you. It is working against the very cognitive function it claims to support.

The argument that social platforms augment connection holds only if the connections they produce are of comparable quality to those they replace or displace. The gorilla experiment suggests a direct relationship: the more demanding the competing task, the more complete the blindness. Connection that comes at the cost of presence isn’t connection. It’s substitution.

The deepest failure is this: any system that makes you selectively blind to your immediate environment cannot claim to respond to your actual circumstances. The person sitting with their child. The person driving. The person in the conversation that matters. A system that doesn’t know — or doesn’t care — which of those moments it’s interrupting has no basis for calling itself designed for you.

This is one of the questions at the centre of my book The Cognitive Crucible — not just what technology does, but what it costs us without announcing the price.

My opinion

Attention is the most finite resource we have. I don’t think the platforms taking it are confused about what they’re doing — the design is not accidental, and the framing of distraction as a personal discipline problem is not a misunderstanding either. What concerns me most is the picture of society these platforms have constructed: not a reflection of how people actually live, but of what drives engagement. The two are not the same thing, and recovering the ability to tell them apart — to notice what is actually in front of you rather than what you’ve been primed to track — is, in the most direct terms I can offer, how you get your cognitive agency back.

What this asks of you

The point is not to understand inattentional blindness. The point is to know where it is operating in your own life.

1. What have you noticed yourself missing lately — at work, at home, in conversation — that you’d have caught if you weren’t distracted?

2. Is there someone in your life who has been trying to tell you something you haven’t fully heard? What would it take to change that?

3. What would you need to change about your environment to make genuine presence your default rather than your exception?

4. If someone was watching how you spend your attention across a typical week, what would they conclude about what matters most to you?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

There is a particular kind of silence that follows a consultation. Not the silence of consideration, where something heard is being weighed. The silence of completion, where a process has been concluded and the findings have been filed. The residents submitted their responses. The staff completed the survey. The service users attended the focus group. And then, at some point later — sometimes months, sometimes longer — a decision was announced that bore little visible relationship to what anyone said.

Most people who have been consulted by a large organisation have experienced this. Many have stopped expecting otherwise. That learned scepticism is not apathy. It is a rational response to a pattern that repeats across sectors with enough consistency to suggest it is not accidental.

The pattern has acquired a name — consultation fatigue — but the name understates what is actually happening. Fatigue implies a willingness that has been worn down. What most organisations have produced is something more deliberate: a process designed to look like listening without requiring the organisation to change course as a result of what it hears.

The mechanics of being heard without being heard

The structure of most consultations gives the game away.

Questions are written by the organisation, which means the answer categories reflect the organisation’s existing understanding of the problem. Respondents are asked to rate, rank, or select from options that were designed before anyone outside the room was consulted. Free-text fields exist, but are rarely analysed systematically — they are too difficult to process at scale and too unpredictable in what they might surface. The consultation closes, the quantitative results are tabulated, and the report that emerges reflects the organisation’s original position with a margin of adjustment around the edges.

This is not cynicism dressed as analysis. It is the logical consequence of designing a consultation process around the outputs you need to produce rather than the understanding you need to reach. In local government, that output is a mandate — a defensible record that affected parties were involved before a decision was made. In the NHS, it is a compliance requirement — evidence of patient and public involvement in service design. In the private sector, it is an engagement score — something to report to the board or include in a regulatory submission. Each has a legitimate purpose. None of them requires the organisation to be genuinely changed by what it hears.

The people being consulted know this. The communities most affected by decisions are typically the ones with the least trust in the processes designed to involve them — because they have the most experience of those processes. When a community has watched a planning consultation produce a predetermined outcome, or seen a workforce survey followed by a management communications campaign rather than structural change, the rational response is to disengage. Not because the issues don’t matter to them. Because they have learned that engaging produces no different outcome than not engaging.

What the same pattern looks like across sectors

In local government, the planning consultation is the clearest case. Residents are invited to comment on a proposal that is already substantially formed. The statutory requirement is to consult; there is no equivalent requirement to be materially influenced by what is said. Communities that oppose a development learn to flood the process with responses because volume is the only variable that demonstrably affects outcomes — not the quality of the argument, not the specificity of the concern, not the evidence that the proposal conflicts with the community’s stated needs.

In healthcare, service redesign consultations run on similar logic. A trust deciding to reconfigure services will conduct patient engagement as part of the process. The engagement is genuine in the sense that it happens; it is limited in the sense that the financial and operational case for the reconfiguration is typically established before the engagement begins. What the consultation can change is the detail. What it rarely changes is the direction.

In the private sector, the employee consultation — particularly in a restructure or acquisition — operates under legal requirements in the UK that mandate a period of consultation before certain decisions are implemented. The requirement is to consult; it is not to reach agreement, and it is not to change course if the arguments against the proposal are sound. HR teams are well practised in running consultations that satisfy the legal threshold while protecting the operational decision already made.

In the charity sector, the gap is perhaps most uncomfortable to acknowledge. Organisations built on the principle of community voice frequently struggle to implement it in practice. Governance structures centre the board and the executive. Impact measurement centres the funder’s reporting requirements. The beneficiary’s account of their own experience sits at the end of a chain of representation — reported by a caseworker, summarised by a manager, condensed into an impact metric by a communications team — that has progressively removed the original voice by the time it reaches anyone with decision-making authority.

The thing that peer validation changes

The problem is not only that organisations fail to act on what they hear. It is that the processes they use mean they often do not hear clearly in the first place.

A consultation that asks closed questions gets closed answers. A free-text field that sits at the end of a fifteen-minute survey gets responses from the minority who feel strongly enough to write something additional. An engagement that channels responses through a manager or a facilitator introduces the interpretation of the intermediary before the data reaches analysis. By the time findings are presented to the people who will make the decision, the original voice has been translated several times.

The methodological shift that matters here is not just collecting open responses — it is validating them within the community before they reach the analyst. When people can see each other’s responses and indicate what resonates, two things happen that closed surveys cannot produce. First, the synthesis reflects the community’s own sense of what matters, rather than the researcher’s coding scheme. Second, people who contribute to a process and see their contribution acknowledged by peers are less likely to conclude that the exercise was performative. The experience of being heard is partly structural — it depends on whether the process was designed to hear or designed to record.

Gobby is one tool built on this logic — open responses rather than closed questions, peer-powered voting so participants can indicate what resonates, AI-assisted synthesis that reflects the community’s own sense of priority rather than the analyst’s coding scheme. It is still used by a minority of organisations. The majority continue to run processes that produce defensible records rather than genuine intelligence. The technology to do otherwise is not the constraint. The will to act on what might be heard is.

That is the harder problem, and no tool solves it. What better processes do is remove the organisation’s ability to claim it didn’t know. If the findings are specific, community-validated, and clearly stated, the decision to set them aside becomes a conscious choice rather than an administrative oversight. That is a different kind of accountability — and it is the reason some organisations prefer the consultation that changes nothing.

My opinion

I have been in enough of these rooms to know the difference between an organisation that doesn’t know how to listen and one that has decided not to. Most consultation theatre is the second kind. Tools like Gobby do something useful: they remove the claim of ignorance. If the findings are specific and community-validated, the decision to set them aside is no longer an oversight. It is a choice. And the organisations that consistently make it know exactly what they are doing.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.