Ask people at almost any company whether they use AI for work, and the honest answer is usually yes — whatever the official policy says. MIT researchers who studied enterprise AI adoption this year found that employees at more than 90% of the companies they surveyed were already using personal ChatGPT or Claude accounts for work, regularly, some of them every day — while only 40% of those companies had bought so much as one official AI subscription. The organisation didn’t approve any of it. It just didn’t get in the way fast enough to stop it.

That gap between what a business has sanctioned and what its people have already gone and done for themselves, is the real story behind a number that’s become almost a cliché this year: roughly 95% of generative AI pilots fail to deliver any measurable return, against $30 to 40 billion in combined enterprise spending. The same research found something more specific underneath that headline figure. Pilots built on generic, off-the-shelf tools reach implementation 83% of the time, because they ask almost nothing of the organisation around them. Custom pilots — the ones meant to touch a real process — mostly never get past the pilot stage at all.

What the pilots are actually funding

Look at where the $30–40 billion has gone and a pattern appears. Most of it has gone into things that generate enthusiasm rather than change: a chatbot here, a summarisation tool there, a demo for the board that photographs well in a slide deck. Each one is cheap to start and, when it doesn’t work, cheap to quietly abandon.

The harder work doesn’t get funded at the same rate: auditing whether the underlying data can actually be trusted, naming the political reason a broken process has never been fixed, admitting a failure that predates the technology by years. None of that demos well, none of it fits inside a single budget cycle, and none of it feels like progress in the moment — even though it’s the only work that would tell you whether a tool could ever have helped in the first place. So it keeps not happening, and the pilots keep multiplying instead.

The mirror, not the tool

AI doesn’t reveal what’s wrong with a business. It runs straight into it. A process that was already unclear stays unclear, just faster. Data that was already messy gets amplified rather than cleaned up, because nothing about deploying a model answers the question nobody asked about where that data came from or whether it was ever trustworthy in the first place.

The technology isn’t lying about the organisation it’s dropped into. It’s showing that organisation exactly what was already there — at a speed that makes the thing much harder to keep ignoring.

The alternative isn’t complicated

None of the harder work requires new technology. Deciding who owns a piece of data, and whether it can actually be trusted, is a decision an organisation could have made before any pilot started. Deciding who’s accountable when a model gets something wrong is a governance question, not an engineering one. Giving staff an approved tool that’s as fast and as unrestricted as the one they’ve already found for themselves is a procurement decision, not a research problem.

What makes this work hard isn’t complexity. It’s that it requires someone to admit, out loud, that the process was already broken before the model arrived — and admitting that is exactly what a shiny new pilot lets everyone avoid doing.

Who is left holding the gap

The employees quietly running work through their own AI accounts aren’t doing it to make a point. They’re doing it because the sanctioned tool doesn’t do the job and the deadline hasn’t moved. That leaves them carrying two things nobody chose to give them: the actual risk of putting company information through a personal, unmanaged account, and the actual judgement calls — what’s fine to paste in, what isn’t — that no governance process has ever talked them through.

None of that is a technology failure. It’s a business that outsourced a hard decision to whoever was closest to the deadline, then called the resulting mess an AI adoption problem.

A better model won’t fix what a business has never been willing to look at directly. It will just show that back, more clearly, and faster than before — and leave someone else to carry what it finds.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Jordan had been using the platform for two years by the time the standard was announced. Not easily — with a screen reader that could parse about half of what the interface contained, and a set of workarounds so habitual they had ceased to feel like workarounds. Every week: export the dashboard data to a spreadsheet to read it, update the spreadsheet offline, re-enter the changes manually. Three extra steps for every task a sighted colleague completed in one. The platform was not designed to exclude Jordan. It was not designed for Jordan at all.

The standard arrived as a regulatory document with an eighteen-month compliance window. Jordan did not know about it immediately. There was no reason to. Standards are announced to the organisations that have to meet them, not to the people who will benefit from them.

In Sam’s team, the announcement landed differently.

The response the organisation had prepared

Sam was not against accessibility. Nobody in the team was against accessibility. The resistance was to the timing, the resource, and the assumption that eighteen months was enough to rewrite the parts of the product the standard required. The roadmap was committed. Three major features were already in progress. The compliance work was not budgeted. The concern was not that the standard was wrong but that it had arrived at the wrong moment.

That concern was genuine in its own terms. Within nineteen months, the team would need to audit the full product against the standard’s requirements, identify every failure point, prioritise the remediation work, retest with assistive technology, and document the compliance case. For a product of any complexity, that is not a small undertaking. The estimate that came back from the initial audit was uncomfortable.

What was also in that audit, if anyone looked carefully enough, was a list of things the team had wanted to do but had never been able to prioritise. The keyboard navigation issues that the accessibility standard required to be fixed had been in the backlog for eight months. The plain language requirement had been discussed at two design reviews and deferred both times. The colour contrast failures had been flagged by one of the designers the year before, logged, and then — as things are logged — left there.

The standard had not introduced new problems. It had made existing ones compulsory to address.

There is a pattern here that runs across the history of technology regulation. When the Disability Discrimination Act was introduced, the adjustments it required were widely resisted as disproportionate by the industries subject to them. The Web Content Accessibility Guidelines were described as technically unrealistic for large, complex products. The EU Accessibility Act was described in similar terms by the industries preparing for its June 2025 deadline. Each time, the argument is that the standard is too far ahead of where organisations are. Each time, the organisations that meet the standard find they were closer than they thought — and further from where they should have been.

From compliance to floor

The reframing that changed the team’s view was not philosophical. It came from the user testing.

The standard required testing with the people it was designed to protect. The team arranged the sessions. For most of them, they were the first time anyone had watched someone use the product with a screen reader.

What they saw was uncomfortable. They watched someone work through the same tasks the team had been designing for two years. The time it took, and the places where the screen reader fell silent because there was no text equivalent for a visual element. The workaround that had developed for the dashboard export — not because it was a natural way to work, but because it was the only way to work. The product had been declared a success on every internal metric. From outside the design assumptions that had produced it, it was a product with a secondary interface for a significant portion of its potential users, and that secondary interface was made of spreadsheets and manual re-entry.

The user testing did not produce guilt. It produced information. The team had not known. Now they knew. The standard had been the mechanism that produced the knowledge, and the knowledge changed what the work meant.

The reframe - from compliance burden to design floor - followed from that session. The standard was not telling the team how good the product could be. It was telling them how bad it was no longer permitted to be. One instruction sets a ceiling. The other raises a floor. The team had been treating the standard as the former. It was the latter. Once they understood that, the work changed from meeting a requirement to building from a new foundation.

What the floor revealed

The improvements came in waves. Screen reader compatibility - the most technically complex of the required changes - produced the first unexpected consequence. Making every interface element accessible to a screen reader required auditing the entire navigational structure of the product. The audit found redundancies, dead ends, and paths through the product that nobody had designed and that made no sense to any user. Fixing the screen reader compliance required fixing the navigation. The navigation that emerged was better for everyone.

The plain language requirement produced the second wave. Error messages rewritten for cognitive accessibility were clearer to all users. Support call volume for error-related queries fell within months of the changes going live. The calls had been telling the team something about the product that the product’s own metrics had not surfaced.

Colour contrast was the third. The standard required ratios that the team’s existing palette did not meet. The redesign of the colour system produced a product that performed better in direct sunlight, on low-quality displays, and - as subsequent user testing confirmed - for users over fifty whose colour sensitivity had shifted with age. The team had been designing for a calibrated monitor in a dim office. The contrast requirement forced a wider assumption about where and how people use things.

This is the curb-cut effect: standards designed for a specific group produce improvements that serve a wider population than the standard required. The lowered kerb that allowed wheelchair access also helped people with pushchairs, delivery workers with trolleys, and cyclists. The plain language, the contrast ratio, the keyboard navigation did the same thing. Jordan could use the product. So could everyone else, more easily than before. The floor that had been raised for Jordan turned out to be below where the product should have been all along.

The team had not set out to make a worse product than they were capable of. They had made the product their design process was equipped to produce, for the users their design process had in mind. The standard forced a wider design process. The wider design process produced a better product.

That sequence is not unique to accessibility. External requirements have forced expanded scope and produced better outcomes in electrical safety, food labelling, and data protection. The resistance is always first. The improvement follows.

What a standard is really testing

The question a well-designed standard asks of a technology product is not only whether it is accessible to people with disabilities. That is the specific requirement. The deeper question is whether the product assists the people who use it — all of them, not only the ones whose needs were visible in the original design brief.

A product that cannot be used by someone with a screen reader was, by definition, only helping the users whose circumstances it had accounted for. Meeting the standard did not add an accessibility layer to a complete product. It completed the product for the first time.

The second question is whether the product adds something real to the capability of the people who use it, or only enables interaction. Plain language that reduces cognitive load does not just make the product reachable - it makes it more useful. The user who previously needed three support calls to understand an error message needed none after the change. That is not accessibility as a separate category. That is a better product, full stop.

The third question - the one that runs underneath the history of the Disability Discrimination Act, the Web Content Accessibility Guidelines, and the EU Accessibility Act - is adaptability. A product designed for one profile of user, however carefully designed, is not adaptive. A product that has been required by a standard to respond to a wider range of human circumstance has, in meeting that requirement, become demonstrably more so. The standard did not produce adaptability. It required the work that produced it. That is what a well-designed standard does: it requires the work the organisation would not otherwise do, for the people the organisation had not otherwise considered.

My Opinion

Technology follows the path of least resistance. So does design. We build for the user we imagine, in the circumstances we assume - and that user does not exist. Real people are older, less sighted, less confident, using screens in conditions we did not test for. A standard forces the reckoning with that gap. It should not have to.

Four questions for the next standard

1. Think of a regulatory or technical standard that applies to technology you build or use. Has meeting that standard improved the product for a wider population than the standard required? If you do not know - would it be worth finding out?

2. If your organisation is currently resisting a new accessibility or ethical design standard, what is the specific argument against it? Is that argument based on evidence of what the standard would cost and produce - or on an assumption made before the work was done?

3. If you are involved in setting standards - in government, in industry bodies, in procurement specifications - are the people the standard is designed to protect part of the process by which it is written? And if not, how confident are you that the standard actually requires what those people need?

4. The argument against technology standards is consistently the argument of the organisation that has not yet had to meet them. What would it take for your organisation to be the one that argues for higher standards rather than against them - and what would that mean for the people your technology is supposed to serve?

Jordan can use the platform now. Not with workarounds. Just - use it. The standard required that as a minimum. The team went further because, once they understood what the floor was, they found the ceiling was higher than they had thought. The standard did not limit what was possible. It clarified what was no longer acceptable.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The email arrived at 3.17 in the morning.

Sam hadn’t slept. They’d been sitting at their kitchen table for four hours watching the progress bar on their screen crawl — file after file locked, folder after folder gone dark. The attack had started somewhere in the accounts system, spread quietly through the network, and by midnight the company’s billing platform, its customer database, and eighteen months of project files were encrypted behind a key Sam didn’t have.

Then the email came. It was polite. It opened with an acknowledgement that this was a difficult situation. It explained, clearly and without technical jargon, what had happened and what the options were. It included a link to a payment portal with step-by-step guidance. It offered a live chat facility for anyone who needed help navigating the process. At the bottom, there was a note that a support representative was available around the clock and would typically respond within the hour.

Sam had never received customer service this good from anyone.

The support desk that worked

This is not an unusual story. The criminal organisations behind ransomware attacks have built support infrastructures that would be impressive in any legitimate technology business — documentation in multiple languages, and in some documented cases a rating mechanism through which affiliates — the criminal operators who carry out attacks on behalf of the platform — could evaluate the service they received.

They have done this because it works. A victim who cannot navigate the payment process is a victim who might not pay. Friction costs money. They removed it.

The DarkSide group, responsible for the 2021 attack on Colonial Pipeline in the United States, operated with the organisational structure of a funded technology company. They developed proprietary tools, leased their capabilities to vetted affiliates who carried out attacks on their behalf, and maintained a support function specifically to guide victims through the payment and decryption process. The attack locked down the billing and operational systems of the largest fuel pipeline on the US East Coast, triggered fuel shortages and panic-buying across multiple states, and prompted a federal emergency declaration. Colonial Pipeline’s CEO, Joseph Blount, confirmed to the Senate Homeland Security Committee that the company had paid the ransom — 75 bitcoin, approximately $4.4 million at the then-exchange rate. The ransom note had demanded approximately five million dollars.¹ The support desk was open the entire time.

The question this raises is not primarily about the criminals. Their logic is coherent. They are running a revenue model and they have optimised it. The question is about what their model reveals about everything else.

What the legitimate system was built to do

A person dealing with a digital crime — whether ransomware, fraud, or identity theft — typically encounters a response system that was designed for a different era and a different scale of problem. The bank’s fraud line connects them with an adviser working from a script that was written before this kind of crime existed in its current form. The police non-emergency line takes a report and assigns a reference number. The relevant government body sends an acknowledgement within five working days.

Each of these institutions is staffed by people trying to help within a structure that was not built to help at speed, at scale, or at the point of maximum need.

The criminal support desk is better not because criminals are more talented. It is better because it was designed entirely around what the paying customer needs at the moment they need it. That is a design choice. The legitimate systems were designed around something else — compliance, liability, institutional process, historical precedent — and the human being in distress is not at the centre of any of it.

“The ransomware gang had better customer service than the bank that was supposed to help you recover from them.”

When a fraud line or a government body responds to a digital crime victim, the response is designed to help that person complete a process — file a report, receive a reference number, trigger a workflow. That is not the same as helping them do what they actually need to do. The paper trail grows. The problem does not. And the response does not adapt: it does not know where this specific person is right now — technically, emotionally — and it was never built to find out.

The criminal support desk was built around exactly that question. Not because of any ethical commitment, but because its commercial success depends on it. That is the uncomfortable mirror it holds up. The legitimate system, by contrast, was designed to be defensible. That is an institutional virtue. It is not the same as a human one.

The Colonial Pipeline attack disrupted fuel markets across multiple states for six days, driving fuel prices above three dollars a gallon for the first time since 2014 and prompting panic-buying across the south-east of the United States.² The polite emails were in service of extortion.

What the comparison reveals

The argument is that something in the legitimate system has been designed wrongly, and that the criminal model makes the wrongness visible by contrast. When a fraud victim gets better guidance from the people who defrauded them than from the institution that holds their money, that is not an anomaly. It is a diagnostic. The system that was built to protect and assist that person has, somewhere in its design, prioritised something other than them.

That matters because the design is not inevitable. It is a series of choices, made by people, in institutions, at particular moments in time.

My Opinion

Cyber crime is not the work of individuals in dark rooms. It is a structured business model, operated by organised criminals with the discipline and infrastructure of any revenue-generating enterprise. The support desk is not absurd — it makes complete commercial sense. Once you accept that framing, the question for every organisation changes. Not “could this happen to us?” but “what is our blast radius, and what do we have in place to recover?” This is not going to become simpler or rarer. Crime now operates at the speed of code. Our response still runs at the pace of policy and procedure. That is an asymmetric challenge, and naming it is the first step to treating it seriously.

This is the territory that The Shadow System covers in its opening section — the professionalisation of the criminal economy, and what that professionalisation reveals about the failures of the legitimate one.

Your experience is part of this

These patterns are not abstract. The design failures described here accumulate in individual moments — a fraud call that went nowhere, a report that produced only a reference number, a process that kept moving while the person at the centre of it stood still. If you’ve encountered any part of this, your account is relevant evidence.

• Does this story relate to you — have you ever found the process of reporting a digital crime or fraud more difficult than the crime itself deserved?

• If you have your own experience of how institutions responded when something went wrong digitally, share it. These patterns only become visible when enough people name them.

• If you could redesign the response a fraud or ransomware victim receives from legitimate institutions, what would be the first thing you would change?

• And the harder question: is this a technology failure, a culture failure, or an incentive failure — and does the answer change what we should do about it?

Leave a comment

References

¹ Joseph Blount, CEO, Colonial Pipeline — Senate Homeland Security and Governmental Affairs Committee testimony, 8 June 2021. Blount confirmed the payment was 75 bitcoin (approximately $4.4 million at the time of payment); the original ransom demand was approximately $5 million. The FBI subsequently recovered approximately $2.3 million by seizing 63.7 Bitcoin (the value had fallen since the date of payment). Sources: CNBC; Nextgov/FCW; Department of Justice press release, 7 June 2021.

² Colonial Pipeline shut down 7 May 2021, restarted 12 May 2021. Fuel price and disruption data confirmed across multiple sources; note that a University of Kansas study (December 2021) found the longer-term gas price impact was more localised than early reporting suggested. Sources: CISA retrospective; University of Kansas study.

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Jordan is sitting with a member of staff who will use the new system every day. Not in a workshop. Not in a project meeting. In the room where the work actually happens, watching how it is done, listening to what is said about it.

The person Jordan is talking to says something that stops the conversation mid-note. Not a complaint — a small, specific observation. A workaround that everyone on the team uses, that nobody has mentioned in any survey or requirements session because it has become invisible through familiarity, and that had the visit not happened would have been absent from every document the technology decision was based on.

Jordan writes it down. It will change what is built.

What the standard process does not capture

The technology decision process here is not badly designed. The business case is thorough. The vendor shortlist is credible. The project workshops have run, and the requirements document reflects what was said in them. The governance process is in place.

What the formal process cannot capture is what is now being heard. The requirements document reflects what the previous system did — the starting point, not the need. The project workshops produced consensus around the visible problems, the ones that are easy to articulate in a meeting room to a project manager with a template. The satisfaction survey measured experience against the current system’s performance, not against what the work actually requires.

Each layer of mediation between the decision-maker and the person doing the work reduces the fidelity of what arrives. The user research output has been simplified for an executive audience. The support tickets have been categorised and aggregated. The workshop outputs have been translated into requirements language. By the time the information reaches the decision-maker, the texture has been processed out of it — the hesitation before an answer, the third thing said after the first two, the small specific observation that reveals the real constraint.

The visit was to hear what the processing had lost.

Going before the direction was fixed

The conditions that distinguish this kind of listening from the kind that confirms rather than informs are timing, directness, and repetition — and each runs against what the project calendar prefers.

The first condition is timing. Going before the strategy is fixed — before the vendor shortlist is final, before the requirements are locked, before the business case has closed around a particular direction — means the listening can still change things. Listening that happens after the direction is set validates rather than shapes. The project team hears what it hoped to hear, because the questions have been shaped by what is already being built, and the person being listened to can sense the direction and answers accordingly. The visit happened when it could still matter, which meant going when the schedule pressure made going feel like a distraction from the real work.

The second condition is directness. Going in person — not through a project manager or a user research agency — means sitting with the people, watching the work, hearing the words in the order and with the weight the speaker intended. There is a real role for structured user research in technology decisions. But there is something it cannot carry: the unprocessed signal that reaches a decision-maker only when the decision-maker is present to receive it.

The third condition is repetition. The first visit produced one set of things heard. The second produced different things, because the trust that had not existed in the first conversation had started to build. People answer differently when they know the person asking is genuinely interested rather than completing a process — and that knowledge does not form in a single visit.

In the third conversation, someone said something they had not said in the first two — not because they had withheld it, but because it had taken that long for the context to feel safe enough for honesty to reach what actually mattered. Single visits produce first answers. Repeated visits produce the ones underneath.

What was not in any document

The visit that mattered most was the third one.

The spreadsheet had come up in the first visit — the workaround every person on the team maintained alongside the current system, because the system could not hold notes in the right place at the right stage in the process. Three years in, nobody had mentioned it in any requirements session because it had become invisible through familiarity. The requirement was added on the way back.

The second visit changed a vendor criterion. Someone was working with three applications open simultaneously, constantly switching, returning to the system from a backgrounded state. The shortlisted vendors had been evaluated against their interfaces during structured demos — clean, navigable, well-presented.

The question of how each vendor’s product behaved when interrupted and resumed had never been asked. It was asked after that visit. Two vendors answered adequately. One did not. The shortlist changed.

What the third visit produced was different. Someone described the hardest part of their day — a step that happened immediately before they opened the system. Technically outside the system’s scope, but one the implementation could have absorbed if the handoff had been designed differently. That single observation reshaped the implementation in a way that addressed the friction at its source, rather than treating the system boundary as a fixed line. The first two visits had not produced it.

None of the changes were expensive. Each required someone to have been in the room. The requirements document that did not include the notes capability was not careless — it simply had no mechanism to surface what the direct conversation produced.

Together, the changes constituted a technology decision made on what was actually true, not on what the formal process had captured. Not complete information — that is never available. But closer to what was actually true than any document had been.

Three questions a leader cannot answer from data alone

Before a significant technology decision, there are questions that no amount of analysis will answer reliably without direct listening.

The first is whether the technology will genuinely help the people who most need it to work. Not the average user in the user story — the person whose day is hardest, whose work is most complex, whose need the requirements document has approximated but not named. That answer is in the room where the work happens. Direct listening — going there — is the only way to hear it. The leader who does not go will answer it from assumption, and the assumption will be derived from data simplified precisely where the difficulty is most real.

The second is whether the technology will genuinely add to what the people it serves can do — or substitute one set of constraints for another. The genuine gain in capability shows up in the specifics of what people find difficult and what they could do if that difficulty were removed. Those specifics are not in aggregated data. They emerged in the second visit, from a person describing a capability they currently lacked that the new system could provide if designed to carry it. That capability had not appeared in any document because no document had asked for it directly.

The third is what the system needs to adapt to — the actual range of people and circumstances it will encounter in use. The answer is not in the modal user’s profile. It is in the outlier’s experience, in the third conversation with the person who finally said the thing they had been circling. Going back is the only way to hear it. A leader who makes the decision without returning will design for the average and discover the outliers after the system is live, when adaptation costs considerably more.

My Opinion

I have sat in enough rooms to know that the gap between the requirements document and what is actually needed is structural, not accidental. At one university, a single conversation around a table surfaced needs nobody had articulated — and capabilities some groups had no idea others could use. I have run procurement exercises where, over four days, real teams put real problems in front of vendors and evaluated how they responded. The vendors that won were not the ones with the best product. They were the ones that listened before they pitched.

Questions worth taking into your next decision

The four questions below are not rhetorical. They are worth answering — for yourself, before the next decision closes.

1. Think of the last significant technology decision you were involved in making. When did you last speak directly — not through a survey, not through a user research report, but in person or by phone — to someone who would be affected by that decision? What did you hear, and what did it change?

2. If you lead technology decisions, what is the mechanism in your process that brings direct, unmediated human experience to you before the strategy is fixed? If that mechanism does not exist — what is stopping you from creating it?

3. Think of a technology decision you made that did not land as intended. At what point in the process would direct listening have changed the direction? Was that opportunity available and missed, or was the process not designed to create it?

4. Is direct listening a luxury that busy technology leaders cannot afford — or is it the one thing that makes everything else more efficient by ensuring the direction is right before the resources are committed?

A McKinsey and University of Oxford study of large IT projects found that budget overruns averaged 45 per cent — but shortfalls in delivered value averaged 56 per cent. Those are not the same problem. The value loss lands quietly, across years of implementation, workaround, and eventual replacement.

Jordan’s listening did not guarantee the right direction. It made the wrong direction less likely — at a cost in time that is negligible against the cost of not going at all.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam walks the lower fields in winter before anything else. Not because it is efficient. Because it is the only way to know what the overnight rain has done — whether the water is moving through or pooling, whether the drainage that has been slowly improving for three years has held, whether the flood-prone corner that floods briefly and recovers quickly if left ungrazed for six weeks is beginning to saturate. This knowledge cannot be looked up. It was built over twelve years of walking this ground in all weathers, at all hours, through all seasons.

The soil sensor is in the barn. It has been there for fourteen months.

The phone would show no signal here even if they took it out. It has shown no signal on this part of the land for as long as they can remember. That is not a technical problem waiting to be solved. It is a characteristic of these forty acres, in this valley, in rural Wales. The land is what it is. They are doing what they have always done: feeling the soil, watching the drainage, reading what this particular ground is trying to say.

What the grant assumed

The precision agriculture package arrived two years ago with a government agri-environment grant attached and a marketing brochure that described “nature-inspired data systems designed to work with natural variation.” They spent three weeks setting it up: the soil moisture sensors pushed into the ground at the intervals the manual specified, the GPS-guided monitoring app configured on the tablet, the yield-mapping software calibrated against the field boundaries.

The sensors required persistent 4G signal to transmit data to the cloud platform. Signal on the lower fields drops to zero.

The app’s recommendations were calibrated for large-scale arable monocultures: uniform rows, single-variety plots, conventional seasonal rotation. This is a twelve-year permaculture polyculture — a food forest, a market garden, mixed livestock — designed around the specific drainage, soil, and microclimate of this ground. The app had no logic for any of it. The yield-mapping tool produced output that mapped onto nothing being grown there or recognisable on this land.

The problem was not the setup, or the interface. It was that the technology had been designed for a farm this land was not, and no amount of patient configuration was going to change that. Within eighteen months, the sensors were back in the barn and the grant had been spent.

The principle the industry never borrowed

Permaculture has a founding principle, stated plainly since Bill Mollison and David Holmgren developed the practice in 1978: observe and interact. You do not design until you know this place. You walk it across seasons. You watch what it does in rain and drought and frost. The design emerges from sustained attention to specific conditions — not from a template applied to all conditions.

The precision agriculture industry describes its products as nature-inspired. The language is in the brochures, the grant applications, the conference presentations. Adaptive systems. Intelligent sensors. Biomimetic design. What the industry has borrowed is the vocabulary. The founding principle — observe first, design second, never assume the average applies here — was left behind.

The reason is structural. Technology development is funded by scalability. A solution that works for eighty percent of farms is investable. A solution built from sustained observation of this specific land, and a different solution built from sustained observation of the farm next door, is not.

The observation step — the one that would reveal what this land actually needs — is designed out before a line of code is written. Scalable and specific are not the same thing.

In nature, there are no average organisms. Only specific ones, each the product of sustained adaptation to particular conditions. The precision agriculture platform was built on the logic that farms are variations on a theme. This land is not a variation. It is a place.

What twenty-five years of forecasts missed

In March 2026, a peer-reviewed study published in Agribusiness (Wiley) tracked twenty-six precision agriculture technologies across twenty-five years of adoption forecasting data. The researchers — Malone and colleagues — worked through the CropLife–Purdue Precision Agriculture Dealership Survey, the longest-running continuous assessment of precision agriculture technology in the United States. Their finding: input dealers overestimated adoption on nearly every technology, in nearly every year of the study. The overestimation gap widened in the early 2020s, as adoption on several technologies flattened or declined while dealers continued to project steady growth.

The industry’s explanation has been consistent: this is an adoption problem. Farmers are slow to change. They lack the skills, the confidence, the willingness to invest. Twenty-five years of the same explanation, in the face of twenty-five years of the same forecasting error, is not an explanation. It is a refusal to look at the other possibility.

The NFU’s 2025 connectivity survey puts a number on part of the gap. Only 22% of UK farmers have reliable mobile signal across their whole farm. Twenty-one percent of UK farms have broadband speeds below 10 Mbps — compared with a national average of less than one percent. The technology assumes infrastructure that the majority of small mixed farms in rural Britain do not have. These lower fields are not an outlier.

They are the norm.

The grant funding attached to the precision agriculture package does not ask whether the farm has 4G. It asks whether the farmer can demonstrate a commitment to sustainable intensification. They could. The application process had no field for “my lower fields have no signal,” any more than the monitoring app had a field for “food forest polyculture.” The system that allocated the grant and the system that failed to work were built on the same assumption: that the farm in front of them was a version of the average farm, and that any deviation from the average was the farmer’s problem to solve.

The kingfisher and the train

In the early 1990s, a Japanese engineer named Eiji Nakatsu was tasked with eliminating the sonic boom produced by what would become the Series 500 Shinkansen when it exited tunnels. The existing nose design was aerodynamically efficient in open air and catastrophically loud underground, where the pressure wave had nowhere to go. The engineer was a birdwatcher before they were a lead engineer, and the problem reminded them of something they had spent years watching: kingfishers moving from air into water without producing a pressure wave. The kingfisher’s beak is a precise solution to that exact problem — the transition between two media of different densities, at speed, without disturbing the surrounding medium.

The engineer did not spend a two-week research sprint watching kingfishers. They had been watching them for years, as a matter of personal attention rather than institutional programme. The insight was available because observation had been treated as real work, sustained over time, without a deliverable attached to it. The redesigned nose reduced the sonic boom, made the train 10% faster, and reduced electricity consumption by 15%.

The technology that arrived at this farm had borrowed nature’s language. It had not borrowed the engineer’s method. It had not spent seasons watching what these lower fields do after overnight rain. It had not asked what twelve years of permaculture observation had produced.

That accumulated knowledge — the only knowledge that could optimise this specific land — was invisible to the system’s design. It was not data. It was noise.

What observation asks of any system

The common thread between permaculture’s founding principle and the engineer’s decades of watching is not the sector or the technology. It is the decision to treat observation as real work, completed before design begins. The questions that follow are worth bringing to any system you deploy, inherit, or commission — because the answers will tell you whether the observation step was there, or whether it was designed out.

1. When a technology you use, or your organisation deploys, was designed — how much time was spent observing the specific conditions it would operate in, rather than the average conditions it was modelled on? Would you know if the answer was none?

2. What knowledge do you carry — about your work, your land, your patients, your tenants, your community — that exists only in your head and has no field in the system you’ve been given? What happens to a decision when that knowledge is absent?

3. If the technology you’re currently using was built for someone else — a different scale, different context, different connectivity — would the evidence be visible in how it performs, or only in what it cannot do?

4. Permaculture’s first principle has been available since 1978. What would technology design look like if it had the same founding requirement: that you do not design until you have observed?

Sam still walks the lower fields in winter. The soil sensor is in the barn. The land is doing what it has always done: varying, specific, unaverageable. The technology that was supposed to augment that knowledge had no knowledge of this place. It never did. It was built for somewhere else.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Alex is in the middle of a routine task when it happens. Not an epiphany — a small, specific thing. The step that used to require four form fields now requires one. Alex completes the task and then pauses, trying to remember when the change appeared, because it was not announced and it was not the subject of a release note and Alex only notices it now because the friction that used to be there is gone.

Somewhere in the background of that pause is a support call from eighteen months ago. Alex had reported exactly this — the unnecessary duplication in the form, the way the same information was asked for in three different places. It had taken fifteen minutes longer than it should have. Alex had said so. The call had ended with an apology and a reference number.

And now the form was different.

What usually happens to feedback

The standard experience of providing feedback to a technology product is an exercise in patience with diminishing returns. The survey that arrives after every interaction, rated one to five, with a final optional text box that almost no one fills in because nothing has ever clearly happened as a result of it. The support ticket that generates an automated acknowledgement, a reference number, and silence. The complaint that is logged, managed, and resolved — which means it is closed, not that the thing that caused it has been changed.

None of these are careless. They are systems doing what they were designed to do. The survey tracks satisfaction, the ticket resolves the case, the complaints process manages liability. None of them were designed to change the product. That distinction matters more than it sounds, because a system designed for one purpose will not accidentally perform another, however good its data.

The difference in Alex’s case was not that the organisation cared more. It was that the system had been designed for a different purpose: to move information from the person experiencing the product to the people building it, and to do something with what arrived.

The three conditions that make a loop

A genuine feedback loop breaks the moment any one of three conditions is missing.

• Feedback that works has to be easy to give — not a survey sent after the fact, not a formal process initiated at a separate URL, but a mechanism present at the moment the friction is felt. Alex did not fill in a survey. Alex called to report something broken, while it was broken, with the specific detail of what had gone wrong. The signal was clean because it was immediate and unsolicited — the kind of feedback that surfaces the actual experience rather than a retrospective reconstruction of it.

• Getting that signal to the people who can act on it is harder than it sounds. In most organisations, the path is long and obstructed: feedback arrives at customer service, which resolves individual cases and closes tickets, while patterns that would be visible to the product team — if they received the same data — never reach them. Alex’s organisation had a short path: a shared channel between support and engineering, a standing item in the sprint review for user-reported friction, a team that treated incoming complaints as product development input rather than cases to be managed. The information arrived with its context intact and at the people who could do something about it.

• Visibility is what closes the loop. Not announced, not marketed — just present. The next time Alex used the service, the thing that had been difficult was different. Without that, the person who provided the feedback has no reason to believe the loop exists and no reason to provide the next signal honestly. A feedback system that receives, processes, and acts — but never demonstrates that it has done so — produces exactly the same user behaviour over time as one that does none of those things. Trust requires evidence, not intention.

All three conditions were present in Alex’s case. Remove any one and the loop would have broken.

What two years of listening produces

Over two years, the service Alex uses has changed in small, specific ways that trace directly back to what people who used it said.

The file upload failure message used to say “Error: upload rejected.” Alex had reported this after twenty minutes trying to diagnose which of five files was causing the problem. The message now says which file failed, why it failed, and what the accepted formats and size limits are. The change took a morning to implement. The information needed to make it had been sitting in the support queue for eight months.

The address entry flow used to require four separate fields in an order that did not match the way most people read their own address. The most common support call in one quarter was people unsure which field their flat number belonged in. The form now uses postcode lookup. Completion time for that step fell by two-thirds.

A third change is smaller and easy to miss. The notification sent when a request was submitted used to say “Your request has been received.” Alex had written, in a feedback form, asking whether anyone actually looked at these requests — because the notification contained no information about what would happen next, or when, or who to contact in the meantime. The notification now includes an expected response time and a named contact for queries. It took one line of copy and the answer to a question that no one on the product team had thought to ask, because no one on the product team had ever waited on the other side of the process they had built.

The cumulative effect is a service that feels attended to. Not perfect — attended to. That quality is not cosmetic. It is the specific evidence that someone received what was said and acted on it, and it is the foundation of whatever trust Alex now places in the service.

What the design is actually for

1. Think of a digital service you use regularly. Can you point to a specific change that service has made in the past year that you believe was a response to real user experience? If you cannot — what does that tell you about whether the feedback loop exists?

2. If you build or manage a technology product, how does feedback from real users reach the people who can act on it? Is the path direct and short, or does it pass through layers that filter, aggregate, and delay?

3. If you have ever provided feedback to a technology product — a support call, a complaint, a survey response — were you ever aware of a change that resulted from it? How did that experience affect your relationship with the product and the organisation?

4. Continuous improvement is standard practice in manufacturing and in software development. Why is continuous listening — treating user feedback as an ongoing product development input rather than a post-launch compliance exercise — still the exception in most technology organisations?

Most feedback systems are designed to manage the organisation’s relationship with feedback rather than to let feedback drive the product. The organisations that reverse that design — that make the path from user experience to product change as short and unobstructed as possible — build technology that improves in the direction that matters. Alex noticed. That noticing is rarer than it should be.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Alex puts the phone face-down on the table while eating dinner. Not because there’s an argument brewing, not because anyone has asked. Just a small, deliberate act. The meal lasts forty minutes. At some point — it turned out to be fourteen minutes in — the phone gets picked up. There was no notification. Nothing rang. The impulse arrived with no identifiable origin, like a hand that reached before the mind could object.

It doesn’t feel like a problem. Just a habit. Something to work on. A matter of willpower.

That framing is exactly what the people who designed that impulse were counting on.

The story we were given

We have been taught to understand compulsive phone use as a failure of self-control. The language we reach for is personal: distracted, addicted, weak, unable to focus. Self-help books tell us to build better habits. App stores sell us tools to limit our screen time. The diagnosis is individual, and so the cure is individual.

This framing is not neutral. It is convenient — and it is convenient for a specific set of people.

The companies whose platforms you cannot stop checking did not produce compelling products by accident. They built something more precise: a compulsion. And they built it using the same mechanisms that keep a gambler at a slot machine at three in the morning, long past the point of enjoyment.

What the engineers understood

Variable reward schedules are the most reliably powerful mechanism in behavioural psychology. The principle is simple: when the timing and size of a reward is unpredictable, the brain cannot habituate to it. It cannot stop paying attention. It keeps seeking.

A vending machine that always delivers the same snack is satisfying but not compulsive. A slot machine that sometimes pays out, in unpredictable amounts, at unpredictable intervals, is something else entirely. The unpredictability is not a design flaw. It is the entire point.

Social platforms were redesigned around this principle deliberately. The pull-to-refresh gesture works like a slot machine lever — and what it delivers operates on the same principle: sometimes a funny post, sometimes a message from someone you love, sometimes nothing worth reading. The brain cannot tell in advance. So it keeps pulling.

The Hook Model — one of the most cited models in social platform design — describes the loop explicitly: trigger, action, variable reward, investment. It was not developed to help you get things done. It was developed to make sure you come back.

What the misdiagnosis costs

When we believe the problem is personal, we look for personal solutions. We download another app. We set a screen time limit. We move our phone to another room at night. Some of these things help, at the margins. None of them changes the underlying architecture.

The system benefits from our misdiagnosis. They want the conversation to remain about you.

The human cost is not abstract. Attention fragmented over years becomes harder to sustain. The capacity for deep work, for genuine presence in a conversation — these are not personality traits immune to outside pressure. They are capacities that can be eroded. And they are being eroded, deliberately, at scale.

Children who grow up inside these systems will have spent their formative years in an environment specifically engineered to prevent them from consolidating attention. The long-term consequences of that are only beginning to be understood.

Against the user

A system that manufactures compulsion rather than enabling choice is not assisting the person using it. The platform’s interest and the user’s interest diverged the moment engagement became the goal rather than the outcome of delivering something worth using.

These platforms amplify a biological vulnerability — the brain’s inability to habituate to variable rewards — and direct it toward commercial ends. That is not augmentation in any useful sense. It is a diminishment dressed as a feature.

The algorithm adapts, yes — but it adapts to exploit you more efficiently. It learns what keeps you engaged, not what serves you. These are different objectives. In most people’s lives, they are opposing ones.

A system that does not assist, does not genuinely add something, and does not adapt to serve is not a neutral tool that some people misuse. It is a system designed, tested, and refined to work against its users’ interests while appearing to serve them. That is where the responsibility lies.

The questions underneath the habit

• When you reach for your phone without knowing why, what exactly are you looking for?

• Has there been a time when you tried to reduce your phone use and found it harder than you expected? What story did you tell yourself about why?

• If you discovered that a product you used every day had been deliberately engineered to make it difficult for you to stop using it — what would you want to be different, and who would be responsible for making that change?

• Is the design of the technology in your life something you have ever actively chosen, or is it something that happened to you?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Jordan noticed it on a Tuesday morning, about four months in. They had opened the app the way they opened other apps — without thinking about it, on the way to something else. The result was there when they looked: a number, with a short explanation of what it meant in relation to where it had been last week. Jordan read it in the time it takes to make coffee. There was nothing to decode.

That moment — a test result arriving with context, at a time they could actually do something with it — was not dramatic. It did not change the condition. Jordan has a condition that requires regular monitoring, and it will require monitoring indefinitely. But something had shifted in the daily experience of managing it. The technology had been built to serve that experience, not to process it. For the first time in years of navigating health systems, the difference was legible.

The system Jordan already knew

Before the app, the management of Jordan’s condition was a patchwork held together by their own effort. Test results arrived on paper, sometimes days after the event, with reference ranges printed beside the numbers but no interpretation of what those numbers meant in context. They went into a folder maintained on their own — disconnected from the records at the GP surgery, disconnected from the records at the specialist clinic. The folder existed because Jordan had created it. No one had suggested it; no system supported it.

Appointments were bookable, but the booking required persistence — the kind that is straightforward when you are well and difficult when you are managing a condition that affects concentration and energy. Jordan had learned which times to call, and which numbers to try when the first was engaged. These were not skills anyone had expected to need.

The condition itself was manageable. What was not manageable, not sustainably over years, was the administrative overhead the health system had attached to it. The paper trail that was entirely the individuals responsibility. The records that told a different story depending on which part of the system you were looking at. The appointments that were partly administrative events — catching up on information that should have been shared already, establishing a baseline that should have been continuous.

The ordinary cost of managing a long-term condition in a system not designed for the person managing it is a cost that most people carrying those conditions pay, quietly, without counting it as a cost. The honest context is that health technology has not, on the whole, served people in this situation well. Two thirds of NHS patients and carers reported at least one administrative problem in the past year, according to the King’s Fund in 2025 — disproportionately those managing long-term conditions. That history is part of why the difference, when it came, was so immediately recognisable.

The question they asked first

The team that built the app began with a question rather than a specification. Not: what should this app do? But: what does it feel like to manage this condition, and where does the friction concentrate?

Those two questions produce different software. The first produces a list of features. The second produces an understanding of what is actually hard about the problem — which is rarely what the first question reveals, and rarely what someone without the condition imagines from the outside.

The answer, when they gathered it, was specific. The friction was not in the absence of information but in its format and its timing. Information arrived too late, without context, through channels not connected to anything else. The friction was in the administration of care, not in the care itself. The app was designed to reduce that specific friction, in that specific form, for people in that situation.

What came next followed from that question. The design was tested throughout development with people who found digital tools genuinely difficult to use — not only with people who were already comfortable with health apps and quick to forgive poor design. The team was accountable, internally, for whether the people using it could actually manage their condition better — not only for whether the app was delivered within budget and adopted within target timescales. And the design was iterated after deployment, based on how people used the app in practice rather than how they had been expected to use it.

None of that requires a large budget or an unusually skilled team. It requires a set of priorities — a decision, made at the start and maintained throughout, that the person using the technology matters more than the efficiency of the organisation providing it. That decision is not always made. When it is not, the technology reflects the absence.

What actually changed

Jordan’s condition is no better or worse than it would otherwise have been. What is better is the experience of managing it, day to day, in the space between clinical appointments.

Test results that used to require interpretation now arrive pre-interpreted — with enough context to inform a decision, to notice a pattern, to decide whether something warrants a call before the next scheduled appointment. No more calls to the surgery to ask what a number means. The conversation with the clinical team is more useful because the patient arrives with information, and the team arrives without gaps to fill. What used to be a partly administrative appointment is now a clinical one.

The relationship with the clinical team has changed because the technology removed the noise that used to surround it. The paper trail is gone. The catch-up is gone. The time spent establishing what the records should already show — gone. That is not a small improvement in a clinical relationship. It is a different kind of care.

There is also more time. Time that used to go to navigating systems that were not working — chasing results, maintaining the folder, ringing back — is available for other things. The calculation is not large on any single occasion. Over years, it compounds.

None of this is dramatic. The condition is still there, still requires attention, still shapes daily life in ways that are not going to change. But the daily weight of managing it is lighter than it was, and that lightness is the direct result of a design team that started with actual circumstances rather than a generic idea of what a patient requires.

What the design actually did

The app does one thing well: manage this condition, in these circumstances, with these constraints. Not a generic patient managing a generic condition — a specific person, with the particular shape of their daily life. The help it provides is specific to where the friction actually is, rather than matched to what the average health app is assumed to need. That specificity is the direct consequence of the question they asked first.

The app adds something that was not there before. The person using it can now participate in their own care in a way that was previously impractical — reviewing their own data with context, preparing for clinical conversations with real information, communicating between appointments without the friction that made that communication more trouble than it was worth. The participation remains theirs. The app did not take over the management of the condition. It provided better tools to manage it.

That distinction, between technology that substitutes for a person’s agency and technology that extends it, was a design decision, not an accident. It was made at the start.

The app also responds to the individual. Over time it reflects what this person uses, finds useful, needs at different moments. The personalisation is not surface-level — it changes how the app functions rather than how it looks, making it more useful over time rather than more impressive in a product demo. That responsiveness requires a feedback loop between user and team — one that exists only because someone was accountable for outcomes. Which is, again, the decision made at the start.

Four questions for the next project

1. Think of a piece of technology — an app, a service, a system — that has genuinely helped you, in a way that felt specific to you rather than generic to all users. What made it different? Can you identify the design decision that produced that experience?

2. If you build or commission technology, how does your design process ensure that the person who will use it is the starting point rather than the endpoint? What specific mechanisms — genuine user research, inclusive testing, accountability for the human outcome — are in place to hold that priority throughout development, not only at the requirements stage?

3. What would it take, in your organisation or sector, to make conditions like these the standard rather than the exception? Not in five years — in the next project. What is the single first change that would make the most difference?

4. Is the gap between technology that burdens and technology that helps primarily a technical gap, a resource gap, or a priorities gap? And if it is a priorities gap, what does that tell us about who has the power to close it?

The conditions that produced an app like this are not secrets. They are known, replicable, and not especially expensive. The gap between what exists and what is possible is not technical. The reason it persists is a question for the people who make the priorities, not the people who write the code.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Alex submitted the third query through the portal on a Thursday afternoon, copying in the reference numbers from the first two. The housing benefit calculation still didn’t match what Alex had been told over the phone three months earlier. The difference was not large — roughly forty pounds a month — but over four months it had accumulated into an overpayment notice that Alex was now expected to repay.

Each of the three queries had been acknowledged within twenty-four hours. Each had been assigned a case reference. Each had received a response. The most recent response explained, in clear language, how housing benefit calculations worked. It did not explain why Alex’s calculation differed from the figure given on the phone. The case had been marked as resolved.

The week after Alex received the overpayment notice, the council’s customer services director presented the quarterly performance report. Digital channel adoption was up. Average response time was down to 1.4 working days. Cost per interaction had fallen 28%. Case resolution rates were running at 94%. The dashboard was green across every metric. Nobody in that meeting had Alex’s name in front of them.

The metrics the council was reporting were accurate. An average response time of 1.4 days is faster — measurably, demonstrably faster — than what came before. A 28% reduction in cost per interaction is a real achievement. A 94% case resolution rate sounds like a service that is working well.

What the dashboard doesn’t count

What none of those numbers captured: whether the question had been answered. The 94% resolution rate counted cases that had received a response and been closed. It did not count whether the response had resolved the issue. A case could be opened, acknowledged, responded to with a standard letter, and closed as resolved — three times — without the person ever getting what they needed. That is not a failure in the data. It is a failure in what the data was designed to count.

The calls that ended without resolution. The cases where the automated response was technically accurate and substantively irrelevant to the specific question asked. The people who received four replies and gave up on the fifth attempt. None of that appears in the performance report. Not because it was hidden, but because the system was never designed to see it.

The people running the service have no signal that anything is wrong. The people funding it have no reason to ask. The people using it — the ones for whom it is not working — have no channel to make the failure visible, except to keep submitting queries that will each receive a response and remain unresolved.

The politics of choosing what to count

This is where Goodhart’s Law applies — the principle that the economist Marilyn Strathern summarised as: when a measure becomes a target, it ceases to be a good measure. Response time was a reasonable proxy for service quality until it became a target. Once it was a target, the system optimised for it. Cases were responded to quickly. Whether the responses were useful was a different question — and not one the target required anyone to ask.

KPI structures for technology services are not purely technical documents. They are political ones. The metrics chosen reflect what the organisation is prepared to be accountable for, and organisations, rationally, choose metrics they can improve. Response time can be reduced through better routing and automation. Case closure rates can be improved by defining “resolution” in ways that are achievable. Cost per interaction falls as volume grows. All of these things can be improved without making the service more useful to the person at the end of it.

The human outcome — did the person get what they needed? — is harder to measure because it requires asking them. That takes time, raises uncomfortable questions, and produces data that is more difficult to present as success. So it is left out. Not maliciously. Structurally. The governance body that approves the metrics does not require it. The reporting cycle that reviews them does not miss it. And the dashboard that summarises them has no row for it.

There is also a governance dimension to this. The people who approve the KPI structure are presented with a set of metrics that look complete because they cover all the dimensions the previous governance model required: speed, volume, cost, compliance. Nobody in that room has ever been asked to approve a metric for whether the service actually resolved the person’s problem. Nobody has been told that such a metric is missing. The governance process validates what it is shown. What it is not shown does not fail — it simply does not exist in the conversation.

This is how a service comes to be celebrated and broken at the same time. The celebration is for what is measured. The brokenness is in what is not. The two things co-exist perfectly until someone submits a third query and receives a fourth standard letter.

The cost that doesn’t appear in the report

The pattern repeats across services. A case that the digital channel cannot resolve eventually gets escalated to a phone call — the kind of contact the digital system was built to reduce. The query is answered. The discrepancy is corrected. The outcome the portal was supposed to deliver arrives via the channel the portal was supposed to replace, weeks later than it should have.

The quarterly performance report will show digital channel adoption at its highest level. What it will not show: the volume of digital cases subsequently resolved by phone; the number of people who returned to a human channel because the digital one had not worked; or the real cost of a failed digital interaction, which includes not just the cost of the digital transaction but the cost of the phone call that followed it.

The service looks more efficient than it is. The measurement system has been designed in a way that makes that possible — by measuring the transactions that occurred rather than the outcomes they were supposed to produce. When a governance body reviews a performance report and sees green across every metric, the natural response is to keep doing what is working. The investment flows toward the metrics that are already performing well. The things that are failing silently receive nothing, because they are not visible as failures.

That is not a data quality problem. It is a design choice. The absence of an outcome metric is itself a decision, with consequences that accumulate quietly in every case the service was not built to see. And because the consequences are not measured, they do not generate pressure to change the measurement. The organisation that cannot see its own failures cannot feel the cost of them, and cannot direct resources toward addressing them. The green dashboard is not just a reporting artefact. It is a protection against accountability — not deliberate, but effective.

My Opinion

I have watched organisations celebrate dashboards that tell them almost nothing about whether their services are working. The instinct to measure is right. The flaw is in what gets chosen. SLAs, KPIs, OKRs all share the same structural problem: they measure what the organisation can control. An automated acknowledgement meets the response SLA. The service still failed. I’ve witnessed a large outsourcing organisation hit their KPIs and SLAs every single month for response times…by putting an automated email response thanking the user for their query.

The organisations closing this gap are measuring outcomes, not transactions — asking users directly whether their problem was resolved. Some are going further, tracking social impact metrics: what actually changed for the person, beyond whether the case was closed. A number is a starting point, not a conclusion. Before accepting one, ask what human reality it represents. If you cannot answer that, the number is not yet telling you what you think it is.

The questions the system was never required to answer

The question a measurement structure should start with is whether the technology is actually helping the specific person using it — not whether it is processing their request efficiently. Those are different things. A system that processes twelve thousand requests per quarter and resolves sixty percent of them is not necessarily more helpful than one that processes eight thousand and resolves ninety. But a performance report that counts volume and response time will describe the first system as the better one.

The second question is whether the technology adds something real to the person’s capability, or only reduces costs for the organisation. A reduction in cost per interaction is not the same as an improvement in the quality of the interaction. Faster and cheaper are not synonyms for better. A measurement system that tracks only cost and speed cannot answer whether the technology is doing what it was built to do. It can only confirm that the technology is doing what it was built to do cheaply and quickly — which is a different question entirely, and a less important one.

The third question — and the one most completely absent from most KPI structures — is whether the system responds to the specific circumstances of the person in front of it. The 94% resolution rate is a population-level figure. It tells you nothing about the person in front of it. It tells you nothing about the six percent for whom the service did not work, or what those cases had in common, or whether they were disproportionately the people who needed the service most. A system that performs well on average while failing consistently at the edges is not performing well. It is performing adequately for people whose situations fit the expected pattern, and invisibly for everyone else. Those are not the same thing, and no dashboard that measures only the average will ever surface the difference.

The question behind the numbers

1. Think of a technology or service you use regularly — at work, or as a citizen or customer. What do you think is being measured about your experience of it? Does that match what you would want to be measured?

2. If you are responsible for reporting on technology performance in your organisation, when did you last ask whether the metrics you report capture the experience of the person at the end of the system — not just the performance of the system itself?

3. If you could add one metric to your organisation’s technology reporting — something currently unmeasured that would tell a more complete story — what would it be, and why is it not already there?

4. A system that hits every KPI while failing the people it serves: is that a success or a failure? And if the answer is complicated, what does that tell us about how we have defined success?

A service can hit every target on its dashboard and still fail the person who needed it. The performance report will describe it as operating at record efficiency. Both things are true. They are not the same truth, and the distance between them is where the accountability isn’t.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The agenda ran to eleven items. Security was seventh. By the time the board reached it, forty minutes had been spent on a supplier contract and a facilities question about the car park. The CISO had twenty minutes. They used eighteen of them.

The security review recommended twelve measures. The board approved three, deferred four, and excluded five as beyond the year’s plan. The meeting ended. Someone went to get coffee. Nine months later, one of the deferred items was the entry point for a ransomware attack that shut the business down for eleven days.

Nobody in that room had been negligent. Nobody had ignored the risk. The decision was made by competent people using the information they had, inside a process designed to balance competing priorities.

The CISO’s brief was forty-two slides. The executive summary was four pages. The risk register mapped each of the twelve recommendations to a likelihood score and a financial impact range. It was thorough. It was also built to satisfy a compliance requirement, not to help a non-technical board member understand which of those twelve things actually mattered most, and why, and what the difference would look like if they got it wrong.

Item seven on the deferred list was a patch management protocol for legacy systems. The brief described it as “medium risk, estimated remediation cost £18,000, recommended timeline Q2.” It sat between a network segmentation proposal and an employee phishing simulation programme. The board chair asked whether it could wait until the budget review. The CISO said it could be managed. The item was deferred.

That sentence — “it can be managed” — actually means: I believe the risk is tolerable in the short term, given everything else on the table. It does not mean: this is safe. The distinction between those two things disappears in the room. It stays disappeared until something goes wrong.

Organisations have been told for twenty years that security is a board-level responsibility. The intention was good. Bring it out of the IT department. Get it in front of decision-makers. Give the CISO a seat at the table. According to DSIT surveys, only 27 to 31 per cent of UK businesses have a board member with specific responsibility for cybersecurity — and this after a decade of sustained regulatory pressure to change it.

What happened instead is that security became a governance exercise. A risk register. A compliance checklist. A set of items to be approved, deferred, or removed, using the same mental framework that the board uses to evaluate a marketing spend or a property lease. The question is always: what does this cost, what does it protect, and can it wait?

The catastrophic failure of that framework is that cybersecurity risk does not behave like the risks that framework was built for. A lease that is deferred costs rent. A security measure that is deferred creates a window. The attacker does not wait for the budget cycle.

The deferred patch was not a theoretical vulnerability. It was a known gap, documented, scored, and filed. It sat in a folder. The attacker found a different route to it than the folder.

The decision that enabled the attack was made in a Monday morning meeting that ended with someone going to get coffee.

Eleven days offline. Not a dramatic number until you translate it. Supply chain halted. Customer orders unmet. Staff unable to work. The IR firm billing £2,400 a day. The ransom demand. The legal fees. The forensic audit. The regulatory notification. The reputational cost that does not appear in any spreadsheet.

The CISO did not lose their job. The board did not face personal liability. The organisation recovered, at significant cost, and updated the risk register. The patch management protocol was implemented in full. It had cost £18,000 on the original recommendation. The recovery cost many times that.

The people in the meeting were not careless. They were doing exactly what the system asked them to do. The system asked them to balance risks against budgets in a room built for a different kind of risk. That is not a character failure. That is a design failure.

What the brief was actually built for

A forty-two slide deck translated into a four-page summary with a risk register does not help a non-technical director make a well-informed decision about which deferred item will cost them thirty-eight times the remediation price. It helps the CISO demonstrate that they have fulfilled their duty of care. Those are not the same thing. The board member who approved the deferral did not understand the difference between “medium risk” and “a known gap in a legacy system with active exploit code circulating in the wild.” The brief did not help them understand it.

The governance structure was built to add something to security oversight — documentation, audit trail, the appearance of rigour. What it does not reliably add is security, because security requires that decisions reflect the actual threat environment rather than the budgetary one. A compliance framework and a threat actor are operating on entirely different timelines. The register records what was known. The attacker acts on what was left open.

The risk assessment treated every organisation of similar size and sector as roughly equivalent. It did not adapt to this company’s specific exposure: which of their legacy systems was most likely to be targeted, what the attacker’s likely entry path would be, what the actual blast radius of a successful attack on that specific infrastructure would look like. A one-size-fits-all risk score applied to a specific organisation on a specific day in a specific sector is not risk management. It is risk documentation.

The problem is not that boards are incompetent or that CISOs are failing. The problem is that the governance architecture built to manage cybersecurity risk was designed for a world where risk is relatively static and consequences are proportionate. Cybersecurity risk is neither. The architecture needs to change, not the people trying to work inside it.

My opinion

The more we rely on digital services, the more we place ourselves in the hands of organisations whose governance structures were not built to protect us. CISOs are operating in conditions no single role should carry. My position is that security risk belongs at the top of every board agenda, treated as a living state rather than an annual exercise — because the threat changes every day. Every new service deployed without security built in from day one creates exposure that did not exist the day before. Secure by design is not a technical principle — it is a condition of doing business.

The questions the risk register doesn’t raise

• Have you ever been in a meeting where a security recommendation was deferred for cost or time reasons? Not because anyone thought it was unimportant — but because the table was full and the budget was finite and it seemed manageable for now?

• If you have sat in a room like that one: what would it have taken for the conversation to go differently? What would the CISO have needed to say, or show, or do, to change the outcome?

• Should the people who make security deferral decisions face personal accountability when those decisions lead to a breach? And if the answer is yes — what would that accountability look like in practice, and who would apply it?

• The gap between the risk register and the real threat environment is not a secret. The people who write the registers know it. The question is whether the governance structures we have built are capable of closing it — or whether they were designed for something else entirely.

The dynamics in that boardroom — the compressed agenda, the deferred risk, the gap between documented exposure and lived consequence — run through The Shadow System. The book takes the argument further than a single meeting.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Casey stood in the kitchen on a Sunday morning wanting to do something small and

ordinary. The lights were already on — the right level, the right temperature, the system having made the calculation sometime before waking. The heating was already at the temperature the house knew a Sunday morning required. The coffee machine had already started its cycle.

Not one of these things had been consciously chosen that morning.

The intent had been to make a deliberate choice about the day — to begin it with a moment of decision rather than discovery. To turn the lights up brighter than usual, perhaps, because it was overcast outside and the mood called for it. To make the coffee later than normal because there was no urgency. Small acts of self-determination in an ordinary morning.

The system had already decided. It had observed enough previous Sunday mornings to know what this one required. The preferences were not wrong — they were, in fact, accurate. They reflected genuine past behaviour, real choices made on real mornings over months. The system had learned well.

That was the problem.

The disappearance of friction

There is a version of this story that reads as convenience. The house anticipates. The morning runs smoothly. That is what the technology was designed to do, and it is doing it.

What that version of the story does not account for is what friction was for.

The small acts of choosing — the light level, the temperature, the pace of a morning — are not merely mechanical. They are the continuous practice of preference: the low-stakes exercise of deciding who you are and what you want on any given day. They are how a person remains in contact with their own inclinations. When those acts are automated away, the automation does not just remove effort. It removes the opportunity to notice whether the choice would have been the same.

Casey’s house had learned a version of Casey built from past behaviour. Not from yesterday’s behaviour, from the cumulative pattern of months. A version that was accurate in aggregate and potentially wrong in any specific instance. The system had no mechanism to register that people change, that moods shift, that a Sunday in January and a Sunday in March might call for different things even if the previous fifty Sundays had called for the same ones. The actual person was no longer quite required.

The moment of choosing had disappeared so gradually that it was only noticeable in its absence. Not when the system got it wrong, it rarely got it wrong in any obvious way. But when the desire arose to do something differently, and the system had already done it, and the act of undoing it felt like more effort than simply accepting what was already there.

That is the trap. Not coercion, convenience so complete it becomes its own constraint.

From the kitchen to everywhere

The smart home is the most intimate version of this pattern, because the home is the space where people are most fully themselves. But it is only the beginning of the scale.

Step outward from the kitchen and the same architecture is running at every level. The streaming platform that has watched enough viewing history to know what will be selected before the selection is made. The news feed that has modelled enough reading behaviour to decide what is worth knowing today. Each of these systems is doing the same thing as the smart home, in a different register: replacing the act of choosing with the delivery of a prediction so accurate that choosing no longer feels necessary.

Step further out. The e-commerce platform that has learned spending patterns well enough to surface only what will be bought — not what might be wanted, not what might be surprising, but what the model says is coming next. The navigation app that routes not by the driver’s preference but by the algorithm’s assessment of the optimal path, such that after a year of use a driver may no longer know the roads they travel every day. The workplace tool that pre-populates responses, pre-schedules meetings, pre-drafts the email — reducing the human contribution to the act of approval.

At each step the friction removed is real. The convenience is not manufactured, it is delivered. And at each step something else is happening simultaneously: the system’s model of the person is deepening, the person’s practice of self-determination is narrowing, and the gap between who the system thinks they are and who they actually are is quietly widening.

The model is built from what was. The person is living in what is. Those two things are not the same, and the longer the system runs without the person having reason to notice the divergence, the larger the gap becomes.

Step to the outermost scale. The political information environment, the social media feed, the search result page, all of them running the same logic at civilisational scope. Not just anticipating individual preferences but shaping them, because a preference exercised without friction, without encounter with the unexpected, without the resistance of a world that does not already agree, is a preference that stops developing. The system does not just reflect who you are. Over time, it shapes who you remain.

The distinction nobody offered

The question of how much anticipation is enough is not rhetorical. It is a real dilemma.

Friction is not inherently good. The effort of choosing is not valuable in itself, it depends entirely on what is being chosen and why. The friction of adjusting the thermostat is not the same as the friction of deciding what to read or what to believe or what kind of person to become. Not all convenience is erosion. The task is to distinguish between the friction that was serving a purpose and the friction that was simply in the way, and to recognise that those two things look identical until the moment of removal.

Most people have not made that distinction deliberately, because it was never offered to them as a choice. The anticipatory systems arrived and removed friction and the question of which friction was worth keeping was never asked. The default was anticipation. The person who wanted to preserve the act of choosing had to actively resist the convenience of not choosing, and resistance is itself a form of friction that most people do not have time or inclination to apply.

The deeper consequence is not about convenience it is about the model. Every anticipatory system builds a representation of the person it serves. That representation is the basis for every subsequent prediction, every pre-emptive action, every choice made on the person’s behalf. The richer and more accurate the model, the more completely it can substitute for the person’s actual judgment. And the more completely it substitutes, the less opportunity there is for the person’s actual judgment to develop, correct, or surprise itself.

A person who has never had to choose their own route no longer knows the roads — a 2024 study confirmed this is not a metaphor: habitual GPS use produces a measurable, causal decline in spatial memory, even in people who previously navigated well. A person whose reading is pre-selected no longer encounters what they did not know they were missing. A person whose home has already decided never fully inhabits the decision to be at home. The capacity that is not exercised does not remain dormant. It diminishes.

That is the cost that does not appear in the product specification. That is the human loss that convenience optimises away.

Who is this for

The question the anticipatory system never asks, because it was not designed to, is who benefits from the anticipation.

The person using the system receives convenience that is real. But the system is not running in their service alone. Every preference recorded, every choice predicted, every behaviour anticipated is data. Data that trains the model, refines the prediction, deepens the platform’s understanding of what the person will do next. The more accurate the anticipation, the more valuable the person becomes as a data source and the more dependent they become on a system whose primary obligation is not to them.

The smart home manufacturer knows the daily rhythms of the household. The streaming platform knows attention patterns more precisely than the viewer knows themselves. None of this knowledge is held in the person’s interest by default. It is held in the platform’s interest, governed by the platform’s terms, subject to the platform’s commercial decisions about how that knowledge is used, shared, or sold.

The anticipatory trap is not just about agency. It is about the terms on which that agency has been surrendered and to whom. Casey’s house is learning Casey — but who owns what has been learned, and what happens to it when the subscription ends, the company is acquired, the terms are changed? The system was presented as a tool serving the household. The data flowing out of it tells a different story about who is serving whom.

The system that anticipates your needs perfectly has built a more accurate model of you than you have of yourself. That is impressive. It is also alarming, not because the anticipation is wrong, but because the entity holding that model is not you, does not answer to you, and did not ask your permission to build it in quite the way it has.

The friction that was removed was not just effort. Some of it was the resistance that kept you in contact with your own preferences. Some of it was the randomness that kept you encountering the unexpected. And some of it was the inconvenience that made you notice, every now and then, that a choice was being made at all.

What it was replaced with is not neutral. It serves someone. The question is whether you know who.

What you might find you’ve forgotten

Think about the systems in your daily life that anticipate your preferences — the home, the phone, the feed, the platform. When did you last make a deliberate choice in a domain where anticipation now operates? Not overriding a suggestion — making an original decision, from scratch, without a pre-populated option in front of you?

The model these systems hold of you was built from your past behaviour. How accurate is it to who you are now? Have you changed in ways the system has not registered? Have your preferences shifted, your interests expanded, your circumstances moved, and has the anticipation moved with you, or is it still running on data from a version of you that no longer quite exists?

Who holds the model? Not in terms of which company’s name is on the product, in terms of what happens to what has been learned about you. Is it portable? Is it deletable? Is it yours, in any meaningful sense? Or has the convenience of the service been partly funded by the permanent transfer of something you did not consciously offer?

And the question underneath all of these: if the anticipation were removed tomorrow — if the systems stopped predicting and simply waited for instruction — what would you find you had forgotten how to choose?

My Opinion

The technology itself is not the problem. What makes ambient systems genuinely difficult is their invisibility. A tool you can see, you can use deliberately. A system that operates below the level of conscious attention has already made the choice about when and how it operates. The line between the two is not technical — it is a design decision, made by someone other than you, before you arrived.

Most ambient technology was not designed to be controlling. It was designed to be easy. But easy and controlling share a direction of travel. The easier a system makes a decision, the less often you will make that decision yourself — and at some point along that gradient, the decision stops being yours. It stopped leaving room for it.

Not everyone needs this. That sounds obvious but it is rarely said plainly. There are households where ambient technology solves a real problem — accessibility, health monitoring, the household stretched across too many responsibilities to manage routine decisions without help. In those circumstances it is doing what it was built for. As a default for everyone, replacing the ordinary friction of a morning with a prediction of what the morning should be, it is a different proposition — one that takes something from the quality of ordinary life that is hard to name until it is gone.

The question is not whether to use these systems. It is whether you know what you are giving in exchange. Convenience offered at no visible cost is rarely free. What is given in return is usually the data that makes the system more accurate, and the habit of not deciding that makes it more necessary. Both transfers happen gradually, without announcement, and by the time they are noticeable, reversing them requires more effort than most people are willing to apply on a Tuesday morning.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan opens the app on a whim. It is one of those data-transparency tools that have become briefly fashionable — the kind that pulls together what advertisers, brokers, and platforms believe about you and shows you a summary. Morgan is curious, not anxious. It takes about thirty seconds to load.

What comes back is strange. The profile is accurate in the way a caricature is accurate — it captures something recognisable while getting most of the important things wrong. Morgan is categorised as a “high-anxiety urban professional,” likely to respond to time-pressure messaging and financially cautious. There are interest tags: travel, wellness, mid-range electronics. A risk band. An inferred household income. Morgan reads it twice, unsure whether to laugh or feel uneasy. Neither feels quite right.

When behaviour becomes identity

The discomfort is worth paying attention to. Not because the data is false — most of it is close enough to be recognisable — but because it is being treated as complete. The profile was assembled from clicks and purchases and search patterns, and somewhere along the way it stopped being a description of behaviour and became a verdict about a person.

That shift happens quietly. A system designed to predict preferences begins to define them. A tool built to segment audiences starts to assign identities. Nobody decided this was the purpose. It emerged from the logic of optimisation, from the pressure to know more in order to sell more, to lend more wisely, to assess risk at scale. The profile is a side effect of a process that was never really about the person at all.

The profile is accurate about behaviour and wrong about the person. The danger is not the bad data — it is that the system believes data is enough.

A trace treated as a portrait

Here is the underlying problem: data captures what someone did, in a particular moment, under particular circumstances. It does not capture why. It cannot hold the week someone was caring for a sick parent and spending on delivery food because there was no time to cook, or the period of job uncertainty that explains a cautious financial pattern, or a single impulsive holiday purchase that rewrote the travel interest score. The data is a trace. The system reads it as a portrait.

This matters because these profiles are not just sitting in a database somewhere. They are active. They shape what someone is offered, what rate they are quoted, sometimes what opportunities appear and which do not. A mortgage application, a job screening, an insurance premium — in each of these, the statistical shadow is treated as the person. And the person has almost no practical way to correct it, because most of the time they have no idea it exists.

The portability of these profiles makes it worse. An assumption made in one context follows into another. A risk score generated for one purpose bleeds into decisions it was never designed to inform. The data travels; the context does not. What began as a description of what someone did last Tuesday is now being used to decide what they deserve next year.

What the profile decides

The human cost here is not always dramatic. Sometimes it is a loan offered at a higher rate than someone with an identical financial history but a different data footprint would receive. Sometimes it is a job application that does not progress past the screening layer. Sometimes it is simply the low-grade unease of being looked at by systems that cannot see you, being assessed by processes you cannot access, being defined by a version of yourself that you did not author and cannot revise.

For those who are already disadvantaged — people in financial precarity, people with gaps in their histories, people whose lives do not fit the patterns the models were trained on — the distortion is not low-grade. It is consequential. The system is not malicious. It is simply indifferent to the difference between a data point and a human being. That indifference is its own kind of harm.

What the design is actually doing

In the original use case — recommendations, relevant offers, faster credit decisions — there is a genuine service on offer. But it holds only while the profile is accurate, and only while the person has some say in what it contains. Where the profile is wrong, or where it follows someone into contexts where it has no business being, the service becomes a constraint.

The efficiency gains for the institutions using these systems are real. What has been added for the individual is less clear: a permanent, portable verdict about who someone probably is. Whether that addition has value depends entirely on who is doing the measuring — and for whom the system was designed.

Where it fails most visibly is in adaptation. The profile is static in the ways that matter most. It captures behaviour but not situation. It updates when you click, but it does not reset when your life changes. It does not know the difference between who you were and who you are, and it has no mechanism for registering who you are trying to become. A system that cannot hold that kind of ambiguity is not adaptive. It is just persistent.

This is one of the questions at the centre of The Cognitive Crucible — not just what the system records, but what it misses by design.

My Opinion

We are living through a period of digital identity fragmentation. Pieces of us — clicks, purchases, search patterns — are being gathered by third parties and reassembled into a composite. The problem is not just the assembly. It is the premise: that this composite represents a person.

It does not. We are not one thing. We are many, depending on context — different at work and at home, under pressure and at ease, at a particular age, in a particular relationship. A system that flattens all of that into a single profile is not describing us. It is describing a version of us that has never existed.

Taking back agency over this will be hard. It may require something counter-intuitive: choosing, deliberately, to define and manage our own digital identities — profiles that represent genuine aspects of who we are in a specific context, rather than ceding that definition to systems that were never designed with us in mind.

The questions underneath the discomfort

There is a reasonable question underneath the discomfort the opening described. Not “how do I get off this list” — that is almost certainly impossible — but something more fundamental: who is this profile actually for?

The profile was not built for the person. It was built for the systems that process them. That distinction changes the nature of the conversation. The question is not whether the data is accurate. The question is whether data should be sufficient.

1. Have you ever encountered a version of yourself through a system — a recommendation, a decision, a risk band — that felt both recognisable and completely wrong? What did you do with that feeling?

2. If you could see your full data profile across every platform and institution that holds one, what do you think you would find — and would you want to know?

3. If you could change one thing about how these systems work — how they are built, what they are allowed to infer, how they can be challenged — what would it be?

4. The system knows your clicks. Should that be enough to know you? And if not — what else should be required before a system is allowed to make decisions about your life?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

The people report comes toward the end of the board pack. Somewhere after the financial performance slides and before the risk register. It contains an engagement score — this quarter seventy-one, up three points from the previous period — and a brief commentary from the people director explaining the movement. The board notes the improvement. Someone asks whether the score is above or below sector average. The answer is above. The item closes.

Six months later, a significant portion of the senior leadership team has left. The reasons, when they eventually surface, are not complicated: a sustained period of organisational change had been managed without involving the people most affected in the decisions being made, and those people had concluded, rationally, that the direction was not one they wanted to follow. None of this was invisible. It was legible, in specific terms, in the free-text responses that sat beneath the engagement score the board had accepted as reassurance.

The board had asked the right governance question — how engaged are our people — and received an accurate answer. The number was real. What it failed to convey was whether the organisation understood the conditions producing it, and whether those conditions were stable or deteriorating beneath a score that was still, technically, above average.

This is not an unusual story. It is a governance pattern.

What a people metric is designed to do

Board-level people reporting has evolved to serve a specific function: to give non-executive directors sufficient assurance that the human risk in the organisation is being managed. The engagement score performs that function efficiently. It is comparable across periods, benchmarkable against sector peers, and defensible in the event of a regulatory or reputational challenge. It answers the governance question the board is equipped to ask.

The problem is that the governance question and the strategic question are not the same question. Is engagement above or below threshold is a governance question. Do we understand what our workforce, our customers, or our communities are actually experiencing well enough to anticipate what is coming is a strategic one. The first produces a number. The second requires intelligence. And most board people reporting has been designed to produce the former while giving the impression it addresses the latter.

The distinction matters because boards make consequential decisions about people — through strategy, through resource allocation, through the acquisitions and restructures they approve — and the quality of those decisions depends on the quality of the understanding they rest on. A board that believes a score of seventy-one means everything is broadly in order is making different decisions than a board that understands why the score is seventy-one, which parts of the organisation are driving it, and what would need to change for it to move in either direction.

The score alone cannot support the second set of decisions. It was not designed to.

What boards are not seeing

The gap between what boards receive and what would constitute genuine people intelligence is wide in most organisations, and wider still in specific situations: post-acquisition integration, significant organisational change, sustained periods of uncertainty, or industries where the workforce is distributed, contracted, or otherwise at a distance from the centre.

In each of these situations, the aggregate score becomes less representative precisely when it most needs to be reliable. The integration that is going well for the acquiring team and badly for the acquired workforce produces an average. The restructure that has settled the leadership layer while creating deep anxiety in the operational layer produces an average. The number consolidates the variation into a single point and presents it to the board as a picture of the whole.

What the board is not seeing, in most cases, is the distribution beneath the average — where the score is concentrated, where it is fragile, which parts of the organisation are carrying the weight of others, and what the people in the lower quartile are specifically saying about their experience. Nor is it seeing the qualitative register: the specific concerns, the named conditions, the articulated reasons that would allow a board to distinguish between an engagement dip that is transient and one that is structural.

Boards that have tried to close this gap by asking for more granular data — scores by function, by tenure, by level — get closer to the distribution problem but do not solve the intelligence problem. Granular numbers are still numbers. They show you where the variation is. They do not show you why.

The question no one asks in the boardroom

There is a version of this problem that is specific to the board’s role, and it sits just beneath the surface of every board pack.

Boards are responsible for oversight, not management. The distinction is important. A board that receives an engagement score and accepts it as adequate people intelligence is not failing to manage — that is not the board’s job. It is failing to ask the questions that oversight requires: what does this number actually tell us, what does it not tell us, and what would we need to know to be confident we are not surprised by something we should have seen coming.

In most organisations, no one in the boardroom asks those questions routinely. The people director presents the data they have been asked to present, in the format that has been agreed, against the benchmarks that have been established. The non-executives receive it in a pack that also contains financial performance, risk, and compliance items that require more active scrutiny. People data gets the time allocated to it, which is rarely enough to interrogate it deeply.

The result is that people risk is governed at a level of abstraction that would be considered inadequate in other domains. A board that accepted a single aggregate technology risk score without understanding what it represented, where the exposure lay, and what conditions were driving it would be considered negligent. The equivalent in people governance is standard practice.

The shift required is not primarily about tools or data. It is about what boards decide to demand. Engagement intelligence that is specific, distributed, and grounded in what people actually say — rather than how they respond to a scale — is achievable. Gobby, a UK platform built on qualitative-first surveys, shows what this looks like in practice: respondents answer in their own words, then validate each other’s responses rather than having them averaged by the system. What reaches the board isn’t a number — it’s what the workforce actually said. It requires a different kind of question at board level, and a people function equipped and willing to answer it.

The score will always be easier to present than the understanding. The question is whether the board is prepared to settle for it.

My opinion

I left an organisation once because the leadership had changed five times in five years and no one seemed to know what direction they were heading in. When I left, no one asked why. The exit process was procedural. I was not a data point anyone was going to interrogate. In another business I left because of a executive level change which resulted in an organisational restructure, after the very first meeting with the new executive I knew what was coming and it wasn’t going to be pretty. The thing which I loved about that business was the one thing that was going to be destroyed - Culture. So I resigned, said my piece and everything I said came true.

The people who collect this kind of information are usually not hiding anything deliberately. They are responding to incentives. They have learned, over time, what the board finds useful and what it finds uncomfortable. They present the former. They convert the latter into a score.

That is not intelligence. It is data dressed as reassurance. And if a board cannot hear the difficult alongside the comfortable — if the culture in the boardroom makes hard truths unwelcome — then the problem is not the measurement. It is a leadership failure at the top of the organisation.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Sam has been reading the same paragraph for the third time. The words are there. The comprehension isn’t. Something, unclear what, has been pulling at the edge of attention for the last twenty minutes without ever resolving.

The message arrives that Sam isn’t going to read yet. The alert that Sam already half-knows exists. Some unfinished conversation sitting just below the surface of thought, unanswered, unresolved, quietly taking up space in the mind that was supposedly committed to this page.

Sam isn’t distracted. Sam is present. Physically here, eyes on the text, doing the right thing. The presence is real. The attention is somewhere else entirely.

The attention you were certain you had

In 1999, two psychologists at Harvard asked participants to watch a short video and count how many times a basketball was passed between players wearing white. The task was specific and demanding.

Halfway through the video, a person in a full gorilla suit walked into the frame. Faced the camera. Beat their chest. Walked out. Fifty per cent of participants never saw it. Not because it was subtle. Because their attention was committed elsewhere, and the gorilla wasn’t part of what they’d been asked to track.

We believe we’re present because we’re physically in a situation. The eyes are open. The body is there. But presence and attention are not the same thing, and the gap between them is where a great deal of modern life is quietly disappearing.

Inattentional blindness — the brain’s tendency to miss even obvious stimuli when focus is elsewhere — isn’t a rare laboratory phenomenon. It’s the operating condition of most people’s daily lives.

The brain doesn’t see everything in front of it. It samples, filters, and constructs a picture based on what it has been primed to look for. When that priming mechanism is overwhelmed — when there are too many things competing for focus — the filter becomes less a tool of selection and more a tool of exclusion.

What has changed since the gorilla experiment isn’t the cognitive architecture. It’s the scale and sophistication of the systems competing to own the priming. Every notification, every feed, every platform is fighting to be the thing your brain gets told to track. When they win, everything else becomes the gorilla. Invisible. Present. Missed.

We don’t lose the ability to pay attention. We lose the choice about where it goes.

The things that disappear without announcement

The gorilla in modern life isn’t a person in a suit. It’s the conversation not had. The warning not noticed. The colleague who needed support and didn’t get it because you were technically in the room while mentally somewhere else entirely.

It’s the child who asked a question and got a distracted non-answer, then stopped asking. The friend who was struggling and mentioned it once, sideways, and you were there but not there when it came up. The decision made quickly because the depth of focus required to make it well wasn’t available that day.

The cost isn’t always visible. It rarely arrives labelled. It accumulates in the quality of what you produce, the depth of the relationships you maintain, and the gradual narrowing of what you actually notice about the people and moments that are directly in front of you.

We have been taught to understand this as a personal failing. A concentration problem. A discipline issue. The technology industry is clear, in its habits if not its language, that this framing suits it perfectly. Because if the problem is you, you don’t look at the design.

What the design is actually doing

A system that competes for the priming of your attention, that inserts itself into the filtering mechanism the brain relies on to distinguish signal from noise, is not supporting you. It is working against the very cognitive function it claims to support.

The argument that social platforms augment connection holds only if the connections they produce are of comparable quality to those they replace or displace. The gorilla experiment suggests a direct relationship: the more demanding the competing task, the more complete the blindness. Connection that comes at the cost of presence isn’t connection. It’s substitution.

The deepest failure is this: any system that makes you selectively blind to your immediate environment cannot claim to respond to your actual circumstances. The person sitting with their child. The person driving. The person in the conversation that matters. A system that doesn’t know — or doesn’t care — which of those moments it’s interrupting has no basis for calling itself designed for you.

This is one of the questions at the centre of my book The Cognitive Crucible — not just what technology does, but what it costs us without announcing the price.

My opinion

Attention is the most finite resource we have. I don’t think the platforms taking it are confused about what they’re doing — the design is not accidental, and the framing of distraction as a personal discipline problem is not a misunderstanding either. What concerns me most is the picture of society these platforms have constructed: not a reflection of how people actually live, but of what drives engagement. The two are not the same thing, and recovering the ability to tell them apart — to notice what is actually in front of you rather than what you’ve been primed to track — is, in the most direct terms I can offer, how you get your cognitive agency back.

What this asks of you

The point is not to understand inattentional blindness. The point is to know where it is operating in your own life.

1. What have you noticed yourself missing lately — at work, at home, in conversation — that you’d have caught if you weren’t distracted?

2. Is there someone in your life who has been trying to tell you something you haven’t fully heard? What would it take to change that?

3. What would you need to change about your environment to make genuine presence your default rather than your exception?

4. If someone was watching how you spend your attention across a typical week, what would they conclude about what matters most to you?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

There is a particular kind of silence that follows a consultation. Not the silence of consideration, where something heard is being weighed. The silence of completion, where a process has been concluded and the findings have been filed. The residents submitted their responses. The staff completed the survey. The service users attended the focus group. And then, at some point later — sometimes months, sometimes longer — a decision was announced that bore little visible relationship to what anyone said.

Most people who have been consulted by a large organisation have experienced this. Many have stopped expecting otherwise. That learned scepticism is not apathy. It is a rational response to a pattern that repeats across sectors with enough consistency to suggest it is not accidental.

The pattern has acquired a name — consultation fatigue — but the name understates what is actually happening. Fatigue implies a willingness that has been worn down. What most organisations have produced is something more deliberate: a process designed to look like listening without requiring the organisation to change course as a result of what it hears.

The mechanics of being heard without being heard

The structure of most consultations gives the game away.

Questions are written by the organisation, which means the answer categories reflect the organisation’s existing understanding of the problem. Respondents are asked to rate, rank, or select from options that were designed before anyone outside the room was consulted. Free-text fields exist, but are rarely analysed systematically — they are too difficult to process at scale and too unpredictable in what they might surface. The consultation closes, the quantitative results are tabulated, and the report that emerges reflects the organisation’s original position with a margin of adjustment around the edges.

This is not cynicism dressed as analysis. It is the logical consequence of designing a consultation process around the outputs you need to produce rather than the understanding you need to reach. In local government, that output is a mandate — a defensible record that affected parties were involved before a decision was made. In the NHS, it is a compliance requirement — evidence of patient and public involvement in service design. In the private sector, it is an engagement score — something to report to the board or include in a regulatory submission. Each has a legitimate purpose. None of them requires the organisation to be genuinely changed by what it hears.

The people being consulted know this. The communities most affected by decisions are typically the ones with the least trust in the processes designed to involve them — because they have the most experience of those processes. When a community has watched a planning consultation produce a predetermined outcome, or seen a workforce survey followed by a management communications campaign rather than structural change, the rational response is to disengage. Not because the issues don’t matter to them. Because they have learned that engaging produces no different outcome than not engaging.

What the same pattern looks like across sectors

In local government, the planning consultation is the clearest case. Residents are invited to comment on a proposal that is already substantially formed. The statutory requirement is to consult; there is no equivalent requirement to be materially influenced by what is said. Communities that oppose a development learn to flood the process with responses because volume is the only variable that demonstrably affects outcomes — not the quality of the argument, not the specificity of the concern, not the evidence that the proposal conflicts with the community’s stated needs.

In healthcare, service redesign consultations run on similar logic. A trust deciding to reconfigure services will conduct patient engagement as part of the process. The engagement is genuine in the sense that it happens; it is limited in the sense that the financial and operational case for the reconfiguration is typically established before the engagement begins. What the consultation can change is the detail. What it rarely changes is the direction.

In the private sector, the employee consultation — particularly in a restructure or acquisition — operates under legal requirements in the UK that mandate a period of consultation before certain decisions are implemented. The requirement is to consult; it is not to reach agreement, and it is not to change course if the arguments against the proposal are sound. HR teams are well practised in running consultations that satisfy the legal threshold while protecting the operational decision already made.

In the charity sector, the gap is perhaps most uncomfortable to acknowledge. Organisations built on the principle of community voice frequently struggle to implement it in practice. Governance structures centre the board and the executive. Impact measurement centres the funder’s reporting requirements. The beneficiary’s account of their own experience sits at the end of a chain of representation — reported by a caseworker, summarised by a manager, condensed into an impact metric by a communications team — that has progressively removed the original voice by the time it reaches anyone with decision-making authority.

The thing that peer validation changes

The problem is not only that organisations fail to act on what they hear. It is that the processes they use mean they often do not hear clearly in the first place.

A consultation that asks closed questions gets closed answers. A free-text field that sits at the end of a fifteen-minute survey gets responses from the minority who feel strongly enough to write something additional. An engagement that channels responses through a manager or a facilitator introduces the interpretation of the intermediary before the data reaches analysis. By the time findings are presented to the people who will make the decision, the original voice has been translated several times.

The methodological shift that matters here is not just collecting open responses — it is validating them within the community before they reach the analyst. When people can see each other’s responses and indicate what resonates, two things happen that closed surveys cannot produce. First, the synthesis reflects the community’s own sense of what matters, rather than the researcher’s coding scheme. Second, people who contribute to a process and see their contribution acknowledged by peers are less likely to conclude that the exercise was performative. The experience of being heard is partly structural — it depends on whether the process was designed to hear or designed to record.

Gobby is one tool built on this logic — open responses rather than closed questions, peer-powered voting so participants can indicate what resonates, AI-assisted synthesis that reflects the community’s own sense of priority rather than the analyst’s coding scheme. It is still used by a minority of organisations. The majority continue to run processes that produce defensible records rather than genuine intelligence. The technology to do otherwise is not the constraint. The will to act on what might be heard is.

That is the harder problem, and no tool solves it. What better processes do is remove the organisation’s ability to claim it didn’t know. If the findings are specific, community-validated, and clearly stated, the decision to set them aside becomes a conscious choice rather than an administrative oversight. That is a different kind of accountability — and it is the reason some organisations prefer the consultation that changes nothing.

My opinion

I have been in enough of these rooms to know the difference between an organisation that doesn’t know how to listen and one that has decided not to. Most consultation theatre is the second kind. Tools like Gobby do something useful: they remove the claim of ignorance. If the findings are specific and community-validated, the decision to set them aside is no longer an oversight. It is a choice. And the organisations that consistently make it know exactly what they are doing.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan comes on shift at 11pm and the menu is gone. Not crashed — gone. The update that rolled out at 6pm moved it somewhere else, and the section of iOPS that handles electronic file transfer to the Crown Prosecution Service is no longer where it was yesterday. There is a person in the custody suite waiting to be processed. Morgan has a legal time limit. Morgan picks up the paper form.

This is Greater Manchester Police in July 2019. iOPS — the Integrated Operational Policing System — went live sixteen months behind schedule. The custody module, which was supposed to manage custody records and transfer files electronically to the CPS, was not working as specified. The announcement had said it was ready. The go-live had happened. Nobody said out loud that it wasn’t done.

It wasn’t done.

Deployed, not delivered

The workaround that custody sergeants had been using for three months no longer worked. The update had changed the interface without adequate retraining. There was no mechanism to refuse the tool — the option was to make it work, improvise a new workaround, or escalate a complaint that had no clear landing point. Officers improvised.

GMP’s own independent review named the consequences: technical shortcomings, inadequate training, and operational disruption that left vulnerable people without referrals to support services. Referrals for high-risk domestic abuse victims fell by 50%. Referrals to Greater Manchester Victims’ Services fell by 87%. Officers spent time running duplicate paper processes alongside the digital system that was supposed to replace them. The system did not help the workforce do the work — it created a second job beside it.

iOPS had gone live not because it was ready but because the organisation needed it to have gone live. Sixteen months of delay had a contractual cost, a political cost, and a considerable cost in credibility. Ready was redefined as deployed, and the workforce inherited the gap between the two.

Why no one says it isn’t ready

A software product that is finished cannot justify an ongoing roadmap. It cannot retain an engineering team. In a subscription market, completion is a liability — a done product is one that doesn’t need what comes in the next release cycle. Agile methodology and continuous deployment gave the industry a vocabulary for this: not unfinished, but iterating. Not incomplete, but evolving. The language made incompleteness sound like a sign of rigour.

Organisations accepted this because the alternative was more expensive. To say the system was not ready meant saying the procurement decision was wrong. GMP’s contract had already been signed, extended, and renegotiated. Admitting the custody module was unfit meant unpicking an obligation that was difficult to unpick. Deploying and adapting was easier.

There was no real mechanism to refuse. The tool was the tool. The workarounds became part of the job. The cost was absorbed shift by shift by custody sergeants who had no power to put it back where it came from.

The cost that didn’t appear in the contract

Birmingham City Council signed a contract for an Oracle ERP system projected at £19 million. By 2026 the total cost is forecast to reach £216.5 million. Two and a half years after go-live, the system was still not, in the auditors’ own words, safe and compliant. In September 2023 Birmingham declared bankruptcy — the largest local authority insolvency in UK history. The system was never ready either.

NHS IT outages disrupted more than 274,000 patient interactions across six major trusts in a single year. In each of those incidents, clinical staff reverted to paper. Not because paper was preferred, but because the digital system had failed and the paper process was the only thing keeping the work going. More than a quarter of UK central government digital systems are now classified as outdated, at an estimated cost to the public sector of £45 billion annually in foregone productivity.

Office workers endure an average of 3.6 technology interruptions a month — software updates, system failures, workarounds required. Nobody counted this cost when the contracts were agreed, because the cost landed on the worker. The vendor’s contract was satisfied. The product had been delivered. That it didn’t work as specified was the next problem, billable separately.

What it was designed for

The most basic test for any system deployed to workers is whether it actually helps them do the specific work it replaced. iOPS failed that test on the night the update moved the workflow officers had learned. The tool stopped being a tool and became an obstacle with extra steps. The work still needed doing. Staff found a way to do it. The system got the credit for a deployment that happened.

The next test is whether what the vendor calls an improvement actually improves anything for the person using it. iOPS received updates. Each notification used the word “improvement.” Custody sergeants had not requested the features those updates added. The gap between what the vendor was building and what the workforce needed is not a failure of communication. It is the expected result of a development roadmap driven by the vendor’s commercial cycle, not by the legal obligations of a sergeant at 11pm.

The final test is whether the system responds to the specific person in front of it, or expects the person to fit. iOPS updated on the vendor’s schedule. Its release windows did not account for shift patterns, operational hours, or the consequences of breaking a workaround mid-cycle. The specific circumstances of night-shift custody work — legal time limits, vulnerable people waiting — were not part of the specification for when an update went live. The system was not designed around the people using it. They were expected to fit.

My Opinion

I have been in enough procurement processes to know that the people who will actually use the system are rarely in the room when it is chosen. Requirements are gathered, but they are normally vetted by people who do not work with the system day to day — so what gets built reflects what the organisation said it needed, not what the work actually demands. Sometimes a system is deployed not because it is ready but to satisfy a contract: the way a UK train leaves a platform on time even when it isn’t. What is missing is oversight with teeth — people who have enough operational experience to know when something is wrong, and enough authority to stop it going live until it is right.

I’ve seen this done properly where in a hospital the IT Development team deployed a capability that clinical staff had been asking for. Then they got smart - they instigated a digital suggestions box for new features/services/products and the hospital staff embraced this as they got a voice on what was needed and important - not what the organisation thought it needed. The even smarter change was the IT Dev team wrapped this all with an agreed SLA for delivering selected capabilities and they stuck to it. What the users asked for they got when it was promised. The downside if it can be called one was the sheer volume of requests once staff saw it working.

Questions the contract never asked

The following questions are not academic. They are for anyone currently running a process that exists only because a system the organisation bought was never quite finished. The vendor moved on. The contract was satisfied. The workaround remained.

1. What workaround are you currently maintaining that exists only because a system your organisation bought was never quite finished?

2. If you had had to sign off on the cost of that workaround — in time, in error rate, in staff goodwill — when the contract was agreed, would the vendor still have won it?

3. When was the last time someone in your organisation said “this isn’t ready” and it made a difference?

4. What would it take to put “not ready yet” back into the vocabulary of public sector procurement?

The cost of perpetual beta is not paid by the organisations that sign the contracts or the vendors that provide them. It is paid by the person on the shift who picks up the paper form when the menu has moved again. That cost is real. It has just never appeared in a budget line.

Leave a comment

Sources & References

• HMICFRS: Greater Manchester Police Integrated Operational Policing System Inspection, 2020. https://hmicfrs.justiceinspectorates.gov.uk/publications/greater-manchester-police-integrated-operational-policing-system-inspection/

• Computer Weekly: “Police IT system failure creates significant backlogs”, 2020. https://www.computerweekly.com/news/252479783/Police-IT-system-failure-creates-significant-backlogs

• Data Centre Dynamics: “Total cost of Birmingham City’s Oracle system failure to reach £216.5m by 2026”, 2024. https://www.datacenterdynamics.com/en/news/total-cost-of-birmingham-citys-oracle-system-failure-to-reach-2165m-by-2026-report/

• The Register: “Birmingham City’s Oracle ERP system still not ‘legally safe’”, April 2024. https://www.theregister.com/2024/04/17/birmingham_city_councils_oracle_erp/

• The Register: “Birmingham City Council goes under after Oracle disaster”, September 2023. https://www.theregister.com/2023/09/05/birmingham_city_council_oracle/

• Security Brief UK: “NHS IT outages disrupt 274,620 patient interactions”. https://securitybrief.co.uk/story/nhs-it-outages-disrupt-274-620-patient-interactions

• Tech Monitor / DSIT: “Legacy technology costs UK public sector £45bn annually”. https://www.techmonitor.ai/government-computing/legacy-technology-costs-uk-public-sector-45bn-annually/

• Ivanti: 2025 Digital Employee Experience (DEX) Report, 2025. https://www.ivanti.com/company/press-releases/2025/tech-disruptions-cost-companies-millions-of-dollars-in-lost-productivity-annually-according-to-research-from-ivanti

  ---

  You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

An experienced endoscopist spends years learning to see. The skill is not in holding the camera. It is in the eye: the trained attention that catches a flat, easily missed growth at the edge of the screen, the kind that turns into bowel cancer if it is not found and cut out. That attention is built by doing the looking, thousands of times, with no one and nothing pointing the way.

Between 2021 and 2022, four endoscopy centres in Poland brought in an AI tool that draws a box around suspect growths as the camera moves. It works. Switch it on and detection goes up. But researchers also watched what happened to the same doctors when the tool was switched off again. In the months after routine AI use came in, their unaided detection rate fell from 28.4 per cent to 22.4 per cent — a fifth of their skill gone, on the procedures where they were once again looking alone. The Lancet Gastroenterology & Hepatology published the finding in August 2025. The doctors had each performed more than two thousand colonoscopies. They were not novices losing a skill they had half-learned. They were experts, and the looking had quietly stopped being theirs.

No one decided that those doctors should get worse at finding cancer. Each centre adopted the tool for an honest reason: on the day, with the box on the screen, more growths are caught. The erosion was the by-product. It arrived without a meeting and without a vote, and the act that had kept the eye sharp — the looking itself — was the very thing the tool took over.

This piece grew out of a conversation. A few weeks ago Jason reviewed Nick Bostrom’s Deep Utopia for his Book Club.

Jason B Perkins

Deep Utopia: The Automation Paradox

What if the future arrives exactly as the technologists promised? Not a collapse. Not a machine uprising. Not even a slow, disappointing muddle. What if artificial intelligence keeps improving, systems keep coordinating, science keeps accelerating, and the world becomes astonishingly good at solving problems? Disease recedes. Scarcity shrinks. Logistics…

Read more

3 months ago · 3 likes · 3 comments · Jason B Perkins

I left a comment that disagreed with the end of his argument rather than the start of it. We have been chatting it over since. What follows is both sides, written honestly and left unresolved on purpose.

We agree on Bostrom’s distinction. Some of what we do has instrumental value: we do it to reach a result. Some has intrinsic value: the doing is the point. As machines deliver the result faster, the instrumental reasons for human effort weaken, and what survives is the effort we would still choose when we no longer have to. The colonoscopy study makes me want to name a third category the distinction misses. Call it formative difficulty: effort whose result we want, and whose doing builds or sustains the capability to get that result at all. Jason reads Deep Utopia as a reason to protect that kind of difficulty on purpose. I think he is right, and I think the harder question is who gets to decide.

That is the architecture answer, and it holds. But the colonoscopy centres were not careless. They adopted the tool for a real, same-day gain in detection, and the erosion still happened. So the question I could not let go of in Jason’s comments is who decides which difficulty is worth keeping, and on what grounds. The decision is already being made, almost always by default. No one in those rooms asked whether the unaided looking was instrumental or formative, because that is not a question a business case is built to ask. The answer arrived as a tool, and a capability that took two thousand procedures to build began to fade in months.

Bostrom says the scarce goods of an abundant world will be meaning, agency, and the texture of things done by hand. That sounds like liberation. Look again and it can describe a market, because someone always sets the price of a scarce good, and someone is always priced out. We have watched smaller versions of this. The handmade thing does not vanish when the machine arrives; it moves upmarket and becomes how the wealthy signal rather than how ordinary people live. There is no law that protects formative effort from the same gravity. The risk is not that the hard parts disappear. It is that they are kept by whoever can still afford to do things the slow way, and lost by everyone measured on speed.

If we are going to defend any difficulty, the test has to be sharper than nostalgia, because most appeals to the value of struggle are sentimentality about how things used to be done. Plenty of difficulty is worth removing. No one should defend long division by hand, or a three-week wait for a letter, or a disease we can now cure. The difficulty worth keeping is the narrow kind that builds or sustains something in the person doing it: skill, judgement, identity, or a relationship that would not otherwise exist. The endoscopists’ looking sustained an eye for cancer. The tool kept the detection and let the eye go, on the quiet assumption that the eye was a by-product rather than the point.

Jason reaches the same line from the other direction, and calls it virtuous and vicious friction. Vicious friction only diminishes us: the pointless meeting, the layer of bureaucracy, the months spent waiting for access. Virtuous friction defines us: the apprenticeship that builds judgement, the demanding conversation that builds trust, the hard way that leads to mastery. He started from the architecture and I started from the person, and we arrived at the same cut. Not effort against ease. Formative difficulty against incidental difficulty. The trouble is that the two are indistinguishable on a dashboard, where both show up only as time the machine could save, and the people best placed to tell them apart — the doctors who can feel their own eye dulling — are rarely the ones holding the budget or writing the rollout plan.

None of this is an argument against the tools. Those doctors should have the AI; on the day, it finds more cancers, and that is not a small thing to set aside. The argument is that someone has to be watching the difference between the difficulty that was keeping a skill alive and the difficulty that was only costing time, and at the moment almost no one is. The Polish researchers were careful: their study was observational, and they could not yet say what happens to doctors who train with the box from the start and never build the unaided eye at all. That is the more troubling question, and it is still open.

We will not lose the hard parts in a single decision anyone could be held to. We will lose them one defensible efficiency at a time, each removal sensible on its own, until a generation arrives that can check the machine’s work but could never have done it alone. If meaning, judgement, and skill really are the scarce goods of the world we are building, the choice in front of every organisation is not whether to keep some difficulty. It is who gets to decide which difficulty, and whether the person it would have formed is ever in the room when the decision is made.

Over to you. Before you automate the next hard thing, Jason offers five questions worth holding it against:

1. Agency — does it strengthen human control, or quietly remove it?

2. Connection — does it deepen trust and human bonds?

3. Judgement — does it sharpen our ability to decide, or outsource it?

4. Opportunity — does it widen what people can collectively become?

5. Safety — can the outcome be audited or reversed?

So we will put the same question to both our readerships and answer it in the thread: which difficulty in your own work is keeping a skill alive, and which is just slow?

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Somewhere in most organisations there is a folder, a shared drive, or an archived system containing several years of things people said. Exit interview notes. Customer complaints in their original form, before they were categorised and counted. Free-text responses from the bottom of annual surveys. Messages sent to a service team that were resolved, closed, and filed. Forum posts from an internal platform that was replaced two years ago and whose contents were migrated to storage rather than deleted.

None of it has been read systematically. Most of it never will be.

This is not a data problem. The data exists. It is a processing problem — or rather, it was. For most of the period in which this material was accumulating, there was no practical way to analyse it at scale without a significant investment in qualitative research that most organisations could not justify for data they had not planned to use. So it was treated as administrative residue: kept because deletion felt risky, ignored because analysis felt impossible.

That calculation no longer holds.

The signal that was never treated as data

The gap between what organisations collect and what they analyse has always been widest in the qualitative register. Numbers get processed. Text gets filed.

This is partly a legacy of the tools available. Survey platforms were built to aggregate ratings, not to synthesise narratives. CRM systems were built to track transactions, not to find patterns in how customers described their experience. HR systems were built to record headcount and performance scores, not to make sense of what people wrote in the comments field at the bottom of the appraisal form. The architecture of most organisational data infrastructure assumes that what matters can be counted.

The consequence is that organisations have spent years collecting a secondary record of their own reality — the version in which people spoke in their own words, described their own experience, and articulated what the numbers could not carry — and treating it as noise.

Exit interviews are the clearest example. Most organisations conduct them. Most collect the responses in some form. Very few analyse them across a period long enough to identify patterns — which managers people were leaving, which structural conditions were driving attrition, which aspects of the culture were named repeatedly by people who had decided to leave but no longer had any reason to be diplomatic. The data exists. The analysis that would make it useful never happens, because exit interview findings are processed as individual cases rather than as a cumulative record.

Customer complaints follow the same logic. The categorised version — number of complaints by type, by channel, by product — reaches a dashboard. The original text, in which customers described in specific terms what had gone wrong and what it had cost them, is archived after the case is closed. Organisations that have been collecting complaints for five years have five years of precise, unsolicited qualitative description of the gap between what they promise and what they deliver. Almost none of them have read it.

What changes when the analysis becomes possible

The shift that is underway is not primarily about new data. It is about the retroactive value of data that already exists, and the prospective design of data collection with analysis in mind from the start.

AI-assisted qualitative analysis has made it practical to process large volumes of unstructured text in ways that were previously either manual and expensive, or automated and crude. The earlier generation of text analytics tools — keyword frequency, sentiment scoring — reduced qualitative data to a thin quantitative proxy. What is becoming possible now is closer to genuine synthesis: identifying themes, finding patterns, distinguishing between types of concern that look similar in aggregate but are structurally different in cause.

The organisations best placed to use this capability are not necessarily the ones with the most sophisticated data infrastructure. They are the ones that recognise they have been sitting on material with analytical value and begin treating it as such. A charity with three years of service user feedback in a case management system. A professional services firm whose client debrief notes have been saved but never reviewed across engagements. A healthcare trust whose patient letters — the ones written by people who felt strongly enough to put something in writing — are archived in a folder that no analyst has opened.

The design question for new data collection is equally important. When an organisation understands that qualitative responses have analytical value, it changes how it structures the collection. Open-response questions get asked more deliberately. Free-text fields get positioned earlier in a survey, not appended as an afterthought. Platforms are chosen with synthesis in mind rather than just with response rates in mind. Tools such as Gobby.io, which combines open response with peer-powered voting — where participants indicate which responses reflect their own experience — produce data that is both richer and more efficiently analysable than free text alone, because the participants have done part of the synthesis work before the data reaches an analyst.

What the evidence was saying all along

The most significant implication of this shift is not operational. It is about what organisations know about themselves, and what they have chosen not to know.

An organisation that has been collecting qualitative signal for years and never analysed it has, in practice, been making decisions without a significant part of the available evidence. That is not a neutral position. It means that the version of reality reaching leadership has been shaped by what the data infrastructure was designed to draw out — typically the quantitative, the recent, and the aggregated — and has excluded everything the data infrastructure was not built to carry.

When that material becomes analysable, organisations sometimes discover that the patterns were visible all along. The attrition problem that presented as a compensation issue in the survey data was described, repeatedly and specifically, in exit interview notes as a management culture problem in a particular division. The customer satisfaction decline that appeared as a score movement in quarterly reporting had been articulated, in precise terms, in complaint correspondence two years earlier. The service redesign that failed to achieve adoption had generated, during its consultation phase, a clear account of the operational reality it failed to account for — an account that was collected, filed, and never read.

None of this means that quantitative data is wrong or that qualitative analysis is always right. It means that organisations have been working with a partial picture and treating it as complete. The data to fill that picture, in many cases, already exists.

The question is whether the organisation is prepared to read it.

My Opinion

I have been through more exit interviews than I can count, and almost all follow the same pattern: questions designed to measure the process, not to capture what the person leaving actually thought. I have asked to read exit interviews from people who left teams I was responsible for and been met, more than once, with a blank look or a direct refusal. A genuine caveat applies: this data is shaped by why someone left, which means it needs to be read in context rather than taken at face value. But there is a difference between data that requires careful reading and data that is never read at all. The true voice of people is the thing most organisations consistently arrange not to hear.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Morgan finds out on a Tuesday evening, scrolling through the news. Not from the company. Not from the bank. Not from anyone who thought they had a duty to say something. From a journalist who found out first.

The article is six paragraphs long. The breach happened seven months ago. Morgan’s name, date of birth, email address, and payment history have been sitting somewhere they were never supposed to be. The notification is still in a queue. The legal team is still reviewing.

By the time Morgan finishes reading, the account has been live in the wrong hands long enough to matter.

The system is working as designed

Companies are not legally required to disclose data breaches immediately. In the UK, under GDPR, the obligation to notify the Information Commissioner’s Office is seventy-two hours. Notification to individuals is required only when there is a likely high risk to their rights and freedoms — and determining that likelihood is left largely to the organisation doing the determining.

The gap between an event happening and a person learning about it is not a failure of the system. It is the system. Seven months is not unusual in documented cases. Twelve is not unheard of. The disclosure process is managed for legal exposure, reputation, and investor sentiment. The person whose data was taken is, structurally, the last consideration.

None of this requires a villain. The process is run by compliance teams doing their jobs inside a framework designed before the scale of digital exposure was fully understood. The problem is not malice. The problem is architecture.

Whose interests the framework serves

The disclosure framework was built around the assumption that a company’s primary obligation in a breach event is to contain it, assess it, and report it in good order. The individual whose data was taken is a downstream consideration — someone to be managed after the institutional response is complete.

This is an understandable way to design a legal framework. It is not an acceptable way to treat a person. The individual whose data was taken has no independent means of knowing what has happened. They cannot check. Cannot act. Cannot protect accounts, change passwords, alert a bank, or do any of the things that might have reduced the damage. The window in which those actions were possible opens and closes while the disclosure is being reviewed.

The data that was taken does not expire. An address does not change. A date of birth does not change. The value to a criminal does not diminish with time. The delay does not reduce the risk — it transfers it entirely to the person.

The company managed the crisis. Morgan just lived inside it.

What the delay actually costs

The visible cost — fraudulent transactions, identity theft, months of remediation — is measurable, if the person can prove causation in time. Most cannot. The link between a breach notification seven months late and the credit application rejected six weeks ago is not the kind of thing an individual can easily trace through a system designed to obscure its own connections.

The invisible cost is something else. Anyone affected comes to understand, in a way that cannot be undone, that there are institutions holding information about them that will manage a catastrophic failure without telling them until the legal minimum has been met. That the decision about when they deserved to know was made by people they have never met, in a process they had no part in, according to criteria they were never shown.

The relationship between a person and the organisations that hold their data is not a relationship between equals. It never was. But the disclosure framework makes the power difference formal, and applies it at the moment a person is most vulnerable.

The notification system, as it currently operates, serves the organisation managing the breach rather than the person affected by it. It assists the company to discharge its legal obligation at a time and in a manner that suits the company’s interests. The individual affected receives a letter when the process is complete, not when the information might have been useful.

A seven-month delay adds nothing to the individual’s position. The argument that thorough investigation improves the quality of notification is technically defensible. In practice, the person receiving the notification has no way to verify the investigation’s quality, no means to challenge the timeline, and no recourse if the delay increased their exposure.

Every person whose data is in a breach has a different threat profile — different accounts, different vulnerabilities, different ability to act quickly. A single delayed disclosure treats all of them identically, at the moment when individual difference matters most. The person who is digitally confident and financially resilient and the person who is neither receive exactly the same letter, at exactly the same time, containing exactly the same limited information.

My Opinion

The architecture of delayed disclosure is not the result of poor drafting. It reflects a consistent choice about whose interests the legal system was designed to protect. The notification framework was not designed and then found to serve institutions better than individuals. It was designed to serve institutions. The conversation about how to change it will not begin with regulators or compliance teams. It will begin when enough individuals decide that a letter arriving seven months late is not a notification. It is an alibi.

The questions no letter asks

If your data has been in a breach — and statistically, it has — the notification you received, if you received one at all, told you what the company was legally required to say. It did not ask what you think about the system that produced that notification. These are the questions the letter didn’t come with.

• Have you ever found out about a breach affecting you through a news story rather than a direct notification?

• If you ran a company and discovered a breach, what would stop you from telling your customers immediately?

• Should notification timelines be standardised and enforced — and if so, by whom?

• What would you want a company to say to you, and when, if your data had been compromised?

These questions sit at the centre of my book The Shadow System — specifically the sections on how companies manage reputational risk against actual risk in the aftermath of a breach.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.

Every year, organisations across every sector spend significant money finding out how engaged their people are. The answer comes back as a number. Sixty-two percent. Seventy-one. A red, an amber, a green. And then, in most cases, very little changes — because a number, however precisely measured, does not tell you what to do next.

This is not a failure of effort. It is a failure of method. The question being asked — how engaged are people, on a scale of one to ten — is structurally incapable of producing the intelligence needed to act. It tells you the temperature of the room. It does not tell you why the room is cold, which walls are letting in the draught, or whether the people in it have stopped expecting anyone to fix it.

The pattern holds across every sector. An NHS trust running annual staff surveys. A private equity-backed scale-up tracking employee sentiment through quarterly pulse checks. A charity trying to understand what its service users actually need from a redesigned programme. A number where there should be a conversation.

What the number misses

The case for quantitative engagement measurement is not wrong. Trends matter. Comparability matters. Knowing that satisfaction dropped twelve points between January and March, or that one team consistently scores lower than the rest of the business, is genuinely useful. The problem is not that organisations measure; it is that most treat the measurement as the understanding, rather than as a signal that understanding is needed.

Qualitative intelligence — what people actually say, in their own words, about their own experience — produces something different. It surfaces the specific, the unexpected, and the things no survey designer thought to ask about. It also surfaces disagreement within a group that aggregate scoring conceals. A team with an average engagement score of sixty-eight might contain people who are deeply committed and people who are quietly preparing to leave. The number averages them into false coherence.

The difficulty has always been scale. Reading and coding open-text responses from three hundred employees, or three thousand service users, is expensive and slow. The analysis introduces its own biases. And the time between collecting the data and acting on it is long enough that the context has already shifted. So organisations default to the form that is easiest to process — the number — and accept its limitations as the cost of manageability.

That trade-off no longer holds as firmly as it did. The combination of open-response collection, peer voting, and AI-assisted analysis — the approach taken by tools like Gobby, which works in this space — means that qualitative data can now be gathered and synthesised at a speed and scale that was not previously practical. People respond in their own words. Those responses are surfaced to peers, who vote on what resonates. The result is not just a fuller picture; it is a picture that has been validated by the community itself, not just interpreted by the analyst.

The pattern across sectors

In the public sector, the gap between what is measured and what matters is often widest. Patient experience surveys ask whether care was clean, timely, and respectful — categories defined by the regulator, not the patient. What they rarely capture is whether the patient felt heard, whether the treatment made sense in the context of their actual life, or whether the system’s definition of a good outcome matched their own. The number that comes back is real. The experience it describes is partial.

Local government consultation has the same problem at a different scale. A planning consultation that asks residents to rate a proposal on a five-point scale produces a mandate for or against. It does not produce understanding of why, which aspects are acceptable and which are not, or what a modified proposal might need to look like to bring the sceptics with it. The binary produces a decision. It does not produce the intelligence that would make the decision stick.

In the private sector, the equivalent is the employee engagement survey and its close relative, the customer satisfaction score. Both have become so ritualised that the measurement has partially replaced the thing it is supposed to measure. Teams know how to answer the survey. Customers know that the score goes nowhere. The process continues because it produces a defensible number, not because it produces understanding. When engagement drops, the standard response is a communications campaign — because the survey told you sentiment, not cause.

The charity and social enterprise sector is arguably most exposed. Organisations whose entire purpose is to serve and represent specific communities often have the least developed means of understanding whether they are doing so. Impact reporting is dominated by output metrics — number of people served, sessions delivered, referrals made. What is missing is the voice of the person at the end of the service, speaking in their own terms about whether it changed anything for them. Not a satisfaction score. A considered account of what happened.

Understanding you can act on

The value of more detailed engagement data is not primarily analytical. It is operational. When the reasons behind a trend are visible — not inferred, but stated by the people who are living them — the path to action becomes clearer and the likelihood of getting it wrong decreases.

A portfolio company workforce that scores low on engagement is a problem. A portfolio company workforce that is able to articulate, collectively and specifically, that the integration plan removed the three things that made the culture work is a solvable problem — or at least one where the board can make an informed decision about the trade-off. A service user group that can say, in their own words and with their peers’ endorsement, that the new referral pathway is harder to navigate than the one it replaced gives a programme team something to act on rather than something to commission further research about.

My Opinion

Every organisation I’ve worked with runs some version of the same survey — historical questions, coded to a scale, returned as a number. I’ve seen what happens when you replace that with something closer to in-the-moment: a city authority that put real-time word clouds — drawn from live citizen responses — on a screen in the corridor where the chief executive walked every morning. The conversation in that building changed. Not because the data was new, but because it was immediate, in people’s own words, and impossible to walk past. That is the difference between recording sentiment and understanding it.

The shift required is not primarily technological. The tools exist. What is required is a change in what organisations believe engagement measurement is for. If the purpose is compliance — to show that a survey was conducted and a score was recorded — then the number is sufficient. If the purpose is understanding — to know what is actually happening and why, well enough to do something about it — then the number is a starting point, not an answer.

The organisations that close that gap are not the ones that find a better survey platform. They are the ones that decide the distinction matters.

Leave a comment

You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.

Share

Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.