Eight times out of ten, the system got the location wrong.
West Midlands Police deployed a predictive crime mapping tool in 2021, under the Home Office’s Grip hotspot policing programme. The task was specific: identify the locations where knife crime and serious violence were most likely to concentrate, so that officers could be deployed there before anything happened. The system produced wrong predictions roughly eight times in ten.
The programme it came from had shown real promise. The pilot had run in Essex. The results were credible enough for the Home Office to fund a national rollout across 18 forces (later to 20). The board presentation was approved. The evidence was genuine. Amnesty International’s review of the programme, published in 2025, found no conclusive evidence that Grip had reduced crime across the forces that deployed it, and documented that the predictive model reinforced racial profiling in the communities it targeted most heavily.
The pilot had worked. That is the point.
Ten million accounts, against twenty-five
GOV.UK Verify ran from 2016 to early 2023. It was the government’s flagship digital identity scheme — the mechanism through which citizens could prove who they were in order to access public services online. It cost more than £200 million to build and run.
At its peak it held 10 million accounts, against a projected 25 million. Only 27 public services adopted it, against a target of 46. The Public Accounts Committee found it was “failing its users.” It closed without much ceremony.
The people Verify worked for had something in common. They had stable personal histories — consistent addresses, continuous records, documentation that hadn’t moved around. They could sustain attention across several screens and hold a reliable correspondence between what they remembered and what the database expected to find. They completed the process because the process had been built for people like them.
Around eleven million adults in the UK lack the essential digital skills needed for basic online tasks. A further 1.6 million remain offline entirely and the NAO’s 2019 report found only 38% of Universal Credit applicants could complete Verify against a 90% target. The Verify journey — identity providers, credit reference checks, document uploads, knowledge-based questions — presumed a baseline of digital fluency and settled history that millions of people simply do not have.
They were not in the pilot population.
Including them would have made the pilot harder.
Pilots are not designed to be hard.
What a pilot is designed to find
In each case, a pilot demonstrated that the technology could work. That demonstration was treated as evidence that it would work for everyone. The people who were not in the pilot were the people most likely to struggle with the technology at scale. They weren’t in the pilot because including them would have made the pilot harder, and the condition of the pilot’s success is also the reason the evidence didn’t travel.
This is not a claim about bad faith. The Essex pilot was run honestly. The Verify pilot was run with genuine intent. The problem is not in who designed the pilots. It is in what the pilots were understood to prove.
A pilot run with selected participants in controlled conditions proves that the technology can work in those conditions, with those participants. It does not prove that it will work at scale, across the full range of people it is intended to serve, in the uncontrolled circumstances of real deployment. Public sector governance currently treats those two things as if they were the same claim.
The question the framework doesn’t ask
The National Audit Office has recommended, repeatedly, that government learn the lessons of past technology failures before approving new programmes. It made that recommendation before GOV.UK Verify closed. It made it after. It will make it again after the next programme reaches the same conclusion. The recommendation exists because the pattern exists. The pattern continues because the governance framework asks the wrong question.
The question it asks is: did the pilot succeed?
The question it does not ask is: who was in the pilot, and who was not, and what happens to those people when we roll this out?
Those are different questions. The first is about the technology’s capability in a controlled environment. The second is about its reliability across the full population it will eventually serve. At the point of rollout approval, the second question is not a formal requirement. It is not even a common one. The NAO can recommend it. The PAC can document the consequences of not having asked it. Neither recommendation nor documentation has made it a condition of proceeding.
Two sectors, one structural question
Grip and Verify failed for different proximate reasons. The Grip tool’s underlying data reflected historical policing activity rather than actual crime distribution. The model learned where police had previously looked, not where crime had actually concentrated, and then directed officers back to those same places. GOV.UK Verify failed in part because government departments refused to commit, building parallel identity solutions and weakening the central standard. These are different causal mechanisms, and saying they share a structural condition is not saying they are the same failure.
What connects them is not why they failed. It is why the failure wasn’t anticipated. In each case, the pilot was designed in a way that made the eventual failure hard to see from inside its own frame of reference. The people who would surface the failure were the people the pilot had not included. They were absent by design, not maliciously, but structurally, because a pilot designed to generate positive evidence will generate positive evidence.
The sectors have different vocabularies and different oversight bodies. Policing technology critics focus on algorithmic bias and civil liberties. Digital government critics focus on departmental politics and procurement process. Neither community asks the same question: what was the pilot actually designed to find and what was it therefore incapable of finding? The structural failure they share has no owner.
The next public sector technology programme will follow the same process. A pilot will be designed. It will succeed. The board presentation will show the evidence. The rollout will be approved. Somewhere in the population that the pilot didn’t include, a person will encounter the technology in a way it was never tested against. The service behind it will continue to process paper, quietly, because the system cannot handle the case.
These, and many others, fall into a single lens approach when it comes to a pilot, a better way is to establish a more appropriate and relevant structural stages process that builds upon the previous stage. Define the problem from an outside perspective, what is it this will need to deliver and why. Then start with a Proof of technology - prove the end-to-end technology will actually do what it says. Then move to a Proof of Concept - test the idea out in full across a broad spectrum of positive + negative use cases - will it work; Then onto Proof of Value - will this actually deliver the outcomes promised in whatever benefit realisation model has been agreed. Only when all of these have passed their individual tests should you move to a Pilot and beyond into scale rollout.
What would it take to require an answer, before rollout is approved, to the question: for whom does this not work, and what will we do about them? Nobody has built that into the governance framework yet. The NAO keeps asking. The pattern keeps repeating.
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.