The Bin Hasn't Changed. Only the Lorry got Faster
Garbage In, Garbage Out. Still true today as it has ever been, but what has changed is the speed at which bad data can have consequences.
Riley submitted the application on a Tuesday morning. The Universal Credit advance, emergency money designed to arrive within 24 hours, was there to bridge a gap that Riley had not expected and could not have planned for. Retirement, a short-term shortfall between pension arrangements, a position they had never been in before. The form took two hours. The supporting documents were gathered and uploaded. The submit button was pressed.
By Wednesday afternoon the money had not arrived. Instead, a message: additional documentation required. No explanation of what had triggered the request. No indication of how long the review would take. No suggestion that anything was wrong, but no money either, and no path through that Riley could see.
Riley had not done anything wrong, they hadn’t committed fraud or attempted to, they had simply applied for emergency money and been flagged by a machine learning model that had read their age and their nationality and produced an output that looked, from the outside, like due diligence.
What the system saw
The DWP’s AI model for Universal Credit advance claims has been in operation since May 2022. Its stated purpose is to flag potentially fraudulent applications before payment, a reasonable thing for a public body to want to do. The model was trained on historical data: years of records from the fraud investigation process, cases that had been referred for review, patterns extracted from that referral history.
The problem is what that data actually contained. The investigation records did not capture who committed fraud. They captured who was referred for investigation, a related but distinct category, shaped not by the behaviour of claimants but by the behaviour of caseworkers. Who the investigators looked at. Which patterns they had been trained to notice. Which demographic characteristics, accumulated over years of investigation practice, had become associated in the institutional record with the word “suspicious.”
The model was trained on the output of that process. It learned the pattern of the investigators, not the pattern of the crime.
Riley’s age, late sixties, and nationality were among the signals it read. The flag was not an error in the conventional sense. The model did exactly what it was designed to do. It identified a pattern in the training data and applied it. The pattern it identified was the product of how human investigators had behaved across years of decisions that nobody had examined before the data was fed into the model.
From Riley’s position, the process was illegible. The notification gave no reason. The documentation requested was not connected, in any explanation Riley received, to the specific concern the model had raised — because the model’s reasoning was not disclosed. The request could have been for any reason. Riley gathered the documents, submitted them, and waited in the particular uncertainty of someone who has done nothing wrong and does not know how to prove it to a process that hasn’t told them what it suspects.
What the data actually captured
The DWP published a fairness analysis in July 2025, the first such analysis of the model’s outputs. Its finding for age was specific: claimants aged over 66 are referred at 49 times the rate of those aged 35 to 44, a number the DWP’s own analysis asks to be read with caution, on account of sample size. Non-British nationals face a separate, statistically significant disparity. The DWP’s own conclusion was that these figures present minimal concerns of discrimination or unfair treatment, a judgement made in aggregate, against the model’s overall fraud-detection performance.
A person is not an aggregate. A system that fails one person has failed that person by a hundred per cent.
The model had been running for three years before those numbers were officially produced. Three years of decisions, made at machine speed, built on a pattern that the people deploying the model had not interrogated.
The finding is damaging not because the DWP is unusual in this respect, but because it is not. The problem of training AI on data that was collected for a different purpose, under different assumptions, by people who are no longer available to explain what it captured is not a DWP-specific failure. A Gartner survey of 248 data management leaders in 2024 found that 63% of organisations either did not have or were unsure whether they have the right data management practices for AI. A 2025 S&P Global survey of over 1,000 IT professionals across North America and Europe found that 42% of organisations had abandoned most of their AI initiatives that year up from 17% the year before. Gartner projected that 60% of AI projects would be abandoned through 2026 for the same reason.
The problem is the default. The question that should be asked before a model is trained — what does this data actually capture, and is it fit for this purpose? — is not in the procurement brief, not rewarded in the governance process that approved the deployment. An AI rollout has a sponsor, a business case, and a delivery deadline. A data audit has none of these things. The person who raises the prior question is seen as slowing the project down.
What happens instead is that the uncertainty in the data, the gap between what the data claims to capture and what it actually captured, is not addressed. It is absorbed. The AI model processes the data and produces outputs. Those outputs look different from the raw data they were trained on: formatted with a confidence that the underlying material did not have. A spreadsheet formula error looks like a spreadsheet formula error. An AI-generated flag looks like an AI-generated flag, processed, weighted, produced by a system that has reviewed thousands of cases. The uncertainty has been cosmetically removed. What was always imprecise is now presented as authoritative.
The speed is the problem. When human caseworkers made decisions based on biased investigative patterns, the bias was slow, inconsistent, and visible to anyone who looked at the records. Individual decisions could be challenged. Patterns could be identified over time by people paying attention. The bias was real and it caused harm, but it was legible.
When those same patterns are encoded in a model and applied at machine speed to every application that passes through the system, the bias becomes fast, consistent, and invisible to anyone without access to the training data and the analytical capacity to trace the pattern back through the outputs. The fairness analysis that revealed the disparity was published three years after the model began operating. Three years of Riley-shaped decisions, each one arriving with the authority of a system that had been approved, deployed, and described as an improvement.
This is what happens when you put dirty data into a fast lorry. The bin hasn’t changed. Only the lorry got faster.
What the system did not do for Riley
The system added friction and delay to an urgent process at the moment the process existed specifically to avoid delay. The machine learning model was positioned as a fraud-detection improvement, an addition to the system’s capability. What it added, in Riley’s case, was the appearance of due diligence applied to a demographic pattern that nobody had verified was a fraud pattern.
Riley was a first-time claimant with no fraud history, in a short-term financial gap, applying for emergency money. None of that changed the flag. The model read age and nationality and produced an output. Riley’s individual circumstances, the specific facts of this specific person’s specific situation, were invisible to the process. What was visible was the pattern. The pattern was wrong.
The right of appeal exists in theory. In practice, Riley received no explanation of why the flag was triggered and therefore had nothing specific to contest. An appeal process that cannot name what it is correcting cannot correct it. The accountability exists on paper. It does not exist at the level of the individual case, for the individual person, at the moment they need it.
Opacity and urgency compound each other in ways they do not in most other AI applications. Riley needed the money quickly. The delay imposed by the flag was not a minor inconvenience, it was the failure of the system to do the thing it existed to do, at the moment it was supposed to do it.
The request could not be contested because the reason for it was not given. The time taken to gather the documentation and wait for review was time Riley could not afford. The system had been designed to detect fraud faster. It had not been designed to protect the person it flagged from the cost of being wrong.
What your organisation has not asked
The last time your organisation deployed or fine-tuned an AI model, did anyone trace the training data back to its original purpose and ask whether it was collected under assumptions that still apply? Or did the procurement process treat the data as given, a condition of the project rather than a subject of scrutiny?
If your AI is working from data that came from a legacy system, a different team, or a previous era of policy, does anyone in your organisation know what was baked into it at the start? Not what the documentation says it captures. What it actually captured, given who produced it and under what instructions.
When an AI output is wrong, does your organisation have a process for tracing the error back to the data or does the inquiry stop at “the model needs retraining”? If it stops there, the same data produces the same pattern. The lorry gets faster. The bin stays the same.
How many decisions your organisation makes on AI-generated outputs have been tested against this question: would this output look different if the training data had been gathered differently? If nobody knows, that is the answer.
Riley eventually received the money. The process took eleven days. The explanation never came.
Authors Note:
Riley is a fictional character. Their story is drawn from a combination of professional observation and personal proximity to real events. The experiences described are real. The person is not.
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.