The Model Drift Problem
When AI improves, past decisions become harder to defend - The barn door is open. Nobody knows when it opened.
Morgan had been thorough. That was the thing that made it so difficult to explain when the question came back six months later.
The assessment had been careful — a credit risk evaluation for a commercial loan application, supported by an AI-assisted analysis platform the team had been using for the better part of a year. The tool had been performing well. The outputs were coherent, explainable, defensible. The assessment reached a conclusion, the decision was recorded, the file was closed.
Six months later, a query arrived from legal. Could the team reproduce the original assessment? Run the same inputs through the same system and show that the same logic applied?
The query was run. The output was different.
Not marginally different. Not a rounding variation. Different in a way that, had it been the original output, might have changed the conclusion. Different in a way that could not be explained without knowing what had changed in the model between then and now — and that information was not available, because nobody had recorded it, because nobody had been asked to, because when the system went live, nobody had thought to ask.
The problem with the update
AI systems are not static. The models underlying them are updated — sometimes by the vendor, automatically, as part of routine improvement cycles; sometimes in response to identified bias or error; sometimes because the training data has been refreshed. This is, in principle, good. A model that learns and corrects over time is more valuable than one frozen at the point of deployment.
The difficulty is that improvement and consistency are in direct tension. A model producing better outputs today than it did six months ago is, by definition, a model that would produce different outputs given the same inputs from six months ago. That difference is the point of the improvement. In any context where a decision needs to be reconstructed or defended, it is also an unanswered question about what the original decision was actually based on.
Most organisations deploying AI-assisted tools have not resolved this tension. They have ignored it. The model updates. The outputs change. Nobody is notified.
Nobody records the version in use when a specific decision is made. The audit trail — if one exists — captures the conclusion. It does not capture the reasoning engine that produced it.
Morgan’s situation was not unusual. It was the predictable result of deploying a consumption-based AI tool without the governance to track what the tool was doing over time. The barn door had been left open. The question was not whether the horse had bolted. The question was which horse, when, and whether anyone had kept a record of the herd.
The infrastructure that was not built
For an AI-assisted decision to be defensible over time, three things need to be true.
The version of the model in use at any given point needs to be recorded.
The inputs to any significant decision need to be preserved in a form that can be rerun.
The outputs need to be traceable back to the specific model state that produced them.
None of these are technically difficult. Version control is a solved problem. Input logging is standard practice in well-governed systems. Audit trails linking outputs to model versions are routine in regulated software environments. The infrastructure exists.
The decision not to apply it to AI deployment is not a technical gap. It is a governance gap — the same gap that appears in every wave of technology adoption where deployment has consistently moved faster than the thinking that should accompany it.
In financial services, decisions about lending, credit, and investment carry regulatory obligations around explainability and reproducibility. In healthcare, an AI-assisted diagnostic or triage decision needs to be reconstructable if a patient outcome is later questioned. In law, the reasoning underlying a recommendation used in proceedings may face scrutiny that requires reconstruction. The sector changes. The underlying failure is the same.
What makes this particularly hard to catch is that model drift is invisible by design. The system continues to function. Outputs continue to arrive. Nothing signals that the model today is not the model that was running when the previous decision was made.
The degradation of the audit trail happens silently, with every update cycle. By the time someone needs to reconstruct a decision, the model that produced it may no longer exist in any recoverable form.
Organisations that deployed AI-assisted tools before any framework for tracking model versions was in place are already in this position: the original model gone, the update history undocumented, past decisions unreproducible. This is not a future risk. It is the present condition for most organisations that moved quickly to deploy.
When the query comes
The most immediate consequence is legal exposure. In financial services, the FCA’s Consumer Duty requires that AI-assisted decisions be explainable and that firms maintain records sufficient to reconstruct the decision-making — not just the conclusion but the model state active at the time. The Data (Use and Access) Act 2025, which received Royal Assent in June 2025, adds further safeguards around automated decision-making, including the right to information about the reasoning used and the ability to contest decisions. A firm that cannot reproduce the original output cannot satisfy either obligation. The decision may have been correct. Without reproducibility, that cannot be demonstrated.
In healthcare, the consequences sit closer to the individual. An AI-assisted triage or diagnostic support tool that has been updated between a clinical decision and a subsequent review creates a gap in the clinical record. The original output cannot be reproduced. The basis for the original decision becomes opaque in retrospect. That opacity matters most precisely when the outcome was not what was hoped for.
In law, the problem is evidentiary. Any AI-assisted assessment used as part of a legal process — risk scoring, document analysis, due diligence support — may be subject to scrutiny requiring the reasoning to be reconstructed. A system whose model has changed since the original output was produced cannot provide that reconstruction. The question of what the AI actually considered, and on what basis, becomes unanswerable.
Beneath all of these is a more general corrosion. Organisations deploy AI tools to improve consistency — more reliable, more defensible, more uniform outputs than human judgement alone. Model drift undermines that promise at its foundation. The consistency was real at the point of deployment. Whether it persisted through subsequent updates is unknown, because nobody was watching. Every benefit of the deployment becomes contingent on the governance infrastructure most organisations have not built.
Closing the door
The work required to manage model drift has to be done before the problem appears, not after. Once a decision has been made, the model has been updated, and the version history has not been kept, the gap cannot be closed retrospectively. The barn door cannot be rehung from inside the field.
Version pinning is the starting point: every AI-assisted decision of significance should record the model version, the input data, and the output at the point it is made. This is not different in principle from the version control already applied to regulated software in financial services, healthcare, and legal environments. The principle is established. The application to AI tools has not been made.
Vendor contracts are the second lever. Most organisations deploying AI platforms under SaaS or consumption-based models have no contractual requirement for model update notification, version documentation, or backward reproducibility. This is a procurement failure as much as a governance one. The capability to reproduce historical outputs may exist technically with the vendor. Without a contractual obligation to preserve and provide it, that capability is at the vendor’s discretion.
The AI Business Office — the governing function responsible for AI consumption and risk across the enterprise — is where both of these responsibilities sit. Not in compliance, not in IT. In a function with genuine authority to set standards and enforce them — not merely to advise. That function is absent in most organisations deploying AI today. Its absence is the systemic condition that makes Morgan’s situation not exceptional but typical.
The deployment decision is the easy part. The governance that makes it defensible over time is slower, less visible, and less likely to feature in the business case. It does not appear in a vendor demonstration. It is the work done — or not done — in the months after the system goes live, once the attention has moved on and the question of what comes next has not yet been asked.
By the time it is asked, in many cases, the answer is already too late for the decisions already made.
The problem predates the AI
Model drift is not new. Statistical and machine learning models have drifted as long as they have been deployed. Any model trained on data and used over time will drift as conditions change, as data is refreshed, as the model is retrained — actuarial models, credit scoring systems, fraud detection engines. The problem is older than the current generation of AI tools.
What AI has changed is the speed, the scale, and the opacity of what changed. Traditional software updates come with version numbers, change logs, release notes. AI model updates change patterns in parameters, not lines of code — what changed is often not just undocumented but undocumentable in the same form. The update cycle is faster. The number of decisions affected is larger. The answer to “what exactly is different, and in which direction?” is harder to state even when someone thinks to ask.
This creates a provenance problem that existing governance frameworks were not built to handle. The question is not only whether the model drifted. It is whether anyone can say what it drifted from, when, and with what consequence for the decisions made in between. Most organisations cannot answer that. Not because they are careless. Because the infrastructure to track it was never put in place.
Organisations moving into AI-assisted decision-making in regulated contexts are taking on an obligation that most have not costed and many have not identified. The question is not whether model drift will affect their decisions. For most organisations that have been running AI-assisted tools for more than a year, it already has. The question is whether they will know.
What the question from legal would reveal
If your organisation has deployed an AI tool supporting significant decisions — in lending, clinical assessment, legal analysis, or any regulated context — can you reproduce the output of a decision made twelve months ago? Not approximately. Exactly, from the same inputs, using the same model.
If your AI platform vendor released an update in the last six months, were you told? Is the update version recorded anywhere against the decisions made before and after it? Does anyone in your organisation own that responsibility?
When you consider the decisions your organisation has made with AI assistance over the last year, which would be most difficult to defend if questioned today? What would you need to reconstruct the reasoning — and is that information still available?
The work that makes AI deployment defensible over time costs money and requires deliberate decisions to build. The absence of it also costs money — in legal exposure, regulatory risk, and decisions that cannot be explained. Which investment is your organisation actually making?
Authors Note
Morgan is a fictional character. Their story is drawn from a combination of professional observation and personal proximity to real events. The experiences described are real. The person is not.
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.