The Board That Didn't Understand What It Had Approved
Board governance was built for financial risk. AI carries a different kind.
Alex had been on the waiting list for eighteen months before the letter arrived. It said that the authority’s housing needs assessment had been completed and that, based on the information held, Alex’s application had been assigned a priority band. Band C. The letter did not explain what Band C meant in practice. It did not describe how the assessment had reached that conclusion. It mentioned, in the third paragraph, that a decision-support system had been used to process applications, and that appeals could be submitted in writing within twenty-one days.
Alex submitted an appeal. The appeal was considered. The band was confirmed.
At no point in this process did Alex learn what information the system had used to arrive at its assessment, what weight it had placed on different factors, or why Alex’s particular circumstances — a care dependency, a fixed income, a medical condition that made the current address untenable — had resulted in Band C rather than Band B. The letter existed. The process had been followed. The appeal had been considered. The outcome held.
Somewhere in that housing authority, a board had approved the system. The paper had been well-prepared. The numbers were credible — the system processed applications in a fraction of the previous time, reduced inconsistency across manual assessors, and met the procurement criteria the implementation team had set out. The board had asked about cost, about timeline, about how the system would handle exceptions. It had asked about data retention and compliance obligations. Nobody had asked what Band C would mean to Alex.
The gap was not in the preparation
The people in that boardroom were not negligent. They were competent, experienced, and attentive. The chair ran a disciplined meeting. The questions raised were reasonable given the information in front of the room. And the information in front of the room had been assembled by people who understood the system — who had selected it carefully, procured it properly, and built a reasonable implementation plan.
The gap was not in the preparation. It was in the scrutiny.
To ask what Band C would mean to the people who received it required knowing enough about how the system worked to understand that the band was not an administrative category but an algorithmic output — shaped by weighted factors, training data, and configuration decisions made during implementation. To ask whether the appeal was a real avenue of recourse required understanding that an appeal reviewed against the same criteria that produced the original decision is structurally different from an appeal reviewed by an independent assessor with full information. To ask whether the failure modes had been examined required knowing that automated decision systems fail differently from human assessors — not randomly, but systematically, concentrated in the people whose circumstances differ most from the data the system was trained on.
Those questions were not asked. Not because they were unanswerable. Because they were not visible to the people in that room.
The risk that doesn’t show up in a financial model
Governance models were designed for a world in which the primary risks of a large technology procurement were financial and operational. Could the organisation afford it? Would it be delivered on time? Would it integrate with existing infrastructure? Did the vendor have a track record? These are legible questions for a board with deep expertise in finance, law, and organisational management. The methods for evaluating them are established. The signals of risk are familiar.
AI and data-driven decision systems carry a different kind of risk. The failure is not that the system goes over budget, though it may. The failure is that the system makes consequential decisions about real people based on patterns extracted from historical data — and if that data reflects the inequities and assumptions of the processes it was trained on, the system reproduces those inequities at scale, faster, with the additional authority of technological neutrality. That failure does not appear in a financial model. It does not surface in a project risk register. It does not emerge in a compliance review that covers data retention and lawful processing but not the distributional consequences of the decisions being made.
It surfaces in a letter.
The boards approving these systems are not, in the main, composed of people with the literacy to see that risk. That is not a criticism of the individuals — it is a description of how board composition evolved in response to the risks governance processes were built to manage. For most of the history of organisational governance, technology decisions whose consequences were primarily operational could be evaluated through financial and legal lenses without significant loss. That is no longer reliably true.
The structure of most board meetings compounds the problem. Papers are summarised to one page. Technical detail is compressed into reassurance. The people with the deepest knowledge of the system are usually the people most invested in its approval. Independent scrutiny — not from the vendor, not from the implementation team, but from someone whose only brief is to look for what could go wrong — is not a standard feature of most technology governance papers. It is an occasional addition, when the stakes are visible enough to prompt it. The stakes were not visible. They were in Band C.
In the EU, the AI Act — in force since August 2024 — classifies AI systems used to assess housing eligibility as high-risk, requiring transparency, human oversight, and challengeable outputs from August 2026. In the UK, the government’s Algorithmic Transparency Recording Standard is mandatory for central government departments; a bill that would have extended comparable requirements to local housing authorities stalled in 2024 without progressing. For the authority that assessed Alex’s application, and thousands like it, no such requirement currently applies.
Until they do, the room that approves the system and the person who receives its output will remain two entirely separate worlds.
The accountability that disperses
When an AI-assisted decision system produces a wrong outcome, the accountability for it is distributed across enough points in the governance chain — the vendor, the implementation team, the commissioning officers, the board itself — that it disperses without fixing to any of them. The appeal process, if it reviews the original decision against the same criteria that produced it, is not a genuine avenue of recourse. It is a confirmation mechanism.
Alex’s appeal eventually succeeded — Band B assigned, not because the system was reviewed, but because one housing officer with the right information made a different call. The system continued processing the next two thousand applications unchanged.
This is the pattern, not the exception. An authority reviewing its own AI deployment will see a system operating as designed — applications processed, appeals handled, no formal complaints logged against the technology. The harm does not show up in a post-implementation review. It accumulates in the cases where the right advocate was not present, where the first appeal was the last one, and where the person whose circumstances fell outside the training data had no way of knowing that fact.
A revised band and a revised system are two very different outcomes that look, from inside the authority, like the same one.
Three questions that change what gets asked before approval
The questions that were not asked in that boardroom are not technically difficult. They do not require a specialist in machine learning or algorithmic auditing. They require a different starting point — one that begins with the person at the end of the system rather than the system itself.
Before approving a technology that will make consequential decisions about people, a governance body should be able to answer three things.
The first is whether this technology actually helps the people it processes — not faster, but better. Does it make the experience easier for the person submitting an application under stress, with limited time, with a care dependency that doesn’t fit neatly into any field? The paper that came to that board had been prepared by the people implementing the system. The people at the end of it had not been consulted. Whether that consultation is required — as a governance condition, not a supplementary appendix — is a governance choice.
The second is whether the technology adds something real, and to whom. A system that processes applications faster is not automatically one that serves applicants better. It may serve the organisation better — reduced staffing costs, greater throughput, more consistent output — while producing outcomes that are worse for the specific people the organisation exists to help. The board had asked about processing time. It had not asked what the improvement looked like from the side of the person submitting the application.
The third is whether the technology can respond to the individual circumstances of the people it affects — and what happens to the people whose circumstances fall outside the parameters it was trained on. That question, asked before approval, changes the procurement criteria. It requires the vendor to demonstrate that edge cases are handled with appropriate human judgement, not simply processed by the same logic as every other case. It makes an independent appeal process a governance condition of deployment rather than a feature added after complaints arise.
A board that cannot answer those three questions before approving a system has not governed the technology. It has approved a budget line. The governance is what happens in between.
My Opinion
Most organisations are not prepared for what is coming. If they treat AI governance as a variant of the regulatory models they already have — compliant, procedural, box-ticking — it will fail. We have not yet reached agreement on what AI actually is, which makes governing it a peculiar challenge: the responsibility sits with people who cannot reliably describe the thing they are responsible for. The relevant legislation is also not contained within a single jurisdiction — a service consumed by a user in another country may bring obligations the commissioning authority has never considered. That is the gap. It is wider than most boards currently know.
Before the next approval
Could the people who approved a significant technology decision at the senior level of an organisation you know describe, in plain language, what it does and what its consequences are for the people at the end of it? If not — what does that say about the quality of the oversight it received?
If you sit on a board or senior governance body, what is your personal level of literacy in the specific technologies your organisation uses or is considering? Is that literacy sufficient to probe the assumptions in the papers you are asked to approve — or are you relying on the competence of the people who prepared them?
If you are a technology leader who presents to boards, what is your responsibility to ensure genuine understanding rather than informed consent? Are you building presentations that equip boards to scrutinise, or ones that build confidence in a decision already made?
And the hardest one: is it possible to be accountable for a decision you did not understand? If accountability requires understanding, and the understanding was not present in the room, where does the accountability for what happened to Alex actually sit?
The board never knew Alex’s name. That is, in the end, the clearest description of the problem.
Authors Note
Alex is a fictional character. Their story is drawn from a combination of professional observation and personal proximity to real events. The experiences described are real. The person is not.
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.