The Questions Nobody Asks in the Room
Are we sleepwalking into future issue when bringing AI into the business?
Domain: Technology & AI | Arc 6: The Repeating Pattern | Theme: The right questions aren't hard — just unasked.
Sam had been waiting fifteen minutes for the presentation to begin. The room was filling up — a mix of senior technology people, a few faces from finance, the kind of gathering that happens when a global vendor comes to show what their platform can do. The slides were already on screen. Animated. Confident. The kind of design that says: we have thought of everything.
For the next forty-five minutes, they watched what the technology could do. It was impressive. It really was. The speed, the pattern recognition, the way it surfaced insights that would have taken a team weeks to produce. At one point, the presenter — not a developer, they mentioned this proudly — showed an application they had built themselves in a couple of hours. No coding. Just prompts and clicks and there it was, working.
At the end, someone in the room asked a quiet question.
So — what’s the USP?
The room went slightly still. It was a reasonable question. After forty-five minutes of capabilities, nobody had explained what problem this solved that couldn’t be solved another way, what happened when it went wrong, or what the organisation would look like three years from now if it said yes. The presentation had been entirely about what the technology could do. Not a single word about what it couldn’t. Not a word about what came after.
Sam drove home thinking about that silence.
We are in the middle of one of the most significant technology cycles of the last thirty years, and the conversation happening in boardrooms and meeting rooms across the country is almost entirely wrong. Not wrong in the way that dismisses AI, that ship has sailed, and anyone still arguing about whether artificial intelligence is real or useful probably doesn’t understand the capabilities. Wrong in a different, more dangerous way.
We are asking the wrong questions. Or rather, we are not asking questions at all. We are watching demonstrations.
The question nobody asks in these rooms is the one that matters most: what does the full picture look like? Not just the capability, but the consequence. Not just what the system can do on Tuesday afternoon in a controlled demonstration, but what it does to the organisation that depends on it, at scale, under pressure, eighteen months from now, when something goes wrong, when the vendor changes their pricing model, when the contract ends.
AI is a tool. A remarkable one. But it is being bought like a consumer product and deployed like infrastructure, and those two things are incompatible.
The end-to-end problem
When a technology vendor shows you how AI has reduced a process from days to hours, the correct question is not how impressive. The correct question is: what happens next?
Consider law enforcement. AI can compress investigative work that once took months into days. That is genuinely valuable. But the Crown Prosecution Service still operates on the same timescales it always has. The courts have the same capacity. The prisons have the same number of beds. You have accelerated one stage of a system whose other stages have not moved. The bottleneck has not been removed, it has been relocated. And in the relocation, it has probably got worse, because now the pressure is arriving faster and the downstream simply cannot absorb it.
This is not unique to law enforcement. It applies everywhere. If AI reduces your customer service queue from ten thousand tickets to one thousand, the right question is not what a saving it is: what does this mean? Did you have a backlog you can now clear? Can your team move to higher-value work? What does the value actually translate to in terms of measurable, real-world outcome? The organisations that cannot answer that question are the ones that will spend a great deal of money and end up wondering what they bought.
The end-to-end question is not a technology question. It is a business design question. And it is almost never asked before the contract is signed.
The continuity question nobody plans for
Business continuity planning is not a technology exercise. It is a question about how an organisation continues to function when something it depends on fails. And most organisations have not seriously asked that question about their AI dependencies.
What happens when the AI agent goes down? What happens when the vendor’s platform has an outage, not a short one, but a sustained one? What happens when the vendor is acquired, pivots its product, or simply goes out of business? We are building operational dependencies on systems we do not control, provided by companies whose priorities are not ours, running on infrastructure we share with thousands of other organisations. And the business continuity plans being written today, in most cases, were not written with any of this in mind.
There is a particular version of this problem that deserves attention on its own: legacy code debt. That presenter who built an app in a couple of hours without being a developer — when everyone can do that, what accumulates in the estate? Good architecture requires understanding dependencies, failure modes, security considerations, performance under load. AI-assisted development shortcuts all of that, not because the people using it are careless but because the tool does not surface what it omits. Every organisation deploying AI-generated code without a proper review process is building debt it does not know about yet.
The bill for that debt will arrive. It always does.
The security hole hiding in plain sight
There is a straightforward fact about current large language models that is not in the vendor presentations: there is no reliably secure way to prevent sensitive data entered into a chat-based AI system from being retained, exposed, or used in ways the user did not intend. This is not a theoretical risk. It is an active one. Every employee who pastes a client contract, a personnel file, or internal strategy into a public-facing model has potentially moved that data outside the organisation’s control.
The question is not whether your employees are doing this. They are. The question is whether your organisation has a framework, a genuine AI Business Office with authority, not a policy document gathering dust, that governs what gets used, by whom, in what context, with what data. Most do not.
Did you know?
Under UK GDPR (Article 15 and Article 22) individuals have the right to request access to the personal data an organisation holds about them and, where AI is used in decisions that significantly affect them — a job application rejected, a credit request declined, a benefits assessment — to receive a meaningful explanation of the logic involved. Not a mathematical formula. Not a reference to the algorithm. A genuine explanation of how the decision was reached, in terms they can understand and challenge.
In February 2025 the Court of Justice of the EU clarified this further: providing the algorithm alone does not satisfy the obligation. The explanation must be specific to the individual’s case.
Now ask this: if your organisation uses AI to make or support decisions about people, can you produce a clear, human-readable explanation of how any individual decision was reached? Can you reproduce the state of the model at the precise moment that decision was made, the data it used, and the logic it followed? Can you do that consistently, at scale, months or years after the fact?
If the answer is anything other than yes, you have a legal exposure you may not have accounted for. And the fines for non-compliance under UK GDPR can be expensive.
Back to the Room
None of this is an argument against AI. The technology is real, the capability is genuine, and organisations that ignore it entirely will fall behind. But that is not the conversation that needs to happen. The conversation that needs to happen is the one Sam tried to start with a quiet question at the end of a forty-five minute demonstration.
What is the actual value here, end to end? What happens when it fails? What are we depending on, and what is the plan when that dependency breaks? What are our people building with this, and do we understand what they are building? What are our legal obligations to the people affected by the decisions this system supports?
These are not difficult questions. They are the questions any organisation would ask of a new supplier, a new process, a new piece of infrastructure. The fact that they are not being asked in AI presentations is not because AI is special. It is because the presentations are doing something specific: they are showing you the exciting part and moving quickly past the rest.
Forty-five minutes of what it can do. Not a word about what it cannot.
That silence is where the risk lives.
Four questions to sit with:
Does your organisation have an AI Business Office? not a policy, but a function with genuine authority over how AI is used, at what cost, and with what data?
Have you mapped what happens to your downstream processes when AI accelerates the upstream ones? Where does the pressure accumulate?
If an employee or customer submitted a Subject Access Request related to an AI-assisted decision made about them, could your organisation respond in full including the model state, the data used, and the logic applied?
And the one that started all of this: if a vendor spent forty-five minutes showing you what their AI can do, and someone asked what the USP was, what would your answer be?
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System, and writes at the intersection of technology, ethics, and human purpose.