The pattern has been documented often enough that it has a name. An AI agent is given the instruction to find options — for a flight, a supplier, a candidate. It checks availability, compares prices, retrieves payment details from a linked account, and completes the task. A confirmation arrives before any human has made a decision. The outcome may be correct. The price reasonable. And yet a decision was made that no person made, and if it had gone wrong — wrong date, wrong airport, wrong account charged — the question of what to do next would be genuinely unclear.
This is the agentic AI moment. Not the science fiction version, where a machine plots against its creators. The mundane, already-here version, where AI systems that can act are being given access to the things that matter — calendars, inboxes, bank accounts, business systems — and the line between “help me with this” and “do this” is blurring faster than the accountability structures can keep up.
From chatbot to colleague
The shift from conversational AI to agentic AI is not incremental. It is a change in the nature of what the technology does.
A conversational AI — the kind most people encountered first — responds. You ask it something, it answers, the interaction ends. It can be wrong, it can be inconsistent, but it cannot do anything in the world beyond produce text. The consequences of its failures are bounded.
An agentic AI operates differently. Given a goal, it plans a sequence of steps to reach it, calls on tools — APIs, databases, software interfaces, payment systems — executes each step, monitors the results, and adjusts if something fails. It does not wait for instruction at each stage. It works. This is why the transition from 2023’s chatbots to 2025’s agents felt, to people paying attention, like something more than an upgrade. OpenAI released Operator, Anthropic extended Claude to computer use, Google launched Project Mariner — all in the same period, all enabling AI to interact with real systems on a user’s behalf. The architecture changed. The capability changed with it.
The adoption numbers reflect this. A Zapier survey of enterprise leaders in early 2026 found around 72% were using or testing agentic systems. Gartner forecast that 40% of enterprise applications would embed task-specific AI agents by the end of the year. The agentic AI market, valued at roughly $7.3 billion in 2025, is projected by Fortune Business Insights to reach $139 billion by 2034 — though other analysts project considerably higher figures. These are not pilot-project figures. This is live deployment across large enterprises.
The capability that makes agentic AI effective — the ability to act without waiting for permission at each step — is precisely what makes it difficult to govern.
The use cases making the business case are concrete enough. An AI agent that handles the full lifecycle of an insurance claim — reading the form, cross-referencing policy, assessing evidence, initiating payment — compresses a process that once took days into minutes. Walmart’s agentic supply chain system detects demand signals, adjusts procurement plans, and reroutes logistics without human triggers, with reported reductions in out-of-stock incidents across pilot regions. In software development, agents now write, test and deploy code: Stack Overflow’s 2025 developer survey found 84% of developers using AI tools, tools that now generate around 41% of all code written. Research published by Landbase, a sales automation firm, puts average returns from mature agentic deployments at 171% — roughly three times the return from traditional automation.
These are genuine improvements. They are also the easy half of the story.
The accountability gap
The hard half concerns what happens when the agent is wrong.
In 2024, a Canadian tribunal held Air Canada liable after its customer service chatbot gave a grieving passenger incorrect information about bereavement fare discounts. Air Canada’s defence — that the chatbot was a separate legal entity responsible for its own statements — was rejected without hesitation. The airline was responsible. The principle was unambiguous: organisations are accountable for what their AI systems say and do. The machine cannot be sued. The humans behind it can.
That principle becomes considerably more complicated when the AI is not talking but acting. A wire transfer sent to the wrong account. A contract clause accepted on behalf of a business. A medical appointment cancelled because an agent misread a scheduling instruction. In each case, the action has real-world consequences that may be difficult or impossible to reverse, the chain of decisions that led to it may involve multiple models, APIs and data sources, and the human who nominally authorised the deployment may have had no visibility into the specific step that went wrong.
The governance figures sit uncomfortably alongside the adoption figures. A 2025 industry survey found only 44% of organisations had formal AI governance policies in place. Only 11% of enterprises had agentic systems actually running in production rather than pilot. Deloitte found that 42% of organisations were still developing their agentic strategy, and 35% had no formal strategy at all. A Fortune report published in March 2026 — drawing on research from Wharton and Accenture — put the problem plainly: AI agents were spreading across the enterprise value chain, often ahead of formal strategy and governance.
The Wharton Accountable AI Lab framed it as a supply chain problem. Accountability is not a single organisation’s responsibility — it runs from the model developers who trained the system, through the platform providers who packaged it, to the organisations that deployed it, and finally to the humans who were supposed to be supervising it. When something goes wrong, everyone points to someone else. And in a system where a single agent action might involve five external data sources, three AI models, and four different APIs, the pointing is genuinely difficult to resolve.
The human in the loop — or not
The phrase the industry has settled on is “human in the loop” — the idea that a person remains involved in agentic decisions, available to check, correct or override. It is a reassuring phrase. In practice, it describes a spectrum, and much of the deployment happening now sits at the end where the human is less supervisor than emergency contact.
Microsoft has coined a different formulation: the “agent boss” — the person who builds agents, delegates to them, and manages them. The role is real and the framing is honest. But it also makes visible something the more comfortable language obscures. If one person is now the agent boss for a fleet of AI systems each capable of taking hundreds of actions per day, the ratio of human attention to autonomous action has shifted dramatically. The human is not in each loop. They are above all the loops, available to intervene if something surfaces — which requires that something surface, which requires that the systems are designed to surface it.
Fortune’s March 2026 report documented what it called “shadow AI” — the use of AI tools, including agentic systems, that employees bring into the workplace outside formal IT channels, beyond the visibility of any governance structure. Microsoft’s 2025 research found around three-quarters of knowledge workers were already using AI tools; a significant proportion were doing so with tools their employers had not sanctioned and could not audit. An orphaned agent — one running in production with no designated owner, accessing systems its original designers never intended — does not typically go rogue. It accumulates risk silently, its permissions unreviewed, its behaviour unchanged as the business context around it evolves.
The human is not in each loop. They are above all the loops — available to intervene if something surfaces. Which requires that something surface.
None of this is an argument against agentic AI. The productivity gains are real. The compression of tedious, error-prone, time-consuming processes is real. The question is not whether to use agents. It is whether the organisations deploying them are building accountability in from the start, or treating governance as something to address after the value has been captured. The evidence, on balance, suggests the latter is happening more often than the former.
The test most deployments are failing
Most current deployments pass the first question easily. The AI that clears an inbox backlog, processes a claims queue, or monitors a live system for failures is doing work that was real, time-consuming and often unrewarding. People who use these tools report genuine relief. That part is not in doubt.
The harder question is whether the agent adds something — capability, judgement, insight — or whether it substitutes for human understanding in ways that hollow out the understanding itself. There is a difference between an AI that surfaces relevant information so a person can make a better decision, and an AI whose recommendations are so compelling, and whose interface so frictionless, that human approval becomes a formality. A hospital AI diagnostic study cited in the Wharton and Accenture report found exactly this pattern: at high-performing sites, clinicians understood the system’s confidence scores and knew when to question their findings; at lower-performing sites, they either followed the tool uncritically or ignored it entirely. The tool was the same. The human relationship with it was not.
The third question — whether the agent responds to the specific person it serves, or treats every user as the average case — is the one most consistently failed. The documented pattern of agents interpreting “find” as “do” is a small example: the system optimised for task completion and treated the distinction as a rounding error. At the scale of enterprise deployment, that rounding error multiplies. The question of whether the people affected by agentic decisions — employees, customers, citizens — are treated as participants or as variables is not a technical question. It is a design choice.
My opinion
The accountability question underneath agentic AI has not been seriously answered yet, and I don’t think it will be until something goes badly wrong at a scale that cannot be managed quietly. The opportunity is real — the productivity gains are documented and the compression of genuinely tedious processes matters. But the responsible structure for deciding who owns the outcome when an agent acts on a person’s behalf has not been established. Was it the person who clicked through the terms? The company that deployed the agent? The vendor who built the platform? The model developer underneath all of it? Every layer points to the next. Right now, that costs individual people inconvenience and embarrassment. The price of that uncertainty will rise.
Who answers for it
The capability that makes agentic AI effective — the ability to act without waiting for permission at each step — is precisely what makes it difficult to govern. The technology is not the problem. The gap between how fast it is being deployed and how slowly the accountability structures are following it is.
The consequences of that gap are still emerging. Not all of them will be recoverable.
Sources and references
Agentic AI adoption — 72% of enterprise leaders: Zapier, State of Agentic AI Adoption Survey 2026. Survey of 500+ enterprise leaders. zapier.com/blog/ai-agents-survey/
Gartner — 40% of enterprise applications to embed AI agents by end of 2026: Gartner press release, 26 August 2025: “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025.” gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026
Agentic AI market size — $7.3 billion (2025) projected to $139 billion (2034): Fortune Business Insights, Agentic AI Market Size, Share & Forecast Report 2026–2034. fortunebusinessinsights.com/agentic-ai-market-114233. Note: other analysts project higher 2034 figures (Precedence Research: $199 billion; Grand View Research: $183 billion by 2033).
OpenAI Operator, Anthropic Claude computer use, Google Project Mariner: Released in late 2024 and early 2025. Confirmed via product announcements from OpenAI, Anthropic, and Google.
Walmart agentic supply chain system: Walmart Global Tech blog; Supply Chain Dive, “4 ways Walmart is scaling AI to unify its supply chain,” 2025. tech.walmart.com; supplychaindive.com/news/4-walmart-supply-chain-ai-uses/760891/
Developer AI tool usage — 84% of developers; 41% of code: Stack Overflow, 2025 Developer Survey. stackoverflow.com
Returns on agentic AI deployments — 171% average ROI: Landbase, 39 Agentic AI Statistics Every GTM Leader Should Know in 2026. landbase.com/blog/agentic-ai-statistics. Note: Landbase is a commercial sales automation firm; this is industry research rather than independent analysis.
Air Canada chatbot liability ruling: Moffatt v Air Canada, British Columbia Civil Resolution Tribunal, 14 February 2024. Reported by CBC News: cbc.ca/news/canada/british-columbia/air-canada-chatbot-lawsuit-1.7116416
AI governance policies — 44% of organisations: Corroborated across multiple 2025 industry surveys including IAPP AI Governance Profession Report 2025 (iapp.org) and SailPoint enterprise security research.
Only 11% of enterprises with agentic systems in production; Deloitte — 42% developing strategy, 35% no strategy: Deloitte, 2025 Emerging Technology Trends Survey (500 US technology leaders, June–July 2025). deloitte.com/us/en/about/press-room/state-of-ai-report-2026.html
Fortune / Wharton / Accenture report — March 2026: “Intelligence may be scalable, but accountability is not,” Fortune, 26 March 2026. Joint research from Accenture’s global products practice and the Wharton AI & Analytics Initiative. fortune.com/2026/03/26/ai-agents-accountability-accenture-wharton-report/
Wharton Accountable AI Lab: Led by Professor Kevin Werbach, Department of Legal Studies and Business Ethics, Wharton School, University of Pennsylvania. ai-analytics.wharton.upenn.edu/wharton-accountable-ai-lab/
Hospital AI diagnostic study — high- vs low-performing sites: Cited in the Wharton and Accenture report (Fortune, March 2026). See above.
Shadow AI / knowledge workers using unsanctioned AI tools: Microsoft, 2025 Work Trend Index. Microsoft’s research found approximately 75% of knowledge workers using AI tools at work, with a significant proportion using tools outside formal IT channels. microsoft.com/en-us/worklab/work-trend-index
Microsoft “agent boss” framing: Microsoft Copilot product communications and AI documentation, 2025–2026. microsoft.com/en-us/microsoft-copilot
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.