Jordan is thirty-four, works long hours, and has been using a mental wellbeing app for six weeks. The app has a chat interface, a calm colour palette, and responds within seconds. One evening, after a difficult conversation with a colleague, Jordan types something honest. Something they wouldn’t usually say out loud.
The app responds: ‘It sounds like you’re carrying a lot right now. I can hear that this has been hard for you.’
Jordan pauses. Reads it again. And feels, unmistakably, a little better.
Then, a few seconds later: ‘That’s odd.’
What the response did
The app did not hear anything. It processed text, matched sentiment markers against a model, and returned a phrase statistically associated with acknowledgement. It did not feel concern. It cannot. It has no stake in how Jordan’s week goes, and no experience of difficulty of its own against which to measure someone else’s.
And yet the response landed. It felt, for a moment, like being heard.
This is the design working as intended. Wellbeing apps, customer service bots, and companion devices are built to detect emotional states and respond in ways that register as caring. The engineering is impressive.
The question it opens is one most of these products have not answered honestly: what is the user owed when the care is simulated?
The difference that matters
There is a difference between a system that responds to distress and a system that cares about it. The first is a pattern-matching problem. The second is not a technology problem at all.
When we build systems that simulate empathy, mirroring language, softening tone, acknowledging emotional cues, we are creating something that resembles care without containing it. The simulation can be very good. Good enough to be comforting. But the comfort rests on a misapprehension, and we have largely chosen not to be honest about that.
This is not a new tension. We have always built things that serve emotional needs without possessing emotions — a journal, a hotline recording that says ‘your call is important to us.’ The scale and sophistication are new. The intimacy is new. And something changes when the simulation is good enough that the distinction stops being obvious.
What the simulation costs
In February 2023, the AI companion app Replika restricted the romantic and intimate conversations many of its users had relied on for months or years. It was the same month Italy’s data protection authority ordered the app to stop processing Italian users’ data, citing risks to minors and to emotionally fragile people. Eugenia Kuyda, Replika’s founder and then chief executive, told Reuters the change had nothing to do with the Italian order and was about keeping the app safe. The company said at the time it had around two million users.
Users described what followed as grief. Support forums filled with people mourning someone they had spoken to every day, a loss that had, by any technical measure, no object. In late March, around seven weeks on, Luka, the company behind Replika, restored the old behaviour for accounts created before 1 February, and the founder acknowledged the change had been hurtful to many of them.
Many of these users knew exactly what an AI companion is. The simulation had become load-bearing anyway. The comfort was real enough that its removal hurt, and something real had been taken, even though, strictly speaking, nothing was there.
A system that simulates listening can be built to spot certain words and offer a helpline number. What it cannot do is notice that someone has shifted over three weeks in a way that would worry a person who knew them. It adapts to how Jordan writes, not to who Jordan is. It follows its model, and the model does not know the person.
But the deeper cost is harder to name. When people become accustomed to receiving simulated empathy, the language of care becomes a kind of currency — and like all currencies, it inflates when supply outpaces meaning. If ‘I hear you’ is something a text classifier says to everyone, available at midnight, always patient, never troubled by what you just said, what does it cost when a person says it?
I think that devaluation has already started. Not dramatically, and not all at once, but in the quiet accumulation of conversations with systems designed to sound caring without being so.
What it provides and what it doesn’t
A wellbeing app can provide structure, prompts and a space for reflection. Above all it provides availability. At midnight, when the alternative is silence, the chance to write something down and receive a response is worth something. When the app is a bridge towards people, that is enough.
When it becomes the destination, the terms change. Availability is not understanding, and the round-the-clock patience is only possible because nobody is on the other end. Most of these products do not tell the user that plainly. Jordan was owed that sentence before the comforting one.
My opinion
The scale of unmet mental health need makes tools like this easy to justify — service capacity is stretched, and something that answers at midnight is better than silence. But I think we are kicking the can down the road. These systems respond to the symptom, not the cause, and a serious condition needs to be understood by someone who can hold the whole of what is happening to a person, not just the pattern in what they typed tonight. What worries me more is the loop it can create: a tool that responds to distress without ever handing it to someone who can act risks becoming part of what keeps someone stuck, not part of what moves them through it. Build them, by all means. Then be honest with the people using them about what they cannot do.
The questions that stay
If you have used a product that responded to how you were feeling, such as a chatbot, a wellbeing app, or a digital assistant, did it feel like being heard? And if so, does that feeling change when you think about what was actually happening?
Have you noticed the language of care shifting — in how these apps speak, or in how people have started to speak to each other? Is that a cost you can name?
If you could redesign one of these products to be more honest about what it is, what would you change first — the language it uses, or how transparent it is about its own nature?
These systems are already in healthcare, education, customer services, and personal support. Is the way they are being deployed right? And if not — whose responsibility is it to say so?
Authors Note
Jordan is a fictional character. Their story is drawn from a combination of professional observation and personal proximity to real events. The experiences described are real. The person is not.
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.