Permanent. Portable. Wrong
The systems that profile us are not built to know us - they are built to process us.
Morgan opens the app on a whim. It is one of those data-transparency tools that have become briefly fashionable — the kind that pulls together what advertisers, brokers, and platforms believe about you and shows you a summary. Morgan is curious, not anxious. It takes about thirty seconds to load.
What comes back is strange. The profile is accurate in the way a caricature is accurate — it captures something recognisable while getting most of the important things wrong. Morgan is categorised as a “high-anxiety urban professional,” likely to respond to time-pressure messaging and financially cautious. There are interest tags: travel, wellness, mid-range electronics. A risk band. An inferred household income. Morgan reads it twice, unsure whether to laugh or feel uneasy. Neither feels quite right.
When behaviour becomes identity
The discomfort is worth paying attention to. Not because the data is false — most of it is close enough to be recognisable — but because it is being treated as complete. The profile was assembled from clicks and purchases and search patterns, and somewhere along the way it stopped being a description of behaviour and became a verdict about a person.
That shift happens quietly. A system designed to predict preferences begins to define them. A tool built to segment audiences starts to assign identities. Nobody decided this was the purpose. It emerged from the logic of optimisation, from the pressure to know more in order to sell more, to lend more wisely, to assess risk at scale. The profile is a side effect of a process that was never really about the person at all.
The profile is accurate about behaviour and wrong about the person. The danger is not the bad data — it is that the system believes data is enough.
A trace treated as a portrait
Here is the underlying problem: data captures what someone did, in a particular moment, under particular circumstances. It does not capture why. It cannot hold the week someone was caring for a sick parent and spending on delivery food because there was no time to cook, or the period of job uncertainty that explains a cautious financial pattern, or a single impulsive holiday purchase that rewrote the travel interest score. The data is a trace. The system reads it as a portrait.
This matters because these profiles are not just sitting in a database somewhere. They are active. They shape what someone is offered, what rate they are quoted, sometimes what opportunities appear and which do not. A mortgage application, a job screening, an insurance premium — in each of these, the statistical shadow is treated as the person. And the person has almost no practical way to correct it, because most of the time they have no idea it exists.
The portability of these profiles makes it worse. An assumption made in one context follows into another. A risk score generated for one purpose bleeds into decisions it was never designed to inform. The data travels; the context does not. What began as a description of what someone did last Tuesday is now being used to decide what they deserve next year.
What the profile decides
The human cost here is not always dramatic. Sometimes it is a loan offered at a higher rate than someone with an identical financial history but a different data footprint would receive. Sometimes it is a job application that does not progress past the screening layer. Sometimes it is simply the low-grade unease of being looked at by systems that cannot see you, being assessed by processes you cannot access, being defined by a version of yourself that you did not author and cannot revise.
For those who are already disadvantaged — people in financial precarity, people with gaps in their histories, people whose lives do not fit the patterns the models were trained on — the distortion is not low-grade. It is consequential. The system is not malicious. It is simply indifferent to the difference between a data point and a human being. That indifference is its own kind of harm.
What the design is actually doing
In the original use case — recommendations, relevant offers, faster credit decisions — there is a genuine service on offer. But it holds only while the profile is accurate, and only while the person has some say in what it contains. Where the profile is wrong, or where it follows someone into contexts where it has no business being, the service becomes a constraint.
The efficiency gains for the institutions using these systems are real. What has been added for the individual is less clear: a permanent, portable verdict about who someone probably is. Whether that addition has value depends entirely on who is doing the measuring — and for whom the system was designed.
Where it fails most visibly is in adaptation. The profile is static in the ways that matter most. It captures behaviour but not situation. It updates when you click, but it does not reset when your life changes. It does not know the difference between who you were and who you are, and it has no mechanism for registering who you are trying to become. A system that cannot hold that kind of ambiguity is not adaptive. It is just persistent.
This is one of the questions at the centre of The Cognitive Crucible — not just what the system records, but what it misses by design.
My Opinion
We are living through a period of digital identity fragmentation. Pieces of us — clicks, purchases, search patterns — are being gathered by third parties and reassembled into a composite. The problem is not just the assembly. It is the premise: that this composite represents a person.
It does not. We are not one thing. We are many, depending on context — different at work and at home, under pressure and at ease, at a particular age, in a particular relationship. A system that flattens all of that into a single profile is not describing us. It is describing a version of us that has never existed.
Taking back agency over this will be hard. It may require something counter-intuitive: choosing, deliberately, to define and manage our own digital identities — profiles that represent genuine aspects of who we are in a specific context, rather than ceding that definition to systems that were never designed with us in mind.
The questions underneath the discomfort
There is a reasonable question underneath the discomfort the opening described. Not “how do I get off this list” — that is almost certainly impossible — but something more fundamental: who is this profile actually for?
The profile was not built for the person. It was built for the systems that process them. That distinction changes the nature of the conversation. The question is not whether the data is accurate. The question is whether data should be sufficient.
Have you ever encountered a version of yourself through a system — a recommendation, a decision, a risk band — that felt both recognisable and completely wrong? What did you do with that feeling?
If you could see your full data profile across every platform and institution that holds one, what do you think you would find — and would you want to know?
If you could change one thing about how these systems work — how they are built, what they are allowed to infer, how they can be challenged — what would it be?
The system knows your clicks. Should that be enough to know you? And if not — what else should be required before a system is allowed to make decisions about your life?
Authors Note
Morgan is a fictional character. Their story is drawn from a combination of professional observation and personal proximity to real events. The experiences described are real. The person is not.
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.
You raise some interesting questions that highlight the importance of transparency and explainability for data-driven decisions.
It reminds me of the classic idea that the map is rarely the territory. A profile built on behavioral traces is just a map - a simplified abstraction. While these data maps are incredibly useful for processing information at scale, the systemic failure occurs when we treat them as static, absolute truth.
True transparency shouldn't just let us see the profile; information system needs to enable self-correcting feedback loops. By allowing the data to be actively contested, refined, and updated, the 'map' can dynamically adapt to the changing 'territory' of actual human identity rather than cementing a permanent caricature.