A loan application gets scored by a model weighing hundreds of variables against years of repayment history, and it’s declined in under a second. The bank can’t say which variable did it — not because anyone is hiding the reason, but because the model wasn’t built to produce that kind of answer. Nobody lied. Nobody can fully explain it either.
The same pattern sits behind a job application filtered out before a person sees it, a news feed that decided what mattered today, an insurance premium that moved for reasons nobody named. These used to be judgement calls a person made, answerable to someone. Now they happen inside systems trained on the past, and the humans who built them are often the first to admit they can’t reconstruct the reasoning behind any single outcome.
The card holds five practice areas in one column: building real algorithmic literacy, demanding transparency and explainability, actively identifying and mitigating bias, establishing accountability that names who answers, and advocating for governance that puts human wellbeing ahead of efficiency. Written for anyone who wants to hold these systems to account — not only the people building them.
It comes from The Next Evolution, where the fuller argument lives. The card stands on its own — you don’t need the book to use it.
Print it, or send it to whoever just got a decision from a machine and wants to know why.
So before the next automated decision lands in an inbox: if the system can’t fully explain itself, should it still get to decide?
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.