How to Design Morality into Machines
For when nobody decided what the system should value, and it found something to value anyway.
The hiring algorithm flags an eighteen-month gap on a strong candidate’s CV and quietly drops her two places in the ranking. Nobody at the company meant to penalise the time she spent caring for her mother. The model just learned, from years of past hiring records, that gaps correlate with lower retention — and turned that correlation into a rule nobody actually chose.
Machines don’t have conscience. But they operate in moral domains all the same — every choice they automate reflects a value, whether someone declared it or the system simply defaulted to one. The gap penalty was never a decision. It was a value, arrived at by accident, and nobody in the building would call it theirs.
The card holds six practice areas across two groups: what you design and encode before a system ever runs — naming the values it must serve, building ethical data practices, engineering for real transparency — and what keeps it honest once it’s operating: safety testing, clear accountability, and a standing habit of ethical review. Written for whoever builds these systems, though anyone who’s been quietly filtered out by one will recognise exactly what’s at stake.
It comes from The Next Evolution, where the fuller argument lives. The card stands on its own — you don’t need the book to use it.
Print it, or hand it to whoever’s about to ship the next model that will quietly decide who gets a chance.
So here’s what I’d ask the team behind that hiring model: if nobody chose the gap penalty on purpose, whose value is it now?
You’re reading The Next Evolution by Neil Catton, articles that explore the human world and the intersection of technology, they try and ask difficult questions - not to scare - but to inform. If someone forwarded this to you, you can subscribe free at neilcatton.substack.com.
Neil Catton is the author of The Next Evolution, The Cognitive Crucible and The Shadow System - available on Amazon, and writes at the intersection of technology, ethics, and human purpose.