Shipping a feature is easy. Shipping one you can still defend six months later takes a few uncomfortable conversations. Most ethical failures in automated systems are not dramatic. They are ordinary decisions made quickly, by busy people, and never revisited.
The twelve questions below are the ones worth asking before anyone flips the switch. They are grouped by the part of the problem they cover: data, bias, privacy, transparency, and accountability. None of them require a philosophy degree. They require one person in the room willing to slow everyone down for an hour.
Start With the Data
Every model is a compressed version of somebody's past. Ask what past, and whose.
1. Where did the data come from, and who is missing from it?
A hiring tool trained on a decade of one company's promotions learns that company's habits, not good judgment. A support classifier trained only on English tickets will quietly fail every customer who writes in Spanish. Check that you have a lawful basis for the data you use, then look at the edges of the set: small towns, unusual names, the groups that barely appear.
2. What happens when the input is wrong, empty, or strange?
Test the ugly cases before users do: a blank field, a rural address, a name with an apostrophe, a photo taken at night in bad light. Decide in advance whether the system should guess, ask a clarifying question, or hand the task to a person. A confident wrong answer is worse than an honest "I'm not sure."
3. Can someone see and correct what is stored about them?
If personal data feeds the feature, people should be able to find out what is held, fix what is wrong, and request deletion where the law allows. "The model already learned it, so we can't" is a design decision, not an act of nature. Someone chose not to build the undo button.
Bias and Real-World Harm
Bias is not a single bug you remove once. It is a set of ways a system can be right on average and wrong for the people who need it most.
4. Who could be harmed by a wrong answer, and how badly?
Sort the stakes. A bad song recommendation is annoying. A bad risk score can block a loan, a job, or a place to live. Write down the three worst realistic outcomes and the people who would carry them. If the harm lands on a group with less power to complain, lower your tolerance for error.
5. How does it perform for the groups it will actually meet?
A single accuracy number hides everything interesting. Break results down by the slices that matter for your product — language, age, region, device, whatever your users actually vary by — and look at false positives and false negatives separately. If you cannot measure a group because you have too few examples, say that out loud rather than assuming it is fine.
6. What does the worst output look like?
Spend an afternoon trying to break it. Feed it hostile prompts, gibberish, and the kind of input a frustrated person sends at 11pm. Keep a written record of what you find and who fixed it. If nobody has done this, you do not yet know what you are launching.
Privacy: What Goes In, What Leaks Out
Two questions keep most teams out of trouble here. The second is the one people skip.
7. What personal data goes in, and does it need to?
Strip anything the feature can work without: full names, exact birthdates, free-text case notes, precise location history. Aggregating or hashing is not a magic trick; if a small group can be identified from the result, you have not really anonymised anything. Ask what happens to the inputs after the answer is produced, and how long they sit around.
8. Could the output reveal something nobody meant to share?
Suggestions can expose more than queries. Prompts about baby products can imply a pregnancy. A recommendation can hint at a health condition or a money problem to whoever is looking over a shoulder. Think about the second and third person who can see the result — a manager, a partner, an advertiser — not just the person who asked.
Transparency: Can You Say What Happened?
People forgive systems that are honest about their limits. They do not forgive discovering, months later, that a machine made a decision about them.
9. Do people know they are dealing with a system, not a person?
Say so early, in plain words, before they type anything personal. Do not hide behind a friendly name and a chat bubble. If the feature writes or speaks on someone's behalf, tell the recipient too.
10. Can you explain a decision to the person affected by it?
Not the full maths — the reasons. "We flagged your order because the delivery address did not match the card" is an explanation. "The model assigned a low score" is not. If you cannot produce one short, honest sentence for the person on the receiving end, the feature is not ready.
If you cannot describe how it fails, you cannot decide whether it is worth shipping.
Accountability: Names, Triggers, and Exits
Ethics without a name and a date attached is a wish.
11. Who owns the outcome when it goes wrong?
One person, by name, plus a backup. That owner answers for drift, complaints, and the decision to keep going. Ideally it is not the same person who championed the launch, because that person has already made up their mind.
12. How do you turn it off, and who decides?
Every feature needs a switch that works in minutes, a written threshold for using it, and a route for users to appeal to a human. Set the trigger now — complaint volume, error rate, a single serious incident — while nobody is feeling defensive about the work.
Putting the Checklist to Work
Answering twelve questions in a document nobody reads changes nothing. Give them teeth:
- Assign an owner and an independent reviewer before the build starts, not after the demo.
- Write a one-page note answering each question in a sentence. Vague answers are the useful output — they show exactly where the work is.
- Break evaluation results into the groups that matter and store them somewhere that survives staff changes.
- Set review dates at 30, 60, and 180 days after launch, each with a named person checking real usage.
- Keep the kill switch tested and the escalation path posted where support can find it at 2am.
If your feature touches health, money, housing, employment, or legal outcomes, treat this list as a starting point and get advice from a qualified professional in your jurisdiction. The questions stay the same; the consequences and the rules do not.
None of this makes launching slower than it needs to be. It makes the next launch faster, because you will have evidence that the first one was handled with care.
Photo: RDNE Stock project / Pexels
