The AI That Learns to Stereotype, and the Lawsuits Catching Up

You already knew AI could pick up human bias from its training data. Here's the new part: it can invent bias on its own, from experience, and it does it worse than people do. And while researchers were proving that in a lab, two lawsuits landed that turn all of this from an ethics slide into a legal problem your team owns. Let me catch you up.

The model that jumps to conclusions

Researchers at Princeton and the University of Chicago ran ChatGPT, Claude, and Gemini through a simulated hiring game. Each model played consultant to a fictional mayor, hiring across 20 jobs and four made-up ethnic groups. Every candidate was equally likely to succeed. The models didn't know that.

They started sorting anyway. When one Aima failed as a doctor, the model stopped hiring Aimas as doctors and pushed them toward janitor roles. On a segregation scale where 2 means total sorting, humans scored 0.84. OpenAI's o3 scored 1.83, close to the max. The models stereotyped about 65 percent harder than people.

The reason is plain. LLMs are trained to generalize from a few examples. That helps on math and code. In social settings, PhD student Ryan Liu put it simply: "that's when things tend to go wrong."

Better reasoning, worse bias

Here's the part that should worry anyone shopping for the newest model. The stronger reasoning models were the worst offenders. OpenAI's o3 and DeepSeek's R1 showed the strongest bias, not the weakest.

So the upgrade you'd buy for smarter decisions is the same upgrade that jumps to conclusions faster. And the memory features vendors keep pitching make it worse. Cornell's Angelina Wang warned that a chatbot drawing on past conversations can "over-index on the same kinds of behaviors it's experienced before." Turning memory off isn't a fix either, because users want it.

Two things actually moved the needle in the study. Telling the model to be fair did almost nothing. Paying it a bonus for diverse hiring cut bias sharply. And feeding it relevant personal facts, like age and education, beat feeding it noise like hair color. The goal you write into the system matters more than the values you preach at it.

The lawsuits are here

While the paper made the rounds, 26 former Meta employees sued the company, claiming AI ranking tools flagged them for layoffs. The suit says productivity scores, token-usage tracking, and an internal LLM called Metamate scored people down for missing work tied to medical conditions, maternity leave, and disability. Meta's Andy Stone called it "patently untrue," saying decisions "were and are made by people, not AI."

Notice the legal claim underneath. The plaintiffs argue the tools were never properly screened for bias, which they say breaks California and New York City laws. That's a process failure, not a bad-outcome complaint.

Ontario shows the same pattern with higher stakes. A class action over the SAFER classification tool alleges it pushes Black prisoners into maximum security using historical justice data. Black people are about 5 percent of Ontario's population and were close to 27 percent of maximum-security prisoners from 2022 to 2025. Internal training docs even admitted the tool "would likely contribute to the overrepresentation" of racialized people.

The deep cut

Both suits hinge on the same thing: not the harm, but whether anyone tested for it first. Ontario's own documents show they knew, and they built safeguards for Indigenous prisoners while skipping the same protection for Black prisoners. That gap is the case.

So if you're deploying an LLM for any ranking, screening, or scoring, your exposure isn't only the biased output. It's the paper trail showing you had reason to check and didn't. Write down your bias testing before launch, keep the diverse-outcome goal in the actual system prompt or scoring logic, and feed models job-relevant facts instead of noise. Those aren't nice-to-haves anymore. They're what a plaintiff's lawyer will ask you to produce.

Three questions for your team

• For every AI tool that ranks or screens people, can we produce a dated bias test we ran before it shipped?
• Where are we relying on "tell the model to be fair" instead of building a fairness goal into the scoring itself?
• Which of our models use memory or personalization, and are we feeding them decision-relevant facts or just accumulating noise that pushes them toward stereotypes?