Your Ban Button Is Now a Product Decision

Trust-and-safety used to live in a legal deck and a support queue. Not anymore. This year, the false ban, the age gate, and the identity check all landed on the product surface, where your users actually feel them. When one bug locks out thousands of people or a new law reshapes your signup flow, that is a design problem now, and it has your name on it. Let me catch you up on what changed and what to bring to your next review.

When the safety net catches the wrong people

Discord just admitted a bug in its AI moderation system wrongfully banned more than 8,000 users over two months. The flagged content was harmless: spreadsheets, chessboards, game textures, plain gray backgrounds. The system matched uploads against a database of known bad material, saw grid-like patterns, and pulled the trigger. A human was supposed to review each flag first. The bug skipped that step and banned people outright.

The damage is real. One user, a game director, lost the account he used for all his work communication. Discord is not alone here. Meta and Tumblr both faced waves of unexplained suspensions users blamed on automation.

The lesson for your team is plain. An automated action that is hard to reverse is a product risk, not just a moderation setting. If your system can ban, mute, or delete on its own, the reversal path has to be as fast and as visible as the action itself.

The age gate that waves everyone through

Australia banned social media for kids under 16 and set fines up to $68 million. Then a team of government advisers ran the test. Software testers opened 50 accounts claiming to be 16, and not one platform asked them to prove it. Instagram, Snapchat, YouTube, and TikTok all let them in. One X account claiming to be 16 got shown pornographic content.

Most platforms lean on age inference, guessing age from behavior. It failed on fresh accounts with no history to read. The one platform that held the line, Kick, simply refused to make accounts without proof of age. Its reason was honest: it is too new to have enough data to guess.

So you get to pick your tradeoff up front. Hard age checks add friction and lose signups. Soft inference keeps the funnel smooth and fails silently. Pick on purpose, and know which one you shipped.

The law now writes part of your signup flow

The Supreme Court let Texas enforce SB 2420, which requires app stores to verify user ages and get parental consent for anyone under 18. The law says use a "commercially reasonable method" but never says what that means. It hits Apple's and Google's stores first, and it flows downstream to you.

Industry groups fought it hard. One argued that no state ever made people prove their age to read a newspaper or enter a bookstore, and now Texas does exactly that for every app on every phone. They lost, for now.

Canada, Indonesia, and the UK are lining up their own versions. Your signup and purchase flows are about to depend on rules that differ by state and country. Treat consent and age status as a first-class part of your data model, not a bolt-on you add the week before a deadline.

Every ID check is a breach waiting to happen

Here is the part that connects it all. To satisfy these laws, platforms collect more identity documents. And 2026 has been brutal on stored identity. Over the past few months, more than two million people's passport and driver license scans leaked from services like a hotel check-in system, a money transfer app, and a UK visa service. Many leaks came from basic, avoidable mistakes.

At the same time, AI made stolen identity far more dangerous. Savi Security launched an app after its founders' mother got a fake kidnapping call that spoofed a real voice and phone number. You can clone a voice from three seconds of public audio. The FTC says imposter scams cost people $3.5 billion in 2025, triple the 2020 figure.

More verification means more sensitive data sitting in your systems. Every ID you store is a target. If you collect it, you own the breach.

The deep cut

These four stories are one story. Laws push you to collect more identity data. Automated systems act on that data at scale. Both the collection and the action can fail hard, and the user feels it as a broken product, not a policy footnote.

So do one concrete thing before your next review: map every place your product acts on a user without a human in the loop, and every place it stores an ID or age proof. For each one, write down the reversal path and the blast radius if it breaks. Discord's bug ran two months because nobody owned that map. Do not let that be your team's postmortem.

Three questions for your team

1. When our system bans or restricts someone automatically, how fast can we reverse it, and who gets paged when a wave of false actions starts?
2. Which age-check approach did we actually ship, inference or hard verification, and have we tested it against fresh accounts the way Australia's advisers did?
3. For every identity document or age proof we collect to meet a law, do we have a plan for the day it leaks, and can we collect less?