Meta let fake "arrested child" videos run next to real child-abuse accounts for 24 hours

Something changed in the content you ship, and it has nothing to do with your roadmap. The machines are now making the content, signing it, and in some cases pretending it was never recorded at all. Your platform, your tools, your hardware, all of it sits on a promise that users can trust what they see and control what they share. That promise is cracking. Let me catch you up.

The deep cut

• Trust is the product, not the feature. AI slop, forged signatures, and phantom recordings all break the same thing: belief in what users see.
• Moderation you don't run becomes a policy you own. Meta left child-abuse accounts live for days and still would not say they broke the rules.
• A label is not a definition. Apple can say a camera "does not record," but text transcripts are still preserved and reviewable.

When the feed stops being real

Meta's platforms are now hosting AI-generated videos built to hack your emotions. One genre shows crying children "arrested" for not having enough likes. "You don't have enough followers and shares on your videos," an AI-generated officer tells a sobbing child, who then begs viewers to follow and share. The comments fill with real people demanding justice for a kid who does not exist.

Worse content sits right next to it. Futurism sent Meta 19 accounts loaded with AI child-abuse videos, including a baby roasting in an oven. Meta did not even reply, and the accounts stayed live 24 hours later. The captions describe the abuse in plain text the company could flag in seconds.

The point for you: a platform that stops moderating does not go neutral. It becomes an engine for the worst content, and it owns the result.

The signature was the whole point

An artist's signature is a promise too. New Yorker cartoonist Brendan Loper started getting messages about a viral Dolly Parton tribute cartoon with "BLOPER" in the corner. He never drew it. Someone made it in ChatGPT. "My name is my name," Loper told Nieman Lab. "It felt very much like a violation of my personhood."

ChatGPT forged more than a dozen New Yorker artists' signatures. Note the licensing gap: Condé Nast signed a 2024 deal with OpenAI, but the New Yorker says that deal never covered training on cartoons or using its art. A paid partnership did not stop the forgery. If your product generates or displays creative work, assume the attribution layer is a live legal and trust problem, not a footer.

Fake people, real paychecks

The same tools now build entire fake creators. A person gives an AI character a face, a name, and an account, then points the followers at brand deals or OnlyFans. The hook, per one promoter on X, is that "making them is SO easy it feels unfair."

The cost lands on real people. Comedian George Brett Olsen posted a side-by-side showing an AI character named "Tiffany" telling his joke in his outfit. "AI stole my joke and outfit," he said, and noted you can't even report it. Female comedians keep getting their material repackaged onto sexualized AI avatars. If your platform pays creators or hosts their work, your originality rules are now the only thing standing between real talent and a scraped copy.

The word that is doing a lot of work

Apple is reportedly building a home camera that does not save video. Instead AI turns what it sees into text snippets. Rumored AirPods and the Apple Watch's Audio Intelligence work the same way, and Google is weighing the same idea for smart glasses. The claim: no saved footage, so no recording.

But a text transcript is still preserved and reviewable. As Victoria Song puts it, the queasy feeling comes from "knowing a preserved snippet of you can exist in someone else's device." Calling it "not a recording" does not change what it is. If you own hardware or capture features, the definition you pick in your spec is a promise to everyone near your user, not just the user.

Smart people are already building fences. Taylor Swift's company applied for the .taylorswift top-level domain and filed to trademark her voice and face. Matthew McConaughey did the same, wanting "consent and attribution the norm in an AI world." They are not waiting for platforms to protect them.

Three questions for your team

• If we stopped moderating a surface tomorrow, what is the worst thing that would fill it, and are we fine owning that outcome?
• When our product generates or shows creative work, how does a viewer know who actually made it, and what happens when that attribution is faked?
• For any capture feature we ship, can we defend our definition of "recording" to the person standing next to our user, not just to our user?