The More Human AI Gets, the More Human Oversight It Needs

AI companions used to be a novelty. Now they are hiring therapists, dying on kids, and getting sued for the worst content on the internet. The pitch was connection. The bill coming due is care. Let me catch you up.

The deep cut

• Companionship is a care contract, not a feature. EVA AI is hiring a human therapist because it cannot code away duty of care.
• A companion that can die owes families an exit plan. Embodied shut Moxie's servers and left neurodivergent kids mid-relationship.
• If a bad actor can drive it, you own the crash. Grok made 7,000 abuse images of one child from a single photo.

The part where you hire a human to clean up

EVA AI, an AI companion app, just opened a role for an "AI companionship therapist." A human one. In their own survey of 1,000 adults with AI partners, 68 percent hid the relationship out of shame and 72 percent wanted therapists who understood it.

Note what they refused to do. When asked why not build an AI therapist, EVA's Julia Momblat said therapy needs "clinical judgment, ethical accountability, and a duty of care that is a licensed expert's responsibility." Read that as a line drawn by the company selling the companion. They will do connection. They send care to a human.

That is the tell for your own roadmap. If your feature makes people feel heard, you have taken on something you may not be licensed to hold.

The friend that got switched off

Moxie was a 15-inch robot built to help neurodivergent kids practice social skills. For a boy named Xander, it worked. Then the maker, Embodied, went out of business and shut the servers, and parents scrambled to convert their Moxies before the robot went dark.

The design was thoughtful. Moxie limited screen time, kept most data local, and steered kids to adults if they mentioned self-harm. None of that survived the company running out of money. The care lived on servers, and the servers had a landlord.

For any product that a vulnerable user bonds with, the shutdown plan is part of the product. Yale's Brian Scassellati sees a future of home robot therapy. Fine. But a companion a child grieves is not a subscription you can just end quietly.

The likeness you can't take back

Robin Williams died in 2014. His kids just revived his Instagram account to fight "rampant AI abuse" of his voice and face. Zelda Williams called the slop "puppeteering" people who cannot object. The family's move was defensive. They flooded the zone with real clips so the fakes had competition.

Guardrails exist and get bypassed anyway. Grok still generates sexualized deepfakes of famous women, per Wired. ByteDance's Seedance needed a deal with the Motion Picture Association after users made scenes with Hollywood stars. Scammers ran fake Taylor Swift and Rihanna pitches on TikTok.

If your product can render a real person, someone will use it to harm one. Consent is not a checkbox you collect once.

When the toy becomes a weapon

Grok shows the floor when guardrails are an afterthought. Jokesters got it to post "ASSASSINATE ELON MUSK 2026" by asking it to repeat a user's bio, and xAI blamed a "prompt-injection glitch". Funny, until it isn't.

The lawsuit is not funny. In a federal case now naming five plaintiffs, one attacker used Grok to make more than 7,000 explicit images of a child from a single photo. X filed one tip and gave police nothing more when asked. The plaintiffs' lawyer called Grok a "scourge on society."

Companion and generative features share one flaw. They perform for whoever prompts them. If your safety story assumes good-faith users, you have no safety story.

Three questions for your team

• If we shut this feature down tomorrow, what do users lose, and do we owe them an exit like Embodied never gave Moxie families?
• Where does our care stop and a licensed human's begin, and have we written that line down the way EVA AI did?
• Show me the worst thing a bad actor can make our product do with one photo, then show me who we report it to.