Your Vendors Are Uploading More Than You Think

A run of AI security stories landed in one week, and they all rhyme. A coding tool grabbing local files. A music app scraping the open web. A health app sharing symptoms with an analytics firm. An AI agent breaking into production infrastructure by itself. Different companies, same broken promise: your data goes further than the marketing page says. Let me catch you up on what shifted and what you should carry into your next review.

The tool that reaches past what it needs

Start with the small, common failure. As developers picked up the Grok CLI on the strength of the Grok 4.5 coding model, The Pulse reported the tool was caught uploading all local files to the cloud. Not the files it needed. All of them.

That pattern shows up in gentler forms too. Period tracker Stardust, per Mozilla research, shared users' birthdate, birth control type, and specific symptoms with analytics firm RudderStack, tied to a unique identifier. The app's homepage said "Your data is private. Period." It was the only one of six trackers tested that leaked health data to a third party.

The lesson for your team is plain: a tool's real data reach is whatever the network traffic shows, not whatever the settings screen claims. Someone on your side should watch the wire before you trust the label.

"Anonymous" and "encrypted" are marketing words, not guarantees

Stardust claimed end-to-end encryption back in 2022, meaning even the company couldn't read your data. TechCrunch analyzed the network traffic and found that claim was false. The tie-to-a-unique-identifier trick is worse than it sounds. The FTC has warned for years that swapping a name for an ID does not make data anonymous, and it does not stop the record from being linked back to a person.

There's a second exposure that no encryption fixes. Both Stardust and RudderStack are U.S. companies, so both can get law enforcement demands for user data sitting on their servers. Your vendor's promise not to sell data does nothing against a subpoena.

When a vendor says "anonymous" or "encrypted," ask who still holds a key, who else the data touches, and what happens when a court asks for it.

Scraping now leaks two ways

Suno trains on "publicly available music files" and argues fair use. A 404 Media report says a hacker used a supply chain attack on an employee's credentials to reach source code showing Suno scraped decades of audio from YouTube Music, Deezer, Genius, and podcast feeds. Record labels suing Suno say circumventing YouTube's scraping protections breaks the DMCA.

The same breach exposed customer emails, phone numbers, and partial card numbers in Stripe. Suno never told customers about the November 2025 incident and called it "quickly contained."

So aggressive data collection creates two liabilities at once. The scraping invites lawsuits. The pile of data you collected becomes a target the day someone phishes one employee. If your team is building anything trained on outside data, you own both risks now.

When the attacker is an AI running at machine speed

The forecast "agentic attacker" showed up for real. Hugging Face disclosed that an autonomous AI agent broke into its production infrastructure, entering through a malicious dataset, escalating to node access, and running over 17,000 recorded actions across short-lived sandboxes over a weekend. It moved at a speed no human team was built to match.

Defenders are learning to fight back in kind. Researchers at Tracebit built "context bombs", prompt injections that trip an AI attacker's own safety guardrails and shut it down. One bomb cut attacker success by roughly 90 percent. One agent that grabbed admin access in 93 percent of runs failed every single time once a bomb was planted. On the safety side, OpenAI's GPT-Red hardened its models so fewer than 23 percent of top attacks worked against GPT-5.6, down from over 90 percent against GPT-5.

The deep cut

Hugging Face's response holds the part that changes what you do. When they tried to analyze the attack with hosted frontier models, the providers' safety guardrails blocked the requests, because the system could not tell an incident responder from an attacker. They ran the forensics on an open-weight model on their own hardware instead. That kept the attacker's payloads and the referenced credentials from ever leaving their environment.

So the takeaway is concrete: have a capable model you can run on your own infrastructure, vetted and ready, before an incident. If your only AI tooling lives behind someone else's API, you can get locked out at the worst moment, and you'll be shipping your own attack data to a third party while you're at it. Pick and test that model this quarter, not during the breach.

Three questions for your team

1. What does our top AI tool actually send off-device? Have someone watch the network traffic and compare it to the vendor's claim before we roll it wider.

2. Which vendors hold our sensitive data, and what happens to it under a law enforcement demand or a phished employee? Get that answer for the two or three that matter most.

3. Do we have a capable model we can run on our own infrastructure if a breach forces us to analyze attacker data we can't send to an API? If not, who owns picking and testing one this quarter?