Four Labs Agreed to One Robots.txt Line, and Reddit Is Putting Up a Login Wall

Three weeks ago your content had one real defense against AI crawlers: hope. Now the biggest labs say they will honor a single opt-out line, the New York Times is suing a browser, and Reddit is about to put a wall around the whole site. The ground under "who gets your content" just moved. Let me catch you up.

The deep cut

• A crawler opt-out is a policy, not a lock. The new robots.txt token tells OpenAI and Google your wish; it stops no one who ignores it.
• Gating content is now a revenue play. Reddit's logged-in users drive over 80% of its money, so the login wall pays for itself.
• Sue the middleman, not just the lab. The Times went after Brave for feeding scraped pages to Anthropic, Perplexity, and four more.

One line to tell four labs no

OpenAI, Google, Microsoft, and Anthropic agreed to honor a shared signal in your robots.txt file. One token, __usage_ai-training, tells all of them whether they can use your pages to train models. Emma Roth at The Verge reports Anthropic already supports it, while Google and Microsoft plan to next year and OpenAI says "coming months." Before this, you had to flag each company one by one.

Read the word "preference" carefully. The signal says what you want. It does not stop a spoofed crawler or a data broker who never reads your file. A rollout guide from Towards AI is blunt: "A preference signal is not a firewall."

The three switches people keep mashing into one

The trap is treating "AI crawler" as a single thing. It is three. Training preference is the new token. Search discovery is a separate bot, like OpenAI's OAI-SearchBot versus its training bot GPTBot. Traffic enforcement is a WAF or rate limit. Each needs its own control.

Mix them up and you break something you cared about. The Towards AI guide warns that a wildcard User-Agent: * group does not carry the new training meaning, so a global block won't document your training wish. It also flags a common cleanup mistake: flattening separate rules into one block and quietly changing what your site said. Give each decision an owner, serve it on every hostname, then actually fetch the live file to confirm it says what you shipped.

Reddit stops being a public library

Reddit is going to require a login to browse. Steve Huffman calls it a big change for someone whose life was "defined by the open internet," then says it is still the right call. Why now? Search referrals fell by more than a million per week in October, and the "vast majority" of unauthorized access comes through logged-out pages.

The money explains the rest. Sarah Perez at TechCrunch notes revenue per daily user in the US jumped 53% even as US users fell. Logged-in people give better ad signals and feed cleaner licensing deals, like the ones Reddit cut with OpenAI and Google. A wall that protects data and raises revenue at once is hard to argue with.

When a wish fails, the lawyers show up

The opt-out standard only works if crawlers behave. The New York Times bets some won't. It sued Brave, calling the browser a "supplier, dealer, and distributor of stolen copyrighted content" that built an index of more than 30 billion webpages and sold access to Anthropic, Perplexity, Mistral, xAI, and Cohere. The suit also targets Brave's paywall bypass feature.

Not everyone wants a clean win for publishers. Wikimedia filed a brief arguing training on your pages should count as fair use, but a chatbot handing users your text should need a license. And Meta shows how fast rules can slip: it kept delaying business terms that publishers read as a quiet grab for training rights.

Three questions for your team

• Who owns our training preference, and does our live robots.txt serve the same rule on every hostname, or only the one we remember?
• If we gate content behind a login the way Reddit did, do the ad and licensing gains cover the search traffic we lose?
• Is our content valuable enough to license directly, and would we rather sign a deal with a lab or send a letter to the middleman selling us?