Cloudflare Just Split the Web's AI Traffic Into Three Lanes

By Omar aka Crypt0 · Sep 16, 2026

Yesterday the default rules of the web changed for AI bots, and most site owners saw the change arrive automatically. Cloudflare's new AI traffic controls took effect on September 15, sorting every automated visitor into three lanes: Search, Agent, and Training. On pages that carry ads, the new defaults allow Search and ask Training and Agent crawlers to stay away. Cloudflare's reasoning is plain: an ad page was built for a human to see, and human attention is the business.

The three lane split is the real innovation. Search crawlers build an index so people can find a site later. Agent crawlers act on a person's behalf in real time, the kind of bot that visits a storefront to fetch product data for an AI shopping assistant. Training crawlers absorb content into a model permanently. Treating those as one group belonged to the old world. In a world where AI agents shop for people, the distinction decides whether a catalog gets read by the agent that ends up recommending it.

The sharpest edge is for mixed use crawlers. Cloudflare now judges a crawler by everything it does, across every purpose it serves. Googlebot, Applebot, and Bingbot crawl for search while also feeding training pipelines, so a site that disallows training applies the strictest rule to those bots as well. Cloudflare's argument is that letting search giants bundle both jobs into one crawler hands incumbents roughly twice the data access of rivals whose bots can be kept out separately. Separate the crawlers by purpose, the company says, or accept the stricter treatment.

Site owners who want finer control get new tools. A content use signal, carried in robots.txt, lets owners choose whether a bot may interact while retaining nothing, index and link back, or summarize and reproduce. Cloudflare is also testing BotBase, a searchable directory showing how each known bot is classified, and it has redefined the 'Verified' label: automatic access everywhere is gone for verified bots, since access now follows the bot's category. The old one click 'Block AI bots' option has become 'Disallow AI Training,' a more honest name for what it does.

For readers who run sites, the change matters right now: new domains, new sites from existing customers, and existing free tier users all moved to the new defaults, with opt outs available in Security settings. For everyone else, this is the opening chapter of a machine economy for attention. Cloudflare already runs a pay per crawl marketplace where publishers can charge bots for access. The web's original deal, crawl me and send me visitors, held for thirty years. The new deal is being written in these defaults, and for the first time the site owner holds the pen.

Sources

New to crypto? Read the crypto glossary, browse frequent questions, read our story, or explore the story archive.