In a major push to restructure the web ecosystem, Cloudflare, Inc. has introduced a comprehensive suite of tools including new crawler classifications, advanced analytics dashboards, and commercial partnerships aimed at enabling a collaborative and equitable agentic Internet. Recognizing that automated bots now account for over half of all web requests, Cloudflare’s new frameworks aim to provide content creators with robust control over how their assets are utilized by artificial intelligence, ensuring they are fairly compensated while maintaining online discoverability. A primary initiative tackles “mixed-use crawlers” that blend search and AI training, giving them a strict deadline until September 15, 2026, to separate their intents before new default settings automatically block training access on monetized pages.
Also Read: CoreWeave Launches ARIA: An Advanced AI Research Agent Built to Accelerate Model Development
Emphasizing the necessity of this shift, Matthew Prince, co-founder and CEO of Cloudflare, noted: “Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem. Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.” Furthermore, Cloudflare is launching an Attribution Business Insights dashboard for data transparency and evolving its “Pay Per Crawl” model into a “Pay Per Use” mechanism alongside partners like Ceramic.ai and You.com, creating a scalable micro-transaction infrastructure that ensures independent publishers and creators profit when their work drives AI-generated answers.
