The open web has for decades been a freely accessible space for any type of traffic, including bots that index content for search engines or that collect real-time information for virtual assistants. However, that scenario is changing radically. From September 2024, a significant portion of websites will begin to block AI agent trackers by default, i.e. those bots that act on behalf of a user who expects an immediate response. This decision, driven by one of the leading web infrastructure providers, marks a turning point for companies that have built their automation flows based on the assumption that the web would remain accessible without restrictions.
The change materializes in a new taxonomy that classifies bots into three categories: search, agent, and training. Search crawlers are those that index pages to answer queries in a deferred manner, such as those of Google or Bing. Agents are automated systems that act in real-time for a user, such as ChatGPT bots that obtain information on the fly or agents that navigate pages to complete tasks. Training systems are those that extract content to feed language models. Until now, sites could block them all with a simple switch, but from September 15 the default values will change: on pages with ads, coaches and agents will be blocked, while search pages will continue to be allowed. This affects both new domains and existing customers on the free plan, unless they opt out of the new settings.
The logic behind this measure is clear: an advertisement is the signal that a page was designed for a human to visit and consume its content. A search bot that sends traffic back to the site is considered a valid referral. But an agent who reads the page and delivers the response to a third party without generating direct visits does not add value to the publisher. That's why sites that rely on ad revenue have every right to restrict that access. For companies deploying AI agents, the problem is immediate: Pages with ads are precisely those that contain news, reviews, competitor prices, and product documentation. Blocking that access means agents will be met with silences or answers built from what little they can still achieve.
One of the most complex aspects of this transition is the difficulty of separating bots according to their behavior. Googlebot, for example, uses the same crawler for both search and training. If a site blocks training, it also blocks Googlebot, which hurts its visibility in search results. Cloudflare's CEO himself has acknowledged that the goal is to pressure big players to separate their trackers, so that publishers can make finer decisions. Until that happens, companies that manage their own agents will need to carefully analyze which Cloudflare accounts will be affected, because the classification is behavioral: it's not enough to self-declare as an agent; Blocking systems detect behavior in real time.
The solution is not to change the name of the user-agent, but to negotiate access. We are facing the birth of a pay-per-use model for web content, where AI agents must obtain licenses or commercial agreements to access pages with advertising. There are already startups that are implementing pay-per-query schemes, where publishers are compensated every time their content appears in AI-generated responses. In fact, more than half of AI crawler traffic is dedicated to re-downloading pages that haven't changed, which shows inefficiency for both agents and servers. Eliminating that waste through economic agreements benefits both parties.
For organizations that want to adapt to this new scenario, having tailored applications that manage access to external sources becomes essential. It's not just about setting up a firewall, but about designing AI systems that know when and how to get data without violating the new rules. At Q2BSTUDIO we understand that AI agents need a flexible architecture that can integrate dynamic permissions, smart caches, and licensing agreements. That's why we offer tailor-made software that allows companies to monitor the status of their sources and react to blockages in an automated way. In addition, our expertise in AWS and Azure cloud services ensures that agents run in scalable and secure environments, adapting to traffic and regulatory changes.
Cybersecurity also plays a crucial role. When negotiating privileged access to protected content, companies must ensure that their agents do not become attack vectors. Implementing robust cybersecurity in the communication between the agent and the publisher is essential to prevent data leaks or unauthorized access. At Q2BSTUDIO we integrate security practices into every layer of development, from authentication to request encryption.
On the other hand, the analysis of the data collected by agents requires business intelligence tools such as power bi, which allow consumption patterns, costs per access, and performance of sources to be visualized. Companies deploying dozens of agents need dashboards that show in real-time what content is being used and how much it costs. This becomes a fundamental input to optimize strategies for obtaining information.
The weakness of this new ecosystem lies in the taxonomy itself. AI companies are the ones that declare what kind of bot they're running, and there's an obvious incentive not to classify a tracker as training if it means being blocked. Cloudflare's infrastructure operates at the network level, detecting patterns of behavior, but it has not yet been explained how it will prevent an agent from masquerading as a search engine. Until that is resolved, publishers will have to rely on the good faith of developers or implement additional verification systems.
For companies that are building agents today, the deadline until September is an opportunity to sort out their flows. It's a good idea to audit which Cloudflare accounts they use, identify which ad-supported pages are critical to their processes, and start negotiating with publishers. Anyone waiting to receive a 403 error will find themselves rebuilding their automations on the fly. Access to the open web has been free and unlimited for thirty years, but now the bill has been broken down. Adapting to this new paradigm is not optional: it is a matter of viability for any system that relies on external information in real time.
In this context, Q2BSTUDIO is positioned as a strategic ally for companies that need to incorporate AI for companies in a safe and efficient way. Our services range from initial consulting to the development of complete agent platforms, ensuring that every interaction with the web is aligned with the new rules of the game. If your organization is evaluating how to get permission for its trackers to continue operating, we invite you to explore our artificial intelligence and automation solutions.
