For years, the fundamental question that web traffic management has been built around is a binary one: is this traffic human, or is it a bot?
Since the beginning of the internet and up until recently, this differentiation was sufficient. Human traffic was browser-based, session-driven, and behaviorally rich, while bot traffic was automated, programmatic, and operated outside the patterns of human browsing behavior. Organizations relied on this working model to separate the two types of traffic for understanding and governing what’s happening on their application. That model is no longer sufficient in the AI era, with an increasingly AI-driven internet, and it leaves a gap that is becoming larger and more consequential for organizations on the internet.
The Binary That No Longer Holds
The human vs bot framework was built for an internet in which humans and bots were the major actors, each with unique characteristics. AI-driven traffic consists of different actors entirely that don’t fit into either of these categories.
AI crawlers and AI agents, the two primary categories of AI-driven traffic now active on the internet, don’t fit cleanly into either side of the traditional binary. They are automated, but in ways that are qualitatively different from conventional good/bad bots, and in some cases indistinguishable from human activity. Applying the old framework to this new category of traffic doesn’t produce a clean differentiation but instead it produces a blind spot.
The AI Crawlers Problem
AI crawlers are the more established of the two categories, and they are already generating significant traffic volumes on the internet. Unlike traditional search engine crawlers, which operate under well-understood conventions and relatively predictable behavior, AI crawlers are diverse in purpose, origin, and behavior. Some are used for building training datasets for large language models (LLMs), systematically harvesting content at scale across millions of targets. Some are powering AI search engines, returning frequently to update their indexes. Some are retrieving real-time data to supply AI-powered services with fresh information. Each of these crawler types has a different frequency, different targets within the application, and different implications for businesses.
Some AI crawlers declare themselves through user-agent strings and published IP ranges, while some other crawlers are not that transparent and are ambiguous, often resembling legitimate traffic. Organizations without effective protection against this traffic absorb the additional infrastructure costs, proprietary data exposure, skewed metrics, and other risks, while having no basis on which to make a decision on their activity. Some examples of major AI crawlers are GPTBot from OpenAI, ClaudeBot from Anthropic, Meta-ExternalAgent from Meta etc.
AI Agents: The More Complex Problem
AI agents on the internet are not crawlers. They don’t simply traverse applications to collect content and information. They interact with applications the way a human would, navigating interfaces, filling out forms, clicking through workflows, and executing transactions. They are designed to operate in the same environment a human user operates in, using real browsers, real sessions, and behavioral patterns calibrated to function similar to human users.
This is the critical distinction. Traditional bot detection has been built and refined around a core assumption: that automated traffic will, in some detectable way, not look like human traffic. It moves too fast, or it doesn’t navigate like a human, or it lacks the environmental signals that genuine browser-based human sessions produce. Bot management systems already possess advanced capabilities to identify these sophisticated differences.
AI agents disrupt that assumption – by increasingly resembling legitimate human behavior in ways traditional models weren’t designed to analyze. Because they are designed to function in human environments, they produce many of the same signals that human traffic produces. They operate through browsers and generate realistic session behavior. They don’t move at inhuman speeds. From the perspective of conventional detection approaches, they can be highly difficult to distinguish from real users. Some examples of AI agents are ChatGPT Agent, Perplexity Comet, Manus, etc.
Managing the Third Category of Internet Traffic
The human vs bot management framework worked previously because it matched the reality of the internet at the time, when there were only two classes of actor. The problem isn’t that the framework was wrong, but that the internet has added a third class of actor, one that the existing framework wasn’t designed to accommodate.
Treating the AI-driven traffic of AI crawlers and AI agents – automated but behaviorally sophisticated, diverse in purpose, and growing rapidly in volume and complexity – as a subset of an existing traffic category or hoping that existing capabilities are enough to detect and manage them is not the right strategy for organizations.
Managing this new class of internet actor requires recognizing it as such, with dedicated capabilities, classification and controls built around what AI-driven traffic actually is, rather than retrofitting it into a framework or solution that predates it.
Think AI-driven traffic might be a blind spot for your organization? Contact us to learn more about our solution for addressing AI-driven traffic and connect with our security experts.