The mix of traffic reaching enterprise applications is changing, and a growing share of it no longer fits the assumptions on which security solutions were built on. Up until a few years ago, bot management solutions operated on a straightforward premise: every incoming request was either from a human user or an automated bot. These solutions enabled organizations to tell the two apart and respond accordingly, by admitting human users and good bots like search engine crawlers, while blocking malicious bots.
This ‘human vs bot’ premise has held less cleanly since the arrival of AI crawlers in web traffic. These crawlers that gather content at scale on behalf of AI platforms were neither clearly malicious nor simply beneficial.
More recently, the introduction of AI agents has weakened that premise to an extent that it is no longer relevant in modern web traffic. AI agents autonomously navigate websites, fill forms, transact, and execute tasks on behalf of a human user. They sit outside the ‘human vs bot’ binary, by being automated in nature but with a human user behind it. With the rising adoption of AI agents by consumers, this grey zone of web traffic is already active in a growing number of organizations today.
Why AI Agents Are Harder to Identify
AI agents and AI crawlers both fall under AI-driven traffic, but beyond this shared categorization, the two behave differently, exist for different reasons, and pose distinct challenges in identification.
AI crawlers, such as OpenAI’s GPTBot and Anthropic’s ClaudeBot, among many others, act on behalf of AI platforms – collecting, indexing, and retrieving content, for training LLMs or AI search. Many of the major AI platforms publish the user-agent strings and IP ranges their crawlers use, providing a documented way to recognize and verify them. Even so, considering the breadth of the AI crawler ecosystem, ongoing changes, and the possibility of spoofed user-agents - identifying, classifying, and managing AI crawlers is a substantial capability by itself.
AI agents, such as ChatGPT Agent, Manus, Genspark, among others, are a harder problem. They act on behalf of human users, follow human-like browsing behavior, with ordinary browser characteristics, and complete tasks largely on their own by moving through the website along paths a human user might take. Identifying AI agents and distinguishing them from human users or other bots calls for a fundamentally different set of capabilities than those built for crawlers. Without that, agentic traffic simply blends in with everything else, creating a blind spot in web traffic for organizations.
Why AI Agent Visibility Matters
The capabilities of AI agents could be put to both good or bad purposes. Genuine human users could use it for shopping, researching or completing a task - real demand and potential revenue for organizations. On the other hand, bad actors could just as easily point AI agents at malicious objectives, slipping through defenses to carry out fraud.
This is precisely why visibility matters. Organizations can only respond appropriately once they understand what makes up the web traffic actually reaching their applications. Decisions about how to treat agentic traffic, and how to tell beneficial agents apart from unwanted ones, depend on first having a clear, accurate picture of it. Without visibility into and full coverage of all types of web traffic, any policy is built on guesswork.
Also important is the growth of AI agent adoption by consumers, with agentic commerce offering a real opportunity for businesses. But to serve that traffic well, and the genuine, revenue-driving customers behind it, organizations must first understand it – which agents are present, how many requests they are sending, and which parts of the website they engage with the most.
The Visibility Gap
For organizations without a capable solution in place, the AI agents reaching their applications are effectively indistinguishable from the rest of the web traffic. Their sessions closely resemble human ones, so their requests register as normal user activity to existing defenses.
Some may have partial visibility, which is visibility into AI agents that self-identify or adhere to identification standards like Web Bot Auth - an emerging standard that lets AI agents prove their identity using cryptographic signatures. But both self-identification and standards-based cryptographic verification are voluntary. Any approach that depends on them sees only the agents that choose to be seen and remains blind to the larger share of AI agents that opt out of both.
How Radware Provides AI Agent Visibility
The Radware Bot Manager’s advanced behavioral, multi-layered detection capabilities enable real-time identification and classification of AI agents beyond those that self-identify or adhere to emerging standards like Web Bot Auth. This means visibility extends not only to the agents that choose to be seen, but also includes even the unsigned agents that do not.
The result is a clear account of the agentic traffic reaching an organization’s applications – which AI agents are present, in what volume, which routes they engage with most, along with detailed analytics on agentic activity. With this in place, agentic traffic is no longer an undifferentiated unknown sitting inside everyday user activity for organizations, but something that can be observed, understood, and reasoned about.
As AI agents become a routine part of how human users interact with the internet, organizations cannot manage, serve, or secure agentic traffic they cannot see. Visibility is the first step towards an informed strategy.
Contact us to learn more about the Radware Bot Manager and connect with our security experts.