In Part 1 of this series, we explored the evolving bot threat landscape in financial services, trends in ATO attacks, and how bot traffic continues to expand the attack surface. In this second blog, we focus on a fast-emerging web traffic category on financial institutions: AI crawlers.
Analysis of Q1 2026 traffic patterns shows that AI-driven crawling is scaling rapidly, driven by large AI vendors and increasingly targeting financial institutions.
A New Class of Traffic Is Surging on FSI
AI crawler activity on the set of financial organizations we observed saw a sharp increase throughout Q1 2026:
Fig 1: AI Crawlers Traffic Volume Month-over-Month in Q1'26 (selected set of organizations)
From our analysis, there was a 50%+ surge in AI crawling activity from January to February, followed by sustained high volumes in March. This reflects a broader shift tied to accelerating AI model training and adoption of AI search services. For financial institutions, this means AI crawlers are quickly becoming a meaningful share of overall automated traffic, alongside traditional bots on the internet – good bots (such as search engine crawlers), and malicious bots.
What Are AI Crawlers on FSI Doing?
Fig 2: Split of AI crawling activity by intent
The vast majority of AI crawling activity on FSI is through training crawlers that systematically collect large volumes of web content to train large language models (LLMs).
In the context of FSI, this includes product pages (loans, credit cards, insurance), market insights and financial research, regulatory disclosures and filings etc. This signals that the valuable data hosted by financial institutions is being used as source material for LLM development.
The second-largest share of AI crawling activity is from retrieval crawlers that fetch information in real-time from websites to answer user queries on AI assistants and copilots. This means financial content is being queried dynamically, not just stored – and this share will keep increasing as more users adopt AI assistants to access real-time financial information.
Within AI crawling traffic, indexing accounts for a relatively small share compared to training and retrieval – highlighting the growing emphasis on data extraction and real-time retrieval over index-building for AI search.
Who Is Driving This Traffic?
AI crawler activity in financial services is heavily concentrated among a few major players:
Fig 3: Split of AI crawling traffic by source platform
Fig 4: Split of AI crawling traffic by source platform and intent
Meta leads in training-focused crawling, reflecting aggressive large-scale dataset collection efforts – potentially aligned with ongoing model development cycles and upcoming releases. OpenAI shows a strong mix of training and retrieval activity, indicating both model development and real-time application usage.
The takeaway is clear: a handful of AI leaders are driving the majority of AI-driven crawling on FSI web properties, each with distinct patterns and objectives.
Why Financial Institutions Are Prime Targets
Financial services platforms are uniquely attractive to AI crawlers due to the high-value, structured content they host. For AI model training, financial content is highly valuable, since it is authoritative and continuously updated. From product pages to market insights and regulatory disclosures, financial institution websites offer a rich source of high-quality data for AI systems.
At the same time, this data is highly relevant across a growing number of AI-driven use cases as users increasingly leverage AI assistants for financial decision-making. In such cases, access to dependable financial data becomes a competitive differentiator.
Finally, the element of trust plays a critical role. Large language models perform better when trained on credible and accurate sources, and financial institutions are seen as authoritative providers of such data. This combination of reliability, structure, and relevance makes data from financial institutions a preferred input for generating high-quality AI outputs.
Why This Matters for FSI Organizations
Data Exposure and IP Leakage
Proprietary insights, pricing, and research content are extracted at scale and used commercially by AI platforms without attribution or control.
Infrastructure Strain and Cost Impact
High volumes of crawler traffic can increase compute and bandwidth costs, impact application performance, and lead to poor user experience.
Lack of Visibility and Control
Without proper classification, organizations struggle to distinguish beneficial crawling and preferred platforms against unwanted data extraction activity.
Compliance and Governance Concerns
For highly-regulated industries like financial services, uncontrolled data access raises regulatory and legal questions, and extraction of sensitive data can lead to severe penalties for non-compliance.
Preparing For the Rise of AI-driven Internet Traffic
AI crawlers exist in a gray zone on the internet - they could serve useful purposes to businesses but can also be exploitative at scale. Financial institutions need more granular capabilities to identify the intent behind AI crawling activity, the source platform it originates from, and the flexibility to enforce control per their business strategy.
To learn how the AI-powered capabilities of the Radware Bot Manager give financial institutions visibility and control over AI crawler activity, contact our security experts here.