What Are Good Bots?


Good BotsGood bots are beneficial to businesses as well as individuals. When you search for a website or phrases related to a website's products or services, you get relevant results listed in the search results page. This is made possible with the help of search engine spider bots, also known as crawler bots (such as GoogleBot, Bingbot, and Baidu Spider, to name a few).

Good bots are generally deployed by reputable companies, and for the most part they respect rules created by webmasters to regulate their crawling activity and indexing rate, which can be defined in a website’s robots.txt file for crawlers to see. Certain crawlers can also be prevented from indexing websites if they are not useful or needed by the business. For example, the Baidu crawler can be blocked if a business does not operate in China and/ or does not cater to the Chinese market.

Apart from search engine crawlers, good bots also include partner bots (e.g. Slackbot), social network bots (e.g. Facebook Bot), website monitoring bots (such as Pingdom), backlink checker bots (e.g. SEMRushBot), aggregator bots (like Feedly), and more. Even good bots such as crawlers can cause problems at certain times, such as when their traffic increases and starts to reach the limits of server capacity, or when their volumes result in skewed analytics.

Data Sheets
Prevent Automated Attacks on Websites, Mobile Apps and APIs

Prevent Automated Attacks on Websites, Mobile Apps and APIs

Radware’s Bot Manager integrates with Radware’s Attack Mitigation Solution (AMS) to offer the industry’s most advanced protection from sophisticated, automated attacks.

Read more
Research
Inside Good Bots

Inside Good Bots

This report outlines the prevalence of good bots and their various types, primary functions and traffic distribution across key industries

Read more
Research
Why Bot Mitigation Is Crucial For GDPR Compliance

Why Bot Mitigation Is Crucial For GDPR Compliance

In light of the severe penalties, brand damage, and loss of trust that can potentially result from GDPR violations, this report outlines why bot mitigation is now crucial to GDRP compliance.

Read more

Radware’s Bad Bot Analyzer

Is Your Website Secure Against Bot Attacks? Find Out Now

Already a Customer?

We’re ready to help, whether you need support, additional services, or answers to your questions about our products and solutions.

Locations
Get Answers Now from KnowledgeBase
Get Free Online Product Training
Engage with Radware Technical Support
Join the Radware Customer Program

Get Social

Connect with experts and join the conversation about Radware technologies.

Radware Blog
Security Research Center