Have you ever gone through your server logs and discovered those aggressive spiders that your servers at crazy request rates?
Here is a quick list of known bots to be inserted into robots.txt
User-agent: AhrefsBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Majestic-12
Disallow: /
User-agent: Majestic-SEO
Disallow: /
User-agent: NCBot
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: Baiduspider
Disallow: /
or in .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
RewriteRule .* - [F]
Spider Data: User Agents
Yandex (RU)
Russian search engine Yandex features the following User Agents:
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
YandexSomething/1.0
Goo (JP)
Japanese search engine Goo features the following User Agents:
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
moget/2.0 (moget@goo.ne.jp)
Naver (KR)
Korean search engine Naver features the following User Agents:
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
Baidu (CN)
China’s number-one search engine Baidu features the following User Agents:
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Baiduspider+(+http://www.baidu.jp/spider/)
SoGou (CN)
Chinese search engine SoGou features the following User Agents:
Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
sogou spider
Sogou Pic Agent
Youdao (CN)
Chinese search engine Youdao (which also spells itself “Yodao” on occasion) features the following User Agents:
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
Majestic-SEO
Link analysis service Majestic-SEO http://www.majesticseo.com/ is using the distributed search engine Majestic-12:
Majestic-12
UA: Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
Copyscape
Copyscape Plagiarism Checker – Duplicate Content Detection Software
Site info: http://www.copyscape.com
Copyscape
User Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
IP: 212.100.254.105
Host: googlealert.com
Photo by Carlos Muza on Unsplash