How Nexcess limits bad bots
Article Number: 1582 | Rating: 5/5 from 1 votes | Last Updated: Fri, Apr 26, 2019 at 11:00 AM
How Nexcess limits bad bots
What are bots, and why do I care?
Over half of all web traffic is caused by web robots, commonly known as bots. Also known as “spiders” or “crawlers,” these automated scripts crawl virtually every page on every site on the Internet to gather as much data as they can.
Good bots benefit your site and do not noticeably affect its performance. Typical examples include commercial crawlers, search engine crawlers, monitoring bots, and feed fetchers, but any of these can qualify as bad bots if they hog your system resources and degrade site performance.
Bad bots slow down or even crash your site. Some are well-intentioned but grossly inefficient. Many are malicious and even attempt to impersonate legitimate human traffic. They may scrape your site for email addresses (spambots), pull content to use elsewhere without your permission, or perform other actions harmful to your site and its visitors.
How we limit bad bots
One traditional way of limiting bots involves editing your site’s robots.txt file, which theoretically sets rules for all bots to follow. However, one prominent characteristic of bad bots is they ignore this rule, making it unreliable.
For our clients, our default solution is to brand each bot with one of three labels: whitelist; graylist, or blacklist. We do not block or limit known good bots; only bots known to be abusive, malicious, or of no meaningful value are added to our graylist or blacklist.
We can tailor these lists as needed. If we are blocking a bot that you need for legitimate purposes, or have identified a whitelisted bot causing excessive traffic or other issues, please contact our 24/7 support team for assistance.
Identifying graylisted and blacklisted bots in your logs
In your Apache transfer logs, graylisted bot requests return HTTP code 429, and blacklisted bots return HTP code 400.
For 24-hour assistance any day of the year, contact our Support Team by email or through the Client Portal.
There are no attachments for this article.
How to activate two-factor authentication in SiteWorx
Added on Fri, Dec 14, 2018
How to set maximum execution time
Added on Tue, Jan 6, 2015
Added on Tue, Aug 6, 2013
How to password-protect web pages with .htpasswd
Added on Wed, Dec 26, 2018
How to change your Client Portal password
Added on Mon, May 21, 2018
How to protect your Magento store from the Credit Card Hijack exploit
Added on Tue, Jan 29, 2019
How can I prevent my site from being compromised?
Added on Mon, Mar 18, 2019
How to send secure data to Nexcess support
Added on Tue, Aug 20, 2019
How to add SSH keys to your Nexcess Cloud account
Added on Tue, Oct 23, 2018
How to use the Nexcess Secure Password Generator
Added on Tue, Jan 15, 2019