- Which bots to allow and which to stop
- Three signals for telling bots apart
- What robots.txt and rate limits each do
- Who gets shut out by blanket blocking
The short answer
A lot of bot traffic is not, by itself, a problem. Any site on the public internet gets automated visitors, starting with search engines. The two questions worth asking are whether your server is struggling, and whether something is reaching places it should not. Identify the type first, then treat only what needs treating.
Bots are not one thing
"Bot" covers every program that visits a site without a person driving it. The purposes vary enormously.
| Type | What it is doing | Default handling |
|---|---|---|
| Search engine crawlers | Reading pages for the search index | Allow |
| AI training and answer crawlers | Reading pages as material for models or answers | Decide a policy, then choose |
| Monitoring services | Checking at intervals that the site responds | Allow, if you signed up for it |
| Social and chat previewers | Fetching the title and image for a pasted link | Allow |
| Scrapers | Collecting prices, listings or articles in bulk | Depends on load and policy |
| Vulnerability probes | Hammering login pages and admin URLs that may not exist | Suppress |
Do not judge the whole list by its last row. Bot does not mean attack. Block search engine crawlers and you disappear from search results.
How to tell them apart
Read three signals together in your logs or analytics.
- Which URLs. Working through the homepage and article pages in order suggests collection. Nothing but login forms and admin paths that do not exist on your site suggests something else.
- The self-identification. The user agent string carries a name. It can be set to anything, so on its own it settles nothing.
- The source. Google documents that its crawlers identify themselves in three ways — the user-agent header, the source IP address and the reverse DNS hostname — and publishes a verification procedure. When you want to know whether a name is genuine, follow the operator's own documentation.
A perfectly regular pattern, such as one request per minute to the same page all night, is usually a monitoring service that you or a supplier set up. Check your list of contracted services before blocking it.
Working through the responses
robots.txt is a request, not a restriction
robots.txt tells crawlers which parts of the site you would rather they did not fetch. Google states plainly that it is not a mechanism for keeping a page out of Google, and that rules may be interpreted differently or ignored by different crawlers. If something must not be seen, put authentication in front of it rather than a line in robots.txt.
Rate limiting
This caps how many requests one source can make in a short window. For "the server has got slow", it is the most direct answer available. Set the ceiling well above what a person browsing normally produces: a page with twenty images generates more than twenty requests per view.
A WAF or CDN feature
This layer inspects the request itself, drops suspicious ones, and can treat traffic from particular countries differently. What a WAF does is covered in its own article.
Some servers and CDNs offer a single switch that handles known crawlers as a group. Before turning it on, check the provider's own documentation for what is on that list and whether you can put individual visitors back. The contents differ between services and change over time.
How this looks in practice
A restaurant taking reservations through its site saw monthly requests triple. Analytics showed no increase in actual visitors. The logs showed one source hitting the search results page every few seconds: a scraper collecting the current menu.
Three things were done. The settings that let search engines and the monitoring service through were confirmed first. A rate limit was applied to the search page only. Caching was reviewed for the hours when image requests peaked. The reservation form was left alone. Page speed returned to normal and search traffic was unchanged.
Where this goes wrong
Tighten everything at once and the first casualties are the visitors you wanted. The usual three:
- Blocking search crawlers and dropping out of search results.
- Dropping calls from a payment or shipping integration, so orders stop.
- Flagging your own bulk requests — a link checker run internally, for example — as an attack.
Before switching on a blanket "block anything that looks automated" setting, use the count-only mode if your service has one, where matching requests are logged but still served. Seeing what gets caught first is much safer than finding out afterwards.
In summary
Sort bots by type. Let search engines and your own monitoring through, judge scrapers by the load they create, and suppress traffic that is probing for weaknesses. The tools have distinct jobs too: robots.txt is a request, rate limiting handles load, a WAF inspects content. Do not react to the raw number — react to whether the site is slow or something is reaching where it should not.