Stopping scrapers without blocking Google
How to tell a crawler you want from one you do not, why user-agent checks are worse than useless, and what to do about the ones that are simply better behaved than your rules.
The problem is not scraping, it is telling them apart
Almost every site wants some automated traffic. Search engines, the preview fetchers behind chat and social links, your own monitoring, a partner's integration, an accessibility tool. Almost every site also has automated traffic it does not want: price scrapers, content mirrors, inventory bots, and increasingly AI training crawlers whose value to you is a matter of opinion.
Both arrive as HTTP requests from software. Neither will tell you the truth about itself unless it feels like it. So the whole problem reduces to classification, and every classification mistake has a cost pointing in a different direction.
Block the wrong one and you fall out of the index — a mistake that takes weeks to notice and longer to undo. Let the wrong one through and your prices are on a competitor's site within the hour. There is no setting that is safe in both directions, which is why this is worth doing carefully rather than quickly.
Why user-agent checks fail
The user agent is a string the client chooses. It is a claim, not evidence, and treating it as evidence fails in both directions at once.
Anything can claim to be Googlebot, and scrapers do — it is one line of configuration and it works against any site that checks the string. If your rule is "allow Googlebot", you have written "allow anything that says Googlebot".
Meanwhile you will block real crawlers you meant to keep, because you cannot enumerate them. There are dozens of legitimate fetchers with strings you have never seen, and a new one launches every month.
The correct verification is not a string comparison. For the major crawlers it is a reverse DNS lookup on the requesting address, confirming it resolves into the operator's own domain, followed by a forward lookup confirming that name resolves back to the same address. Several operators also publish signed IP range lists you can fetch and cache.
This is not hard, but it must be done for every crawler you care about, kept current as ranges change, and cached so you are not doing DNS lookups in the request path.
robots.txt is a sign, not a fence
robots.txt tells well-behaved crawlers what you would prefer. Search engines honour it because being caught ignoring it would cost them more than the content is worth. A scraper has no such incentive and no such reputation to protect.
Worse, it is a public document listing the paths you consider sensitive enough to mention. Disallowing /admin, /export or /api/internal announces that those exist. Write it as guidance for crawlers you trust, not as a security control.
It is still worth maintaining. A correct robots.txt with crawl-delay directives reduces the load from crawlers that do respect it, which is most of the volume on most sites, and it is the only lever you have over AI crawlers that publish an opt-out token.
What actually separates them
Ordered by how much of the answer each one gives you.
Verification for the ones that can be verified. Reverse-and-forward DNS or a published range list settles the major search engines conclusively. This should be a hard allow, and it should sit above every other rule.
Transport fingerprints for the rest. A scraper is a program: curl, a Python library, a headless browser, a Go HTTP client. Its TLS handshake announces which, through the ordered set of cipher suites and extensions summarised as JA3/JA3N. A client claiming to be Chrome on Windows while presenting a fingerprint that Chrome does not produce has told you what it is.
Behaviour over the session. Requesting resources but never executing them; a request rate that is even to the millisecond; a path through the site that follows the sitemap rather than the navigation; no mouse movement on a page that has interactive elements. None of these are conclusive alone and together they are close to it.
Reputation for first contact. A brand-new address has no history with you. If it scraped someone else last week, that is the only signal available on its very first request.
Enforcement that does not backfire
What you do about a confirmed scraper matters as much as detecting it.
Do not serve a CAPTCHA. It costs a determined scraper a fraction of a cent through a solver service and costs you the real users it also catches.
Do not block search engines by accident. Verified crawlers should be allowed by a rule that runs before any scoring, so a bug in the scoring cannot cost you your index position. Check this after every change, not before.
Consider a slow lane rather than a wall. Serving a scraper at a deliberately low rate, or from stale cache, is often better than a 403: it costs them time rather than telling them to change tactics, and it does not create the incident you get when you block something you needed.
Rate limit by the thing that is expensive. Your search endpoint and your product API deserve limits that your static pages do not.
Watch what you serve, not just what you block. A scraper that has started getting through looks exactly like a quiet week on your blocking dashboard. Track the ratio of blocked to served over time, and alert on the ratio.
Where Karma sits
Karma verifies legitimate crawlers itself and excludes them from your verdicts — and from your bill, since charging you to identify Googlebot would be charging you for the product's own job.
Everything else is scored from behavioural and transport signals, with the verdict handed to your gateway so you decide whether that means a block, a slow lane or stale cache. Your own allow and deny lists always outrank the platform, so a partner you depend on can be pinned before you turn anything on.
The free Detect plan runs the classification with enforcement off. That is the right way to start here specifically, because the expensive mistake in this area is discovering your rules were wrong from a drop in search traffic three weeks later.