How to block the Screaming Frog SEO Spider
Screaming Frog SEO Spider is a desktop crawling tool used for SEO audits. This page shows how its default user agent can be addressed in robots.txt, why a configurable tool may bypass it, and when blocking actually makes sense.
robots.txt rule and its limits
By default Screaming Frog respects robots.txt and uses a self-identifying user agent. To address that default token:
User-agent: Screaming Frog SEO Spider Disallow: /
However, Screaming Frog is a configurable desktop tool: an operator can switch it to ignore robots.txt or spoof a different user agent. So a Disallow is a signal to cooperative use, not a guarantee. For genuine access control, use authentication, not robots.txt.
- Default agent respects robots.txt
- Operators can disable that or change the user agent
- robots.txt is not an access-control mechanism
When blocking is worth it
Blocking the default agent discourages casual, cooperative audits and keeps tidy logs, which is reasonable. But because the tool is operator-controlled, do not rely on robots.txt to stop a determined crawl.
If unwanted crawling is a real problem, combine server-side controls (rate limiting, authentication) with monitoring. robots.txt handles etiquette; infrastructure handles enforcement.
How it appears in analytics and logs
A burst of hits from the Screaming Frog default user agent usually means someone is running an SEO crawl of your site — your own team or a third party — not organic traffic.
Diagnostic use case
Discourage uninvited Screaming Frog audits of your site, while understanding that the operator can change the user agent or ignore robots.txt by configuration.
What WebmasterID can help detect
WebmasterID classifies Screaming Frog's default agent as an SEO crawler, so audit bursts are separated from human analytics and visible as tooling activity.
Common mistakes
- Assuming a Disallow stops every Screaming Frog crawl regardless of configuration.
- Using robots.txt as security instead of authentication.
- Counting an audit burst as a traffic spike.
Privacy and accuracy notes
The rule matches the Screaming Frog default token. It concerns a tool, not a person, and robots.txt is a request that a configurable desktop crawler may be set to ignore.
Related pages
- Screaming Frog SEO Spider
Screaming Frog SEO Spider is a desktop application that site owners and SEO professionals run themselves to audit a site. It is not a public, continuously operating crawler like Googlebot; its user agent is user-controlled and its crawling is initiated by whoever runs the tool.
- How to block SemrushBot in robots.txt
SemrushBot is the crawler Semrush uses to build its SEO datasets. Semrush documents several specialised sub-bots under related tokens, so this page covers the base disallow rule and explains why you may need to target multiple tokens to cover the activity you care about.
- Monitoring robots.txt for changes and errors
robots.txt is a single file that can accidentally block an entire site. This page explains why monitoring it matters, which failure modes to watch (Disallow: /, 404, 5xx, unexpected diffs), and how crawl-behavior signals confirm a problem.
- Bot intelligence
Spot Screaming Frog audit bursts apart from real visitors.
Sources and verification notes
- Screaming Frog — SEO Spider user agent and robots.txt configurationDocuments the default user agent and configurable robots.txt behavior.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.