Screaming Frog SEO Spider
Screaming Frog SEO Spider is a desktop application that site owners and SEO professionals run themselves to audit a site. It is not a public, continuously operating crawler like Googlebot; its user agent is user-controlled and its crawling is initiated by whoever runs the tool.
What this means
Screaming Frog SEO Spider is a downloadable desktop application, not a hosted service that crawls the web on its own schedule. It runs on a person's machine and crawls a site when that person starts it, typically to audit titles, links, redirects, and other on-page SEO factors.
Because it is operator-run, seeing it in your logs usually means a human is auditing the site — often the site owner or an SEO contractor — rather than an external search engine indexing you.
User-controlled user agent
The SEO Spider's user agent is user-configurable. By default it identifies itself with a Screaming Frog identifier, but the operator can change it to mimic other crawlers for testing. As a result, you cannot rely on the user agent alone to attribute the crawl, and there is no public IP range to verify against — the source is whoever is running the tool.
- Desktop SEO auditing tool, run by site owners themselves
- Not a public, continuously operating crawler
- User agent is user-controlled and configurable
How it appears in analytics and logs
Screaming Frog requests indicate someone is running the desktop SEO Spider against your site — frequently you or a contractor auditing it. The tool's user agent is user-configurable, so it can be set to the default Screaming Frog identifier or changed by the operator.
Diagnostic use case
Recognise that Screaming Frog hits are usually a person running an SEO audit (often yours), not an external search engine, and interpret its user-controlled user agent accordingly.
What WebmasterID can help detect
WebmasterID classifies Screaming Frog crawling server-side as SEO-tool activity and shows it separately from human traffic, so audit crawls are visible and not mistaken for real visitors.
Common mistakes
- Treating Screaming Frog crawls as external search-engine indexing.
- Counting audit-tool hits as human visits.
- Assuming a fixed IP range exists — the tool runs on the operator's machine.
Privacy and accuracy notes
Identification uses the user agent, which is operator-controlled — no end-user identity is involved. WebmasterID records such crawls as bot events, separate from human analytics.
Related pages
- AhrefsBot — Ahrefs SEO crawler
AhrefsBot is the crawler operated by Ahrefs to build its SEO and backlink index. It is a third-party crawler, not a search engine, so it does not affect Google or Bing rankings directly. It uses the AhrefsBot robots.txt token and is documented as respecting robots.txt and crawl-delay.
- SemrushBot — Semrush SEO crawler
SemrushBot is the crawler operated by Semrush to gather data for its SEO toolset. It is a third-party crawler, not a search engine, so it does not affect search rankings directly. It uses the SemrushBot robots.txt token and is documented as respecting robots.txt.
- Bot vs human
Separate automated and audit-tool crawls from real human visits.
Sources and verification notes
- Screaming Frog — SEO Spider documentationDocuments the desktop SEO Spider and its configurable user agent.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.