Sitechecker crawler
Sitechecker is an SEO platform that crawls a website to run technical audits, monitor on-page health, and track changes over time. Its crawler fetches a site's pages to check status codes, metadata, links, and other on-page signals for its subscribers. It is an SEO audit crawler, typically invoked on a user's own verified site, rather than a search engine building a public index.
What this means
Sitechecker runs SEO audits and ongoing monitoring, crawling a website to find technical issues — broken links, status-code problems, missing metadata, and on-page errors — and tracking how they change over time.
Unlike a search engine, Sitechecker generally crawls a site its user controls or has added for monitoring. The result populates audit reports rather than a public index.
How it identifies itself
Sitechecker crawling carries a Sitechecker-identifying user-agent. Because its exact public token and IP ranges are not exhaustively published, this entry is marked partially verified; the Sitechecker identity and audit purpose are the reliable signals.
As with any crawler, the user-agent is a claim and can be copied. If you did not expect an audit, corroborate with whoever manages your SEO tooling.
- Operator: Sitechecker (SEO audit and monitoring)
- Scope: a site's own pages — status codes, metadata, links
- Typically user-initiated on a monitored site
robots.txt considerations
Sitechecker audits can usually be configured to honour robots.txt, and audit crawlers often offer a setting to ignore it when auditing your own site. To express a preference for the public crawler, target its documented user-agent token in robots.txt.
robots.txt is honoured by compliant crawlers and is not an access control. For your own audits, you may intentionally allow the crawler full access.
How it appears in analytics and logs
A Sitechecker request usually means an SEO audit of your site is running, often started by you or your team. It is audit bot traffic, not a human visit and not a search-index crawl; spikes typically coincide with a scheduled or manual audit.
Diagnostic use case
Recognise Sitechecker's audit crawler in logs, distinguish a user-initiated site audit from search indexing, and confirm it against an audit you or a team member started.
What WebmasterID can help detect
WebmasterID classifies the Sitechecker crawler server-side as an SEO audit bot and surfaces it on the bot-intelligence surface, so audit crawl load stays separate from human analytics.
Common mistakes
- Mistaking a site-owner's own audit crawl for an external search engine.
- Blocking the audit crawler and then wondering why your own SEO reports are incomplete.
- Counting audit crawl hits as human sessions in analytics.
Privacy and accuracy notes
Identification uses only the request user-agent. No visitor identity is involved. WebmasterID records the fetch as a bot event, separate from human analytics, and never attaches it to a profile.
Related pages
- Ubersuggest crawler
Ubersuggest is an SEO platform, associated with Neil Patel, offering keyword research, content ideas, site audits, and backlink analysis. Its crawler fetches web pages and link data to populate those features for users. It is an SEO data crawler rather than a search engine, building datasets for its dashboards instead of a public consumer index.
- SEranking and Mangools crawlers
SEranking and Mangools are SEO software platforms whose crawlers fetch web pages, backlinks, and on-page signals to power site audits, rank tracking, and keyword research for their subscribers. They are SEO data crawlers, not search engines: they build private datasets for marketing tools rather than a public index. Both publish self-identifying crawler user-agents so operators can recognise and control them in robots.txt.
- Managing third-party SEO crawler load
Third-party SEO crawlers such as AhrefsBot and SemrushBot can generate significant request volume without contributing to search visibility. You can manage their load by targeting their tokens in robots.txt, using crawl-delay where the crawler supports it, and blocking those that bring no value to you.
- Website observability
See SEO audit crawlers reaching your site, recorded server-side.
Sources and verification notes
- Sitechecker — SEO audit and monitoringWebsite SEO audit crawler; exact token/IP ranges not exhaustively published.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.