Oncrawl bot — OnCrawl technical-SEO crawler
Oncrawl is a technical-SEO and log-analysis platform whose crawler fetches pages to build site-structure and on-page audits for its subscribers. It is a third-party SEO tool crawler, not a search engine. Oncrawl documents the crawler and provides robots.txt guidance for operators who want to identify or restrict it.
What this means
Oncrawl is a technical-SEO platform that combines a site crawler with log-file analysis. When a subscriber runs a crawl, Oncrawl's bot fetches pages to map structure, internal links, and on-page issues. It is an audit tool crawler, not a search-engine indexer, so it does not affect rankings.
Many Oncrawl audits are configured by the site owner or their agency, so seeing it usually means a planned audit is in progress rather than unexpected scraping.
How the Oncrawl crawler identifies itself
Oncrawl's crawler self-identifies with an Oncrawl token and a self-identifying URL in its user-agent string. Because operators can also configure a custom user agent for their own crawls and the default token is less broadly documented than search-engine bots, this entry is marked partially verified — match on the Oncrawl token but confirm against Oncrawl's current documentation, especially when a custom UA may be in use.
The user agent is a claim that can be copied; verify where authenticity matters.
- robots.txt token: Oncrawl's documented crawler token (verify current value)
- User agent contains an Oncrawl-identifying URL
- A technical-SEO audit crawler, not a search-engine indexer
robots.txt control
Oncrawl's crawler can be set to honour robots.txt, and audit tools often offer an option to ignore it for the owner's own site (since the owner controls both). To disallow the default crawler site-wide you would target its token with a standard Disallow rule.
For your own audits, a better approach than blocking is to allow the crawler and configure crawl speed in Oncrawl. robots.txt is a request honoured by compliant crawlers, not an access-control boundary.
How it appears in analytics and logs
A request carrying Oncrawl's crawler token is the Oncrawl platform auditing a URL on a subscriber's behalf — a bot event, not a human visit. It reflects a technical-SEO crawl, often configured by you or an agency, and should be counted as crawl coverage rather than audience.
Diagnostic use case
Identify Oncrawl's crawler when a technical-SEO audit runs against your site, allow it for your own audits, and restrict it via robots.txt otherwise.
What WebmasterID can help detect
WebmasterID classifies Oncrawl's crawler server-side as an SEO crawler and surfaces its activity on the bot-intelligence surface, separate from human analytics, so you can see technical-SEO audit hits without log parsing.
Common mistakes
- Counting Oncrawl audit hits as human sessions in analytics.
- Blocking your own scheduled Oncrawl audit at the robots.txt level by mistake.
- Assuming Oncrawl indexes pages for a search engine — it audits them.
Privacy and accuracy notes
Oncrawl crawler detection uses only the request user-agent. No human identity is involved. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- Lumar (DeepCrawl) — enterprise SEO crawler
Lumar (formerly DeepCrawl) is an enterprise technical-SEO platform whose crawler audits large sites for indexability, structure, and on-page health. It is a third-party SEO tool crawler, not a search engine. Lumar documents its crawler and supports robots.txt and crawl-rate controls for operators.
- Botify crawler — enterprise SEO platform
Botify is an enterprise SEO platform whose crawler fetches pages to build crawl, indexability, and content analyses for large sites, often combined with log-file analysis. It is a third-party SEO tool crawler, not a search engine. Botify documents its crawler and supports robots.txt and crawl-rate controls.
- Managing third-party SEO crawler load
Third-party SEO crawlers such as AhrefsBot and SemrushBot can generate significant request volume without contributing to search visibility. You can manage their load by targeting their tokens in robots.txt, using crawl-delay where the crawler supports it, and blocking those that bring no value to you.
- Website observability
See crawler and audit-bot activity against your pages over time.
Sources and verification notes
- Oncrawl — bot / crawler documentationOncrawl documents its crawler; default token and custom-UA option verify in current docs.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.