How to block the Oncrawl crawler
Oncrawl is a technical-SEO platform whose crawler fetches your pages to analyse crawlability, structure, and on-page factors. This page shows the robots.txt token to target and why a Disallow steers only compliant fetchers. Note that an authorised Oncrawl audit you commission usually needs the crawler allowed.
What the Oncrawl crawler is
Oncrawl is a technical-SEO platform that combines crawling with log analysis. Its crawler fetches your public pages to evaluate structure, internal linking, and crawlability for site audits. If you are an Oncrawl customer running your own audit, you generally want this crawler allowed.
Match on the documented Oncrawl user-agent token rather than a version string. Confirm the exact self-identifying token from your logs before relying on a rule.
- Purpose: technical-SEO crawling and log analysis
- Used for site audits you may have commissioned
- Token is self-identifying; confirm it from your logs
robots.txt rule
To ask the Oncrawl crawler to stay off your site, target its documented token:
User-agent: Oncrawl Disallow: /
Only block this if you did not commission the audit — blocking your own Oncrawl crawl will produce an incomplete report. robots.txt is honoured by compliant crawlers and is not enforcement; confirm the crawler backed off via crawl behavior.
How it appears in analytics and logs
Requests carrying the Oncrawl token are SEO-audit crawl events, not human visits. If you did not initiate the audit, a third party may be analysing your site; classify the hits as bot traffic.
Diagnostic use case
Stop an Oncrawl audit crawler from fetching your pages when the audit is not authorised by you, while remembering an audit you commission needs the crawler allowed.
What WebmasterID can help detect
WebmasterID classifies the Oncrawl crawler server-side and shows whether it keeps reaching your pages after a robots.txt rule, helping you tell authorised audits from uninvited ones.
Common mistakes
- Blocking your own commissioned Oncrawl audit and getting an incomplete report.
- Guessing the token instead of confirming it from the self-identifying user agent.
- Expecting robots.txt to enforce the block rather than request compliance.
Privacy and accuracy notes
Blocking the Oncrawl crawler uses only the request user-agent token. No visitor identity is involved, and WebmasterID records the crawl as a bot event separate from human analytics.
Related pages
- Oncrawl bot — OnCrawl technical-SEO crawler
Oncrawl is a technical-SEO and log-analysis platform whose crawler fetches pages to build site-structure and on-page audits for its subscribers. It is a third-party SEO tool crawler, not a search engine. Oncrawl documents the crawler and provides robots.txt guidance for operators who want to identify or restrict it.
- How to block the JetOctopus crawler
JetOctopus is a technical-SEO crawler and log analyser that fetches your pages for site audits. This page shows the robots.txt token to target and why a Disallow steers only compliant fetchers. As with other audit tools, an audit you commission usually needs the crawler allowed.
- How to block the Screaming Frog SEO Spider
Screaming Frog SEO Spider is a desktop crawling tool used for SEO audits. This page shows how its default user agent can be addressed in robots.txt, why a configurable tool may bypass it, and when blocking actually makes sense.
- Bot intelligence
Categorise SEO-audit crawlers reaching your site.
Sources and verification notes
- Oncrawl — crawler/bot informationOncrawl SEO crawler; token matched on the self-identifying user agent.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.