How to block the JetOctopus crawler
JetOctopus is a technical-SEO crawler and log analyser that fetches your pages for site audits. This page shows the robots.txt token to target and why a Disallow steers only compliant fetchers. As with other audit tools, an audit you commission usually needs the crawler allowed.
What the JetOctopus crawler is
JetOctopus is a technical-SEO platform that crawls sites and analyses server logs for audits of crawlability, structure, and indexation. Its crawler fetches your public pages to build those reports. If you commissioned the audit, you generally want the crawler allowed.
Match on the documented JetOctopus user-agent token rather than a version string. Confirm the exact self-identifying token from your logs before relying on a rule.
- Purpose: technical-SEO crawling and log analysis
- Used for site audits you may have commissioned
- Token is self-identifying; confirm it from your logs
robots.txt rule
To ask the JetOctopus crawler to stay off your site, target its documented token:
User-agent: JetOctopus Disallow: /
Only block this if the audit is not yours — blocking your own crawl produces an incomplete report. robots.txt is honoured by compliant crawlers and is not enforcement; confirm the crawler backed off via crawl behavior.
How it appears in analytics and logs
Requests carrying the JetOctopus token are SEO-audit crawl events, not human visits. If you did not initiate the audit, a third party may be analysing your site; classify the hits as bot traffic.
Diagnostic use case
Stop a JetOctopus audit crawler from fetching your pages when the audit is not authorised by you, while remembering an audit you commission needs the crawler allowed.
What WebmasterID can help detect
WebmasterID classifies the JetOctopus crawler server-side and shows whether it keeps reaching your pages after a robots.txt rule, helping you tell authorised audits from uninvited ones.
Common mistakes
- Blocking your own commissioned JetOctopus audit and getting an incomplete report.
- Guessing the token instead of confirming it from the self-identifying user agent.
- Expecting robots.txt to enforce the block rather than request compliance.
Privacy and accuracy notes
Blocking the JetOctopus crawler uses only the request user-agent token. No visitor identity is involved, and WebmasterID records the crawl as a bot event separate from human analytics.
Related pages
- JetOctopus crawler — technical-SEO auditor
JetOctopus is a technical-SEO platform whose crawler audits large sites for structure, indexability, and on-page issues, alongside log-file analysis. It is a third-party SEO tool crawler, not a search engine. JetOctopus documents the crawler and supports robots.txt and crawl-rate controls.
- How to block the Oncrawl crawler
Oncrawl is a technical-SEO platform whose crawler fetches your pages to analyse crawlability, structure, and on-page factors. This page shows the robots.txt token to target and why a Disallow steers only compliant fetchers. Note that an authorised Oncrawl audit you commission usually needs the crawler allowed.
- How to block the Screaming Frog SEO Spider
Screaming Frog SEO Spider is a desktop crawling tool used for SEO audits. This page shows how its default user agent can be addressed in robots.txt, why a configurable tool may bypass it, and when blocking actually makes sense.
- Bot intelligence
Categorise SEO-audit crawlers reaching your site.
Sources and verification notes
- JetOctopus — crawler/bot informationJetOctopus SEO crawler; token matched on the self-identifying user agent.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.