Botify crawler — enterprise SEO platform
Botify is an enterprise SEO platform whose crawler fetches pages to build crawl, indexability, and content analyses for large sites, often combined with log-file analysis. It is a third-party SEO tool crawler, not a search engine. Botify documents its crawler and supports robots.txt and crawl-rate controls.
What this means
Botify is an enterprise SEO platform used by large sites to analyse crawl budget, indexability, and content quality, frequently pairing its crawler with server log analysis. The crawler fetches pages on a subscriber's behalf; it does not feed a search index and does not affect rankings.
Because Botify is used on large properties, its crawls can be substantial, which is why operators usually schedule and rate-limit them.
How the Botify crawler identifies itself
Botify's crawler self-identifies with a Botify token and a self-identifying URL in its user-agent string. Because operators can configure crawl settings and the default token is less broadly documented than search bots, this entry is marked partially verified — match on the documented Botify token but confirm the current value in Botify's documentation.
The user agent is a claim that can be copied; verify where authenticity matters.
- robots.txt token: Botify's documented crawler token (verify current value)
- User agent contains a Botify-identifying URL
- An enterprise SEO crawler, not a search-engine indexer
robots.txt and crawl rate
Botify can honour robots.txt, and its crawl rate is configurable so large audits do not overload servers. To disallow the default crawler site-wide, target its token with a standard Disallow rule.
For your own audits, prefer allowing the crawler and configuring crawl speed in Botify rather than blocking it. robots.txt is a request honoured by compliant crawlers, not an access-control mechanism.
How it appears in analytics and logs
A request carrying the Botify token is the Botify platform auditing a URL on a subscriber's behalf — a bot event, not a human visit. It usually reflects a scheduled enterprise crawl and should be counted as crawl coverage, not audience.
Diagnostic use case
Identify Botify when an enterprise SEO audit runs against your site, allow it for your own crawls, and restrict or throttle it via robots.txt otherwise.
What WebmasterID can help detect
WebmasterID classifies Botify server-side as an SEO crawler and surfaces its activity on the bot-intelligence surface, separate from human analytics, so you can see enterprise audit hits without log parsing.
Common mistakes
- Counting Botify audit hits as human sessions in analytics.
- Blocking your own scheduled enterprise crawl at robots.txt by mistake.
- Assuming Botify indexes pages for a search engine — it audits them.
Privacy and accuracy notes
Botify crawler detection uses only the request user-agent. No human identity is involved. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- Lumar (DeepCrawl) — enterprise SEO crawler
Lumar (formerly DeepCrawl) is an enterprise technical-SEO platform whose crawler audits large sites for indexability, structure, and on-page health. It is a third-party SEO tool crawler, not a search engine. Lumar documents its crawler and supports robots.txt and crawl-rate controls for operators.
- Oncrawl bot — OnCrawl technical-SEO crawler
Oncrawl is a technical-SEO and log-analysis platform whose crawler fetches pages to build site-structure and on-page audits for its subscribers. It is a third-party SEO tool crawler, not a search engine. Oncrawl documents the crawler and provides robots.txt guidance for operators who want to identify or restrict it.
- Managing third-party SEO crawler load
Third-party SEO crawlers such as AhrefsBot and SemrushBot can generate significant request volume without contributing to search visibility. You can manage their load by targeting their tokens in robots.txt, using crawl-delay where the crawler supports it, and blocking those that bring no value to you.
- Website observability
See crawler and audit-bot activity against your pages over time.
Sources and verification notes
- Botify — documentationBotify documents its crawler; verify current default token in its docs.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.