Lumar (DeepCrawl) — enterprise SEO crawler
Lumar (formerly DeepCrawl) is an enterprise technical-SEO platform whose crawler audits large sites for indexability, structure, and on-page health. It is a third-party SEO tool crawler, not a search engine. Lumar documents its crawler and supports robots.txt and crawl-rate controls for operators.
What this means
Lumar is the platform formerly known as DeepCrawl, used by larger organisations for technical-SEO audits and site health monitoring. Its crawler fetches pages to assess indexability, internal linking, and on-page issues at scale. It does not feed a search index and does not affect rankings.
Because both the old DeepCrawl name and the Lumar name appear in the wild, logs may show either identifier; treat both as the same platform's crawler.
How the Lumar crawler identifies itself
Lumar's crawler self-identifies with a Lumar or DeepCrawl token and a self-identifying URL in its user-agent string. Because the rebrand changed naming and operators can configure crawl settings, this entry is marked partially verified — match on the documented Lumar/DeepCrawl token but confirm the current value in Lumar's documentation.
The user agent is a claim that can be copied; verify where authenticity matters.
- robots.txt token: Lumar/DeepCrawl documented token (verify current value)
- User agent contains a Lumar/DeepCrawl-identifying URL
- An enterprise technical-SEO crawler, not a search-engine indexer
robots.txt and crawl rate
Lumar can honour robots.txt, and its crawl rate is configurable so large audits do not overload servers. To disallow the default crawler site-wide, target its token with a standard Disallow rule.
For your own audits, prefer allowing the crawler and tuning crawl speed in Lumar rather than blocking it. robots.txt is a request honoured by compliant crawlers, not an access-control mechanism.
How it appears in analytics and logs
A request carrying the Lumar/DeepCrawl token is the Lumar platform auditing a URL on a subscriber's behalf — a bot event, not a human visit. It usually reflects a scheduled enterprise audit and should be counted as crawl coverage, not audience.
Diagnostic use case
Identify Lumar/DeepCrawl when an enterprise technical-SEO audit runs, allow it for your own crawls, and restrict or throttle it via robots.txt otherwise.
What WebmasterID can help detect
WebmasterID classifies Lumar/DeepCrawl server-side as an SEO crawler and surfaces its activity on the bot-intelligence surface, separate from human analytics, so you can see enterprise audit hits without log parsing.
Common mistakes
- Treating Lumar/DeepCrawl audit hits as human traffic in analytics.
- Not realising DeepCrawl and Lumar are the same platform under different names.
- Blocking your own scheduled enterprise audit at robots.txt by mistake.
Privacy and accuracy notes
Lumar crawler detection uses only the request user-agent. No human identity is involved. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- Botify crawler — enterprise SEO platform
Botify is an enterprise SEO platform whose crawler fetches pages to build crawl, indexability, and content analyses for large sites, often combined with log-file analysis. It is a third-party SEO tool crawler, not a search engine. Botify documents its crawler and supports robots.txt and crawl-rate controls.
- Oncrawl bot — OnCrawl technical-SEO crawler
Oncrawl is a technical-SEO and log-analysis platform whose crawler fetches pages to build site-structure and on-page audits for its subscribers. It is a third-party SEO tool crawler, not a search engine. Oncrawl documents the crawler and provides robots.txt guidance for operators who want to identify or restrict it.
- Managing third-party SEO crawler load
Third-party SEO crawlers such as AhrefsBot and SemrushBot can generate significant request volume without contributing to search visibility. You can manage their load by targeting their tokens in robots.txt, using crawl-delay where the crawler supports it, and blocking those that bring no value to you.
- Website observability
See crawler and audit-bot activity against your pages over time.
Sources and verification notes
- Lumar (formerly DeepCrawl) — documentationLumar documents its crawler; the DeepCrawl rebrand means logs may show either name. Verify current token.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.