Brave Search crawler
Brave Search operates its own independent web index, and uses a dedicated crawler to gather pages for it. Brave documents the crawler and its robots.txt token. The exact token and verification specifics should be confirmed in Brave's documentation, so this entry is marked partially verified.
What this means
Brave Search is built on Brave's own independent web index rather than reselling a larger engine's results. To build that index, Brave operates a dedicated crawler. Being crawled by it affects visibility in Brave Search specifically.
Because this entry must not invent a token or full user-agent string, the exact crawler token and verification details are deferred to Brave's own documentation, where current specifics are authoritative. The stable point is that Brave runs its own crawler for its own index.
robots.txt considerations
Brave's crawler is expected to honour robots.txt as a compliant search crawler; control it by targeting the token Brave documents. The user agent is spoofable, so verify against Brave's published guidance where authenticity matters, and never fabricate the token, IP ranges, or a full user-agent string.
- Brave Search builds its own independent index
- Crawler token and verification: per Brave's documentation
- Honoured as a request, not an access-control boundary
How it appears in analytics and logs
A request from Brave's search crawler is a bot event fetching a URL for Brave's own index — not a human visit. Because Brave runs an independent index, allowing its crawler contributes to a result set that is not a syndication of the major engines.
Diagnostic use case
Recognise Brave Search crawling as feeding an independent index, and apply robots.txt policy for Brave's crawler per its documentation.
What WebmasterID can help detect
WebmasterID classifies Brave's search crawler server-side as a search crawler and shows its activity separately from human traffic, so Brave crawl coverage is visible without log parsing.
Common mistakes
- Assuming Brave Search reuses Google or Bing results — it runs its own index.
- Guessing Brave's crawler token instead of reading its documentation.
- Counting crawler hits as human visits.
Privacy and accuracy notes
Identification uses only the user agent — no human identity. WebmasterID records Brave's search crawler as a bot event, separate from human analytics.
Related pages
- MojeekBot — independent index crawler
MojeekBot is the crawler for Mojeek, a search engine notable for building and operating its own independent web index rather than reselling another engine's results. Its robots.txt token is MojeekBot. Mojeek documents the crawler, though some specifics are marked partially verified.
- DuckDuckBot — DuckDuckGo's crawler
DuckDuckBot is the crawler operated by DuckDuckGo. DuckDuckGo also draws on third-party indexes for some results, so DuckDuckBot is one part of how its results are built. DuckDuckGo documents the crawler and publishes IP addresses operators can use to verify it.
- Bot intelligence
See search-engine crawlers separated from human traffic.
Sources and verification notes
- Brave — Search documentationBrave Search help; crawler token and verification specifics to confirm here, so partially verified.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.