Kagibot — Kagi's web crawler
Kagibot is the crawler run by Kagi, an independent paid search engine. Kagi documents the crawler, its robots.txt token, a self-identifying URL in the user agent, and published IP addresses with hostnames for verification. Notably, if no Kagibot-specific robots.txt rule exists, Kagibot follows the Googlebot directives instead.
What this means
Kagibot is the web crawler for Kagi, an independent, subscription-based search engine. Kagi publishes a documentation page describing the crawler, the robots.txt token operators use to control it, and how to verify its requests. Allowing Kagibot lets Kagi fetch your public pages for its search index; disallowing it asks Kagi's crawler to stay out.
A distinctive detail is the fallback behaviour: Kagi states that if there is no rule targeting Kagibot but there is a rule targeting Googlebot, Kagibot follows the Googlebot directives. So your Googlebot policy can govern Kagibot by default unless you set a Kagibot-specific rule.
How Kagibot identifies itself
Kagibot uses the robots.txt user-agent token Kagibot. Its user-agent string contains the Kagibot token together with a self-identifying URL pointing at Kagi's bot documentation. Match on the stable token rather than a full version string, which can change over time.
Because any client can copy a user-agent string, treat the user agent as a claim. Kagi publishes the IP addresses and associated hostnames its crawler uses, so for requests where authenticity matters you can verify the source against those published values rather than trusting the user agent alone.
- robots.txt token: Kagibot
- User agent contains the Kagibot token plus a Kagi bot URL
- Verification: Kagi publishes crawler IP addresses and hostnames
robots.txt considerations
Kagi states Kagibot respects standard robots.txt directives that target the Kagibot token. To disallow it site-wide:
User-agent: Kagibot Disallow: /
If you set no Kagibot rule, remember the Googlebot fallback: Kagibot will follow your Googlebot directives, so a Googlebot Disallow can also restrict Kagibot. To give Kagibot its own policy, target the Kagibot token explicitly. robots.txt is honoured by compliant crawlers and is not an access-control mechanism.
How it appears in analytics and logs
A request carrying the Kagibot token is Kagi's crawler fetching a URL — a bot event, not a human visit. If you have no Kagibot rule but do have Googlebot rules, Kagibot follows the Googlebot directives, so its behaviour may track your Googlebot policy.
Diagnostic use case
Confirm whether Kagibot has crawled a page, set robots.txt policy for Kagi's crawler, and verify a request claiming to be Kagibot against Kagi's published IPs before trusting it.
What WebmasterID can help detect
WebmasterID classifies Kagibot server-side as an AI/search crawler and shows its activity on the bot-intelligence and AI-visibility surfaces, so you can see Kagi crawl coverage page by page without parsing raw logs.
Common mistakes
- Forgetting that Kagibot follows Googlebot directives when no Kagibot rule exists.
- Trusting the Kagibot user agent without verifying against Kagi's published IPs.
- Counting Kagibot crawl hits as human traffic in analytics.
- Treating a robots.txt Disallow as enforcement rather than a request to compliant crawlers.
Privacy and accuracy notes
Kagibot detection uses the request user-agent and, for verification, Kagi's published IPs and hostnames. No human identity is involved — a crawler is not a person. WebmasterID records the crawl as a bot event, never attached to a visitor profile.
Frequently asked questions
- Does Kagibot follow my Googlebot robots.txt rules?
- Kagi documents that if there is no rule targeting Kagibot but there is a rule targeting Googlebot, Kagibot follows the Googlebot directives. To give Kagibot a different policy, add a rule that targets the Kagibot token specifically.
- How do I verify a request is really Kagibot?
- Kagi publishes the IP addresses and hostnames its crawler uses. Check that the source IP matches those published values; a user agent that says Kagibot but originates elsewhere is not Kagibot.
Related pages
- Verifying AI crawlers
Any client can copy a user-agent string, so a token alone is a claim, not proof. Some vendors, such as OpenAI for GPTBot, publish IP ranges or verification guidance; many do not. Verify before trusting, and never invent IP ranges to fill the gap.
- AI crawler user-agent spoofing
Any client can put GPTBot or ClaudeBot in its User-Agent header, because that header is supplied by the client and never validated by HTTP. Spoofers do this to borrow a trusted crawler's reputation or to get around rules. The defence is verifying the request's network source against the operator's published ranges, not trusting the string.
- DuckAssistBot — DuckDuckGo assist crawler
DuckAssistBot is the token DuckDuckGo uses for crawling that supports its AI assist features. DuckDuckGo documents its crawlers and robots.txt handling. Where a specific detail is not clearly documented, it is marked partially verified rather than guessed.
- Bot intelligence
Deterministic categorisation of crawlers, search bots, and automation.
Sources and verification notes
- Kagi — About KagiBotToken, self-identifying URL, Googlebot fallback, and published IPs/hostnames.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.