How to control Bingbot in robots.txt
Bingbot is Microsoft's search crawler. You can target it in robots.txt with the bingbot token, but fully disallowing it typically removes your pages from Bing search over time. For load concerns, Bing offers crawl-control settings in Bing Webmaster Tools rather than relying on a blanket block.
What Bingbot is and when to block it
Bingbot is Microsoft's primary search crawler; Bing search results also power other surfaces. Because of that, fully disallowing Bingbot is rarely what you want on a production site — it can remove your pages from Bing over time, just as blocking Googlebot would for Google.
More common is restricting specific low-value paths while leaving the rest crawlable.
The rule and the better alternative for load
To disallow a path for Bingbot specifically:
User-agent: bingbot Disallow: /private/
To disallow it entirely (rarely advisable):
User-agent: bingbot Disallow: /
If your concern is crawl load rather than visibility, Microsoft provides crawl-control settings in Bing Webmaster Tools, and Bing has historically honoured the crawl-delay directive. Prefer those over a blanket block when you simply want Bingbot to crawl more gently.
- Token: bingbot
- A full Disallow can remove you from Bing search
- For load, use Bing Webmaster Tools crawl control or crawl-delay
How it appears in analytics and logs
A request carrying the bingbot token is Microsoft's search crawler. Disallowing it stops crawling of those paths, which over time can drop them from Bing results — a search-visibility trade-off, not just a load change.
Diagnostic use case
Adjust what Bingbot may crawl, or slow it, without accidentally removing your site from Bing search results.
What WebmasterID can help detect
WebmasterID shows Bingbot crawl activity separate from human traffic, so you can confirm a robots.txt change affected the crawler as intended.
Common mistakes
- Disallowing bingbot entirely on a production site and losing Bing visibility.
- Using robots.txt for load when crawl-rate tools in Bing Webmaster Tools fit better.
- Assuming a Googlebot rule covers Bingbot — each has its own token.
Privacy and accuracy notes
Managing Bingbot is a crawl and search-visibility choice in a public file. It involves no visitor data.
Related pages
- Bingbot — Microsoft Bing's web crawler
Bingbot is the crawler Microsoft Bing uses to discover and index web pages. It uses the bingbot robots.txt token and can be verified through Bing's reverse-DNS method and published IP ranges. Bing also powers results for other surfaces, so Bingbot coverage has reach beyond Bing.com.
- The crawl-delay directive in robots.txt
Crawl-delay is a non-standard robots.txt directive that asks a crawler to wait between requests. Support is uneven: Google does not use it and points to Search Console instead, while Bing and Yandex have historically honoured it. This page explains the directive and the safer alternatives.
- User-agent groups and matching in robots.txt
robots.txt rules are organised into user-agent groups. A crawler does not combine every group — it selects the single most specific group whose token matches its name, falling back to the * group only when no named group matches. Understanding this prevents rules that never apply.
- Bot intelligence
See Bingbot crawl activity separate from human traffic.
Sources and verification notes
- Bing — Bingbot and crawl controlDocuments bingbot and crawl-control options in Bing Webmaster Tools.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.