How to control YandexBot in robots.txt
YandexBot is the crawler for Yandex, a major search engine in Russia and nearby markets. You can target it in robots.txt with the YandexBot token. Yandex documents its robots.txt handling, has historically honoured crawl-delay, and provides additional crawl controls in Yandex.Webmaster.
What YandexBot is
YandexBot is the primary crawler for Yandex, which holds significant search share in Russia and several neighbouring markets. If your audience includes those regions, YandexBot crawling supports your visibility in Yandex search, so a full block carries a regional-visibility cost similar to blocking Googlebot for Google.
The rule and Yandex-specific controls
To restrict a path for YandexBot:
User-agent: YandexBot Disallow: /private/
Yandex documents support for standard Allow/Disallow rules and has historically honoured the crawl-delay directive for slowing the crawler. For finer control, Yandex.Webmaster offers crawl-rate and related settings. As always, robots.txt is a request to the compliant crawler, not enforcement.
- Token: YandexBot
- Historically honours crawl-delay
- Yandex.Webmaster offers additional crawl controls
How it appears in analytics and logs
A request with the YandexBot token is Yandex's search crawler. Disallowing paths stops their crawling and can reduce their presence in Yandex search, which matters most for sites serving Yandex's markets.
Diagnostic use case
Adjust what YandexBot crawls — relevant if you target Russian-language or regional audiences — without unintentionally dropping out of Yandex search.
What WebmasterID can help detect
WebmasterID classifies YandexBot as a search crawler separate from human traffic, so you can confirm a robots.txt change affected it as intended.
Common mistakes
- Blocking YandexBot while still expecting visibility in Yandex's regional markets.
- Assuming a Googlebot rule covers YandexBot — it has its own token.
- Misspelling the token — it must be exactly YandexBot.
Privacy and accuracy notes
Managing YandexBot is a crawl and search-visibility choice in a public file. It involves no visitor data.
Related pages
- YandexBot — Yandex's web crawler
YandexBot is the main crawler for Yandex, a search engine with a strong presence in Russian-language search. It uses the YandexBot robots.txt token and can be verified through reverse DNS, where the IP should resolve into a Yandex domain, confirmed by a matching forward lookup.
- The crawl-delay directive in robots.txt
Crawl-delay is a non-standard robots.txt directive that asks a crawler to wait between requests. Support is uneven: Google does not use it and points to Search Console instead, while Bing and Yandex have historically honoured it. This page explains the directive and the safer alternatives.
- User-agent groups and matching in robots.txt
robots.txt rules are organised into user-agent groups. A crawler does not combine every group — it selects the single most specific group whose token matches its name, falling back to the * group only when no named group matches. Understanding this prevents rules that never apply.
- Bot intelligence
See YandexBot crawl activity separate from human traffic.
Sources and verification notes
- Yandex — Using robots.txtDocuments YandexBot robots.txt handling and crawl-delay support.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.