How to block Yahoo Slurp in robots.txt
Slurp is Yahoo's historical web-crawler token. This page gives the robots.txt rule to disallow Slurp and explains that, because Yahoo Search has long been powered substantially by Bing, blocking Slurp may not change how Yahoo presents your site — verify what is actually crawling you first.
What Slurp is
Slurp is the long-standing robots.txt token for Yahoo's web crawler. Yahoo documents Slurp in its help pages. However, Yahoo Search has for many years drawn heavily on Bing's index, so the practical impact of blocking Slurp is smaller than it once was, and you may see little Slurp traffic in modern logs.
Before acting, check your logs: if your Yahoo-facing presence depends on Bing's crawl, then Bingbot policy is the lever that matters, not Slurp.
The rule
To disallow Slurp site-wide, target its token:
User-agent: Slurp Disallow: /
Match the stable token exactly. Because Yahoo Search relies substantially on Bing, also review your Bingbot rules if your goal is to control how Yahoo presents your site. robots.txt is a request to compliant crawlers, not enforcement.
- Token: Slurp
- Operator: Yahoo (historical crawler)
- Yahoo Search relies substantially on Bing — check Bingbot too
How it appears in analytics and logs
A request carrying the Slurp token is Yahoo's historical crawler fetching a URL. If you rarely see Slurp anymore, that is expected — much of Yahoo's crawling shifted toward Bing's infrastructure.
Diagnostic use case
Disallow Slurp if you still see the token in your logs, while understanding that Yahoo Search results are largely served via Bing, so Bingbot policy may matter more.
What WebmasterID can help detect
WebmasterID classifies Slurp by its token as a search crawler, separate from human analytics, so you can see whether the token still reaches your site at all.
Common mistakes
- Assuming blocking Slurp controls all of Yahoo when Bing serves much of it.
- Misspelling the token — it must be exactly Slurp.
- Ignoring that Slurp may rarely appear in modern logs.
Privacy and accuracy notes
Blocking Slurp is a publishing-policy choice in a public file. It involves no visitor data and is not an access-control boundary.
Related pages
- Yahoo Slurp — Yahoo's web crawler
Slurp is the web crawler historically operated by Yahoo. Modern Yahoo Search blends results from search partners rather than relying solely on its own index, so Slurp's role is narrower than it once was. Its robots.txt token is Slurp; current scope is best confirmed in Yahoo's documentation, so this entry is partially verified.
- How to control Bingbot in robots.txt
Bingbot is Microsoft's search crawler. You can target it in robots.txt with the bingbot token, but fully disallowing it typically removes your pages from Bing search over time. For load concerns, Bing offers crawl-control settings in Bing Webmaster Tools rather than relying on a blanket block.
- How to block Applebot in robots.txt
Applebot is Apple's crawler, used to power features such as Siri and Spotlight Suggestions. This page gives the robots.txt rule to disallow the Applebot token and explains that it is separate from Applebot-Extended, the token that governs AI-training use.
- Bot intelligence
See whether the Slurp token still reaches your site.
Sources and verification notes
- Yahoo — Slurp crawler helpDocuments the Slurp token and robots.txt handling.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.