AdSense and Mediapartners-Google in robots.txt
Mediapartners-Google is the crawler Google AdSense uses to read your pages so it can serve relevant ads. This page explains how it interacts with robots.txt, why blocking it can hurt ad targeting, and the exact rule if you have a reason to disallow it.
What Mediapartners-Google does
AdSense uses the Mediapartners-Google crawler to analyse the content of pages showing AdSense ads so it can serve contextually relevant ads. If you block it, AdSense cannot read those pages, which can reduce ad relevance on them.
This crawler is separate from Googlebot. Allowing or blocking Mediapartners-Google does not change Search indexing, and blocking Googlebot does not by itself control AdSense crawling — they are different tokens.
- Reads AdSense pages to match relevant ads
- Separate token from Googlebot Search crawling
- Blocking it can reduce AdSense ad relevance
If you must disallow it
To stop AdSense from crawling a section, target the token in its own group:
User-agent: Mediapartners-Google Disallow: /private/
Google documents that Mediapartners-Google honours its own robots.txt group. Keep the rule scoped to the paths you truly want excluded, since a site-wide Disallow degrades ad targeting everywhere AdSense runs.
How it appears in analytics and logs
Mediapartners-Google hits mean the AdSense crawler is reading a page to match ads to its content. It is not Search indexing and not a human visit.
Diagnostic use case
Decide whether to allow Mediapartners-Google on AdSense pages (for relevant ads) or disallow it on sections that should not be analysed for ad content.
What WebmasterID can help detect
WebmasterID classifies Mediapartners-Google as a Google bot distinct from Googlebot, so you can confirm AdSense crawling and keep it out of human analytics.
Common mistakes
- Blocking Mediapartners-Google site-wide and wondering why ads became less relevant.
- Assuming a Googlebot rule also governs AdSense crawling.
- Confusing AdSense crawling with Search indexing.
Privacy and accuracy notes
The rule matches the Mediapartners-Google token only. It concerns ad crawling of your own pages, not visitor data, and robots.txt is a request to compliant crawlers.
Related pages
- How to block GoogleOther
GoogleOther is a generic crawler Google uses for internal research and development fetches, separate from Googlebot. This page shows how to disallow GoogleOther in robots.txt while leaving Search indexing by Googlebot intact.
- How to opt out of Google AI with Google-Extended
Google-Extended is a robots.txt user-agent token Google provides so site owners can opt out of having their content used for certain Google AI products. Crucially, it is a standalone control: disallowing Google-Extended does not affect Googlebot crawling or your appearance in Google Search.
- User-agent groups and matching in robots.txt
robots.txt rules are organised into user-agent groups. A crawler does not combine every group — it selects the single most specific group whose token matches its name, falling back to the * group only when no named group matches. Understanding this prevents rules that never apply.
- Bot intelligence
Distinguish AdSense crawling from Googlebot and humans.
Sources and verification notes
- Google — Overview of Google crawlers (Mediapartners-Google)Documents the Mediapartners-Google AdSense crawler and its token.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.