User-triggered fetchers vs crawlers
Google groups its automated agents into common crawlers, special-case crawlers, and user-triggered fetchers. User-triggered fetchers act because a person asked for something now — like reading a page aloud or fetching a preview — and are treated differently from indexing crawlers, including how they relate to robots.txt. Understanding the distinction prevents wrong robots.txt and analytics decisions.
What this means
Google documents three groups of agents: common crawlers (like Googlebot) that build the index and obey robots.txt; special-case crawlers tied to specific products; and user-triggered fetchers that run because a user asked for something right now — for example Google-Read-Aloud reading a page, or a preview fetch.
The key difference is intent and timing. Indexing crawlers decide what to fetch as part of building search; user-triggered fetchers fetch exactly what a person requested at that moment.
Why the distinction matters
Google states that user-triggered fetchers are generally not governed by robots.txt the way the indexing crawlers are, because they act on a user's direct request. So a robots.txt Disallow that stops Googlebot may not stop a user-triggered fetcher, and blocking by robots.txt is the wrong tool for that behaviour.
In analytics, user-triggered fetches are automation acting for a person, but they are still not human page views and should not inflate audience metrics. Reading the category correctly keeps both robots.txt policy and analytics honest.
- Common crawlers: index-building, obey robots.txt (e.g. Googlebot)
- Special-case crawlers: product-specific (e.g. AdSense)
- User-triggered fetchers: act on a person's immediate request
How it appears in analytics and logs
A user-triggered-fetcher token in logs means an agent acted on a person's immediate request, not an indexing crawl. Treating it as crawl coverage, or expecting robots.txt to block it like an indexing crawler, leads to wrong conclusions.
Diagnostic use case
Decide the right control for a given Google agent: a robots.txt rule for indexing crawlers behaves differently from how user-triggered fetchers are handled.
What WebmasterID can help detect
WebmasterID categorises Google's user-triggered fetchers separately from its indexing crawlers, so request-time fetches do not inflate search-crawl coverage or human analytics.
Common mistakes
- Expecting robots.txt to block user-triggered fetchers like indexing crawlers.
- Counting user-triggered fetches as human page views.
- Lumping all Google agents together as Googlebot.
Privacy and accuracy notes
These agents are identified by user-agent token only. Even though a person triggered the fetch, no visitor identity is exposed in the request; WebmasterID records it as a bot/automation event, not a human profile.
Related pages
- Google-Read-Aloud fetcher
Google-Read-Aloud is a user-triggered Google fetcher that retrieves a page so Google can convert its text to speech and read it aloud to a user. It is documented among Google's user-triggered fetchers, is not the search crawler, and because it acts on a user's request it generally ignores robots.txt the way other user-triggered fetchers do.
- Feedfetcher vs APIs-Google
Feedfetcher-Google and APIs-Google are both special-case Google fetchers that are easy to confuse with each other and with Googlebot. Feedfetcher pulls RSS/Atom feeds for Google products that subscribe to them; APIs-Google delivers API-triggered messages such as push notifications. Neither is the search crawler. Google documents both in its crawlers and fetchers list.
- Search crawlers vs SEO crawlers
Search-engine crawlers like Googlebot and Bingbot build the indexes that determine search visibility. Third-party SEO crawlers like AhrefsBot and SemrushBot feed analysis tools and do not affect rankings directly. Distinguishing them matters for crawl-budget reasoning and for deciding what to allow or limit.
- Bot vs human
How automated fetches are separated from real visitors.
Sources and verification notes
- Google Search Central — Google crawlers and fetchers overviewDefines common crawlers, special-case crawlers, and user-triggered fetchers.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.