Bedrockbot — Amazon Bedrock crawler
Bedrockbot is a crawler Amazon documents in association with Amazon Bedrock, used to retrieve web content for Bedrock features. Amazon lists it among its crawlers with a robots.txt token. It is distinct from Amazonbot; identify each by its own token and set policy separately.
What this means
Bedrockbot is a crawler Amazon documents in connection with Amazon Bedrock, its managed service for building generative AI applications. It is used to retrieve web content for Bedrock features that draw on external sources.
It is a separate crawler from Amazonbot, Amazon's general web crawler. Because the two serve different purposes and carry different tokens, a policy decision for one does not apply to the other. Consult Amazon's crawler documentation when setting rules.
How Bedrockbot identifies itself
Bedrockbot uses a robots.txt user-agent token of the form bedrockbot, and its user-agent string contains that token. Match on the stable token rather than a full version string.
Amazon's documentation is the authoritative source for the exact token form and behaviour; where a specific detail is not clearly covered, this entry leaves it unstated rather than guessing. The user agent is a claim and can be copied, so use Amazon's guidance where authenticity matters and do not invent IP ranges.
- robots.txt token: bedrockbot (Amazon Bedrock-associated)
- Distinct from Amazonbot — separate token and purpose
- User agent contains the token; match on the stable token
robots.txt considerations
To disallow Bedrockbot site-wide, target its token specifically:
User-agent: bedrockbot Disallow: /
This affects only Bedrockbot, not Amazonbot. robots.txt is a request honoured by compliant crawlers per Amazon's documentation, not an access-control boundary.
How it appears in analytics and logs
A request carrying the Bedrockbot token is Amazon's Bedrock-associated crawler fetching a URL — a bot event, not a human visit. It is separate from Amazonbot, so do not assume one rule covers both.
Diagnostic use case
Identify Bedrockbot activity in logs by its token and set robots.txt policy for Amazon's Bedrock-associated crawler independently of Amazonbot.
What WebmasterID can help detect
WebmasterID classifies Bedrockbot server-side by its token and surfaces it on the bot-intelligence surface as an AI crawler, so you can see its activity per page without parsing logs.
Common mistakes
- Assuming one rule covers both Bedrockbot and Amazonbot — separate tokens.
- Matching a full version string instead of the stable token.
- Inventing IP ranges to verify Bedrockbot.
- Counting Bedrockbot crawl hits as human traffic.
Privacy and accuracy notes
Detection uses only the request user-agent token. No human identity is involved — a crawler is not a person. WebmasterID records the crawl as a bot event, separate from human analytics.
Related pages
- Amazonbot — Amazon crawler
Amazonbot is the web crawler operated by Amazon. Amazon documents the crawler, its robots.txt token, and how site owners can control it. Amazonbot honours robots.txt and identifies itself with the Amazonbot token plus a self-identifying URL.
- Real-time AI fetcher agents
Real-time AI fetcher agents — such as ChatGPT-User, Claude-User, and Perplexity-User — retrieve a specific page live when a person asks an assistant about it. They are user-triggered, not bulk crawls, and each has its own robots.txt token controlled separately from the vendor's background crawler.
- Bot intelligence
Deterministic categorisation of crawlers, search bots, and automation.
Sources and verification notes
- Amazon — Bedrock and crawler documentationBedrockbot is associated with Amazon Bedrock; some specifics are marked partially verified.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.