Amazonbot — Amazon crawler
Amazonbot is the web crawler operated by Amazon. Amazon documents the crawler, its robots.txt token, and how site owners can control it. Amazonbot honours robots.txt and identifies itself with the Amazonbot token plus a self-identifying URL.
What this means
Amazonbot is Amazon's web crawler. Amazon publishes documentation describing the crawler and the robots.txt token operators can use to control it. Allowing Amazonbot lets Amazon fetch your public pages; disallowing it asks Amazon's crawler to stay out.
Amazon's documentation is the authoritative source for what Amazonbot is used for and how it behaves, so consult it directly when setting policy.
How Amazonbot identifies itself
Amazonbot uses the robots.txt user-agent token Amazonbot. Its user-agent string contains that token together with a self-identifying URL pointing at Amazon's Amazonbot documentation. Match on the stable token rather than a full version string.
The user agent is a claim and can be copied, so for requests that must be trusted, use Amazon's published verification guidance rather than the user agent alone.
- robots.txt token: Amazonbot
- User agent contains the Amazonbot token plus an Amazon URL
- Honours robots.txt per Amazon's documentation
robots.txt considerations
Amazonbot honours robots.txt. To disallow it site-wide:
User-agent: Amazonbot Disallow: /
This targets only Amazonbot. robots.txt is a request honoured by compliant crawlers, not an access-control boundary.
How it appears in analytics and logs
A request carrying the Amazonbot token is Amazon's crawler fetching a URL — a bot event, not a human visit. Treat sustained Amazonbot activity as crawl coverage rather than audience growth.
Diagnostic use case
Confirm whether Amazonbot has crawled a page and set robots.txt policy for Amazon's crawler independently of other crawlers.
What WebmasterID can help detect
WebmasterID classifies Amazonbot server-side and surfaces its crawl activity on the bot-intelligence surface, so you can see Amazon crawl coverage per page without parsing server logs.
Common mistakes
- Trusting the Amazonbot user agent without verification where authenticity matters.
- Counting Amazonbot crawl hits as human traffic.
- Expecting robots.txt to enforce access rather than request compliance.
Privacy and accuracy notes
Detection uses only the request user-agent. No human identity is involved — a crawler is not a person. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- Bytespider — ByteDance crawler
Bytespider is a web crawler affiliated with ByteDance. Its robots.txt token is Bytespider, and it appears in server logs as an automated fetcher. Public documentation is limited, so some specifics about its purpose and behaviour are marked partially verified rather than guessed.
- GPTBot — OpenAI's web crawler
GPTBot is the crawler OpenAI uses to fetch publicly available web content that may be used to help train its foundation models. It is a declared, well-documented crawler with a stable robots.txt token, and OpenAI publishes both documentation and an IP range list so operators can identify and control it.
- Bot intelligence
Deterministic categorisation of crawlers, search bots, and automation.
Sources and verification notes
- Amazon — Amazonbot documentationDocuments Amazonbot, its token, and robots.txt handling.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.