How to block Bytespider in robots.txt
Bytespider is a web crawler affiliated with ByteDance. This page gives the robots.txt rule to disallow its token and is honest that, because Bytespider's documentation and robots.txt compliance are less clearly published than for major crawlers, the rule should be treated as a request rather than a guarantee.
What Bytespider is
Bytespider is a crawler affiliated with ByteDance. It appears in server logs as an automated fetcher carrying the Bytespider token. Comprehensive official documentation is limited, so claims about its exact purpose should be treated cautiously rather than asserted as fact.
The rule and its caveat
To disallow Bytespider site-wide, target its token:
User-agent: Bytespider Disallow: /
Unlike major crawlers that publish clear robots.txt-compliance statements, Bytespider's adherence is not as well documented. Treat this as a forward-looking request, watch your logs to see whether requests stop, and remember robots.txt is never an access-control boundary. Do not invent IP ranges to verify it.
- Token: Bytespider
- Affiliated with ByteDance
- Compliance less documented — verify the effect in logs
How it appears in analytics and logs
A request carrying the Bytespider token is a ByteDance-affiliated crawler fetching a URL. After a disallow, watch whether Bytespider requests actually stop — continued activity suggests the rule is not being honoured.
Diagnostic use case
Disallow Bytespider site-wide to reduce crawl activity from a ByteDance-affiliated crawler, while understanding compliance is not as clearly documented.
What WebmasterID can help detect
WebmasterID classifies Bytespider by its token, separate from human traffic, so you can see whether a disallow rule changed its activity in practice.
Common mistakes
- Assuming Bytespider definitely honours robots.txt like major crawlers do.
- Inventing IP ranges to verify Bytespider — none should be fabricated.
- Misspelling the token — it must be exactly Bytespider.
Privacy and accuracy notes
Blocking Bytespider is a publishing-policy choice in a public file. It involves no visitor data and is not access control.
Related pages
- Bytespider — ByteDance crawler
Bytespider is a web crawler affiliated with ByteDance. Its robots.txt token is Bytespider, and it appears in server logs as an automated fetcher. Public documentation is limited, so some specifics about its purpose and behaviour are marked partially verified rather than guessed.
- How to block all bots in robots.txt
A single robots.txt group can ask every compliant crawler to stay off your whole site. This page gives the exact rule and is blunt about the caveats: robots.txt is advisory rather than enforced, blocking search crawlers can remove you from results, and it is not a security boundary.
- Writing an AI crawler policy for robots.txt
An AI crawler policy is a deliberate decision about which AI-related tokens you allow and which you disallow in robots.txt. This page offers a structured way to make and document those choices, while staying realistic: robots.txt is a request to compliant crawlers, not a legal or technical guarantee.
- Bot intelligence
See whether a Bytespider disallow changed its activity.
Sources and verification notes
- ByteDance — crawler reference (token observed in logs)Token Bytespider is observed; official robots.txt-compliance docs are limited, so specifics are marked partially verified.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.