How to block PerplexityBot in robots.txt
If you do not want Perplexity's indexing crawler fetching your site, you can disallow PerplexityBot in robots.txt. This page gives the exact rule, clarifies it does not cover Perplexity-User (the real-time, user-triggered fetch), and stays honest about the limits of robots-based blocking.
The rule
Add this group to disallow PerplexityBot across the whole site:
User-agent: PerplexityBot Disallow: /
To block only part of the site, list specific paths instead of /, and use Allow to carve out exceptions.
What it does and does not affect
This rule targets only the PerplexityBot token, which Perplexity uses for indexing public pages for its answer engine. It does not affect Perplexity-User, the separate token Perplexity uses when a person asks it to read a specific page in real time. If you want to restrict that behaviour too, target Perplexity-User in its own group.
The rule has no effect on crawlers from other companies — each is controlled by its own token.
- Affects: PerplexityBot (indexing crawler)
- Does NOT affect: Perplexity-User (real-time user-triggered fetch)
- Does NOT affect: other companies' AI crawlers
The limits
robots.txt is honoured by compliant crawlers but cannot force compliance, and a disallow does not retroactively remove content already indexed. Treat it as a forward-looking request. Where authenticity matters, rely on Perplexity's published verification guidance rather than the user agent alone.
How it appears in analytics and logs
After adding a PerplexityBot Disallow, compliant PerplexityBot requests to blocked paths should stop. Continued requests claiming to be PerplexityBot warrant checking against Perplexity's published guidance.
Diagnostic use case
Disallow PerplexityBot site-wide or on specific paths while leaving Perplexity-User and other crawlers governed by their own tokens.
What WebmasterID can help detect
WebmasterID shows PerplexityBot crawl activity before and after the change, so you can confirm the block took effect for the compliant crawler and flag any client ignoring it.
Common mistakes
- Expecting a PerplexityBot block to also stop the Perplexity-User real-time fetch.
- Typos in the token — it must be exactly PerplexityBot.
- Assuming robots.txt removes content already indexed in the past.
Privacy and accuracy notes
Blocking PerplexityBot is a publishing-policy choice expressed in a public file. It is not a privacy or access-control mechanism.
Related pages
- PerplexityBot — Perplexity's web crawler
PerplexityBot is the crawler operated by Perplexity to index publicly available web pages for its AI answer engine. Perplexity documents the crawler and its robots.txt token. It is separate from Perplexity-User, which fetches a page in real time in response to a user's question.
- robots.txt basics: what it does and what it cannot do
robots.txt is a plain-text file at your site root that tells compliant crawlers which paths they may request. This page covers the directives, how user-agent groups are matched, and the limits that trip people up: robots.txt is advisory, it does not hide pages from search, and it is not a security boundary.
- Writing an AI crawler policy for robots.txt
An AI crawler policy is a deliberate decision about which AI-related tokens you allow and which you disallow in robots.txt. This page offers a structured way to make and document those choices, while staying realistic: robots.txt is a request to compliant crawlers, not a legal or technical guarantee.
- AI visibility analytics
Confirm AI-crawler activity before and after a robots change.
Sources and verification notes
- Perplexity — crawler documentationDocuments PerplexityBot and Perplexity-User as separate tokens.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.