Trailing slashes in robots.txt paths
In robots.txt, whether a path ends in a slash changes what it matches. Disallow: /dir/ blocks the directory and everything under it; Disallow: /dir (no slash) is a prefix that also matches /directory and /dir.html. Misreading the trailing slash is a frequent cause of rules that block too much or too little. This page makes the distinction concrete.
Directory slash versus prefix
robots.txt path matching is prefix-based: a rule matches any URL path that starts with the rule's value. The trailing slash changes the prefix.
Disallow: /dir/ matches /dir/ and everything beneath it — /dir/page, /dir/sub/ — but not /dir itself or /directory.
Disallow: /dir (no slash) is the prefix "/dir", so it matches /dir, /dir/, /dir/page, and also /directory and /dir.html, because all of those start with the string /dir.
That last part surprises people: the no-slash form can sweep in sibling paths that merely share the prefix.
- Matching is prefix-based from the start of the path
- /dir/ targets the directory and its contents only
- /dir also matches /directory and /dir.html — same prefix
Writing precise rules
To block exactly one directory and its contents, include the trailing slash: Disallow: /private/. To block a specific file, write its full path: Disallow: /private/report.pdf. When you need to anchor the end of a match — for example, only URLs ending in a particular extension — use the end-of-string wildcard:
Disallow: /*.pdf$
The dollar sign anchors the match to the end of the URL. Combine the trailing slash for directories with $ for endings, and test the result against representative URLs so a stray slash does not block or expose the wrong pages.
- Add the trailing slash to scope a rule to a directory
- Use the full path to target a single file
- Use $ to anchor matches to the end of a URL
How it appears in analytics and logs
If a robots.txt rule blocks more or fewer URLs than expected, a trailing-slash mismatch is a common cause. A rule without a trailing slash is a prefix match that can catch sibling paths you did not mean to include.
Diagnostic use case
Write Disallow and Allow rules that match exactly the paths you intend, avoiding accidental over- or under-blocking from a missing or extra slash.
What WebmasterID can help detect
WebmasterID shows which URL paths crawlers actually fetch, so you can confirm whether a trailing-slash rule is matching the set of pages you intended or catching extras.
Common mistakes
- Writing Disallow: /dir and unintentionally blocking /directory and /dir.html too.
- Omitting the trailing slash when you meant to block only a directory's contents.
- Not testing trailing-slash rules against real URLs before deploying.
Privacy and accuracy notes
Path matching concerns URLs only, never visitor identity. WebmasterID records crawler fetches governed by these rules as bot events, separate from human analytics.
Related pages
- robots.txt path matching and case sensitivity
robots.txt path rules are compared against the URL path, and that comparison is case-sensitive: /Page and /page are different. This page covers how Google matches paths, why case and encoding matter, and how trailing characters and wildcards change the rule that applies.
- Wildcards and path matching in robots.txt
Although the original protocol used simple prefix matching, major crawlers support two wildcards in path rules: * matches any sequence of characters, and $ anchors the end of the URL. This page covers how they behave, useful patterns, and the mistakes that make a rule too broad.
- How to test your robots.txt
A robots.txt rule is only useful if it does what you think. This page covers how to test it — checking the live file, using Google Search Console's robots.txt report and URL Inspection, and confirming in your own logs that the intended crawlers are or are not fetching the affected URLs.
- WebmasterID docs
Reference guides for robots.txt path matching.
Sources and verification notes
- Google — robots.txt specification (path matching)Prefix-based matching, trailing slash, and the $ end-anchor.
- Robots Exclusion Protocol (RFC 9309) — path matching
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.