How Allow and Disallow precedence works
When robots.txt has both Allow and Disallow rules that match a URL, the outcome depends on rule precedence. This page explains Google's most-specific-match rule, how length decides conflicts, and the tie-break when rules are equally specific.
Most specific rule wins
Google resolves conflicts by choosing the rule with the longest matching path (counting wildcards toward length). The more specific rule wins regardless of whether it is the Allow or the Disallow.
Example — allow one subfolder inside a blocked directory:
User-agent: * Disallow: /admin/ Allow: /admin/public/
For /admin/public/page, Allow: /admin/public/ (longer match) wins, so the URL is crawlable, while the rest of /admin/ stays blocked.
- Longest matching path wins, Allow or Disallow
- Wildcards count toward the match length
- Order of lines does not decide the winner in Google
Ties and engine differences
When an Allow and a Disallow match with equal length, Google uses the least restrictive rule — meaning the Allow wins the tie. Different crawlers can differ here: not every engine implements the same precedence, and some older parsers were order-sensitive.
Because of these differences, write rules that are unambiguous: prefer clearly more-specific Allow exceptions over relying on tie-break behavior, and test with a robots.txt tester for the engines you care about.
How it appears in analytics and logs
If a URL you expected to be blocked is still crawled (or vice versa), an Allow/Disallow precedence conflict is a common cause — a more specific competing rule won.
Diagnostic use case
Predict exactly whether a URL is crawlable when overlapping Allow and Disallow rules apply — for example allowing one subfolder inside an otherwise blocked directory.
What WebmasterID can help detect
WebmasterID shows which URLs crawlers actually fetch, helping you catch cases where an Allow/Disallow conflict resolved differently from what you intended.
Common mistakes
- Assuming line order decides the winner in Google — specificity does.
- Relying on tie-break behavior that differs between crawlers.
- Writing an Allow that is shorter than the Disallow it was meant to override.
Privacy and accuracy notes
Precedence rules govern crawling of your own URLs, not visitor data. They do not enforce access; robots.txt remains a request to compliant crawlers.
Related pages
- robots.txt path matching and case sensitivity
robots.txt path rules are compared against the URL path, and that comparison is case-sensitive: /Page and /page are different. This page covers how Google matches paths, why case and encoding matter, and how trailing characters and wildcards change the rule that applies.
- Wildcards and path matching in robots.txt
Although the original protocol used simple prefix matching, major crawlers support two wildcards in path rules: * matches any sequence of characters, and $ anchors the end of the URL. This page covers how they behave, useful patterns, and the mistakes that make a rule too broad.
- How to test your robots.txt
A robots.txt rule is only useful if it does what you think. This page covers how to test it — checking the live file, using Google Search Console's robots.txt report and URL Inspection, and confirming in your own logs that the intended crawlers are or are not fetching the affected URLs.
- WebmasterID docs
Verify which URLs crawlers reach under your robots.txt rules.
Sources and verification notes
- Google — How Google interprets robots.txt (order of precedence)Documents most-specific-match precedence and the equal-length tie-break.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.