robots.txt and JavaScript/CSS files
Google renders pages with a headless browser before indexing, so it must fetch the JavaScript and CSS your page depends on. Disallowing those resources in robots.txt can prevent proper rendering and harm how the page is understood. This page explains why render-critical resources should stay crawlable.
Why JS/CSS must stay crawlable
Google indexes the rendered DOM, not just raw HTML. Its Web Rendering Service runs the page in a headless Chromium, which fetches the JavaScript and CSS the page references. If robots.txt disallows those files, Google renders the page without them — potentially missing content, layout, or links.
Google's guidance is explicit: do not block CSS, JavaScript, or image files that are needed to render the page. Blocking them can degrade how the page is understood and indexed.
- Google renders pages before indexing and needs render-critical resources
- Blocking JS/CSS can hide content and layout from the renderer
- URL Inspection in Search Console flags blocked page resources
Practical guidance
Audit any Disallow rules that touch asset directories — /static, /assets, /_next, /js, /css, /dist — and confirm they do not block files the page needs to render. Third-party scripts and APIs the page calls during rendering also need to be reachable on their own hosts.
If you must block something (for example a private admin bundle), keep it on a path the public, render-critical pages never load. Use Search Console's URL Inspection "Test live URL" and check the rendered screenshot and the list of blocked resources to confirm nothing important is hidden.
How it appears in analytics and logs
If Google cannot fetch a page's JS/CSS, the rendered version it indexes may be missing layout and content. The URL Inspection tool flagging blocked resources is a signal that a robots.txt rule is hurting rendering.
Diagnostic use case
Decide whether to block /assets, /js, or /css in robots.txt, and avoid the common mistake of hiding render-critical resources from Google's renderer.
What WebmasterID can help detect
WebmasterID records which paths crawlers fetch, so you can see whether Googlebot is reaching (or being denied) the JS and CSS bundles a page needs to render.
Common mistakes
- Disallowing /assets or /static and breaking how Google renders the page.
- Blocking a JS bundle that injects main content, leaving the rendered page empty.
- Forgetting that third-party scripts called during render must also be reachable.
Privacy and accuracy notes
This is a crawl-rendering topic about static resource files, not visitors. No personal data is involved in deciding which resource paths are crawlable.
Related pages
- robots.txt and page rendering
Google indexes the rendered version of a page, fetched in a second pass by its Web Rendering Service. robots.txt rules that block render-critical resources cause the renderer to skip them, producing an incomplete rendered DOM. This page explains the rendering pipeline and how robots.txt interacts with it.
- robots.txt and evergreen Googlebot
Googlebot runs an "evergreen" rendering engine — a regularly updated Chromium — so it can execute modern JavaScript and CSS. This raises the stakes for robots.txt: an evergreen renderer that supports your framework still cannot use resources you disallow. This page explains the implications for robots.txt on JavaScript-heavy sites.
- robots.txt common mistakes
Most robots.txt problems come from a handful of recurring mistakes. This page collects the big ones — blocking the CSS and JS crawlers need to render, trying to deindex with Disallow, advertising secret paths, and treating an advisory file as enforcement — with the correct approach for each.
- Website observability
See which resource paths crawlers reach on your site.
Sources and verification notes
- Google — robots.txt introduction (do not block resources)Google advises not blocking CSS/JS needed to render pages.
- Google — how Google renders pages (WRS)Rendering fetches page resources before indexing.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.