How often AI crawlers revisit pages
AI crawlers revisit pages on their own schedules, influenced by perceived importance, update frequency, and each operator's budget. There is no fixed interval, and it differs per crawler. Reading recrawl recency from logs tells you how current each AI system's view of a page is — and stale recency on important pages is a coverage signal worth acting on.
No fixed interval
There is no universal recrawl schedule. Each operator decides how often to return based on signals like how important a URL seems, how often it changes, and how much budget the crawler has for your site. Two AI crawlers can revisit the same page on very different cadences.
Because operators rarely publish exact intervals, this entry marks the specifics partially verified: describe the drivers, not invented numbers. Do not claim a crawler revisits 'every N days' unless the operator states it.
What influences frequency
Pages that change frequently and are well linked tend to be recrawled more often, because crawlers infer they are worth rechecking. Pages that rarely change, are deep in the structure, or return errors tend to be revisited less. Sitemaps with accurate lastmod and clean internal linking can help a crawler prioritise fresh content.
Server behaviour matters too: persistent errors or very slow responses can cause a crawler to back off and recrawl less, while reliable, cacheable responses support steadier revisiting.
- Update frequency and link prominence raise recrawl likelihood
- Deep, static, or error-prone pages are revisited less
- Accurate sitemap lastmod helps crawlers prioritise fresh URLs
Reading recency, not guessing intervals
The useful measurement is recency per token and URL: when did this crawler last fetch this page. Compare that against when the content last changed. A page updated last week but last fetched by a token six months ago is stale in that AI system's view.
Act on stale-but-important pages — strengthen internal links, update the sitemap, ensure clean responses — rather than trying to predict an exact recrawl date you cannot know.
How it appears in analytics and logs
If an important page's most recent fetch by a token is months old, that AI system's view of it may be outdated. Frequent recrawls of pages that change often indicate the crawler is tracking your updates; long gaps suggest it is not.
Diagnostic use case
Understand how fresh each AI crawler's copy of your content is by tracking recrawl recency per token, so you can spot pages that AI systems last saw long ago.
What WebmasterID can help detect
WebmasterID keeps the most recent fetch per token and URL, so recrawl recency is visible per page on the AI-visibility surface without parsing logs.
Common mistakes
- Assuming a fixed recrawl interval that operators do not publish.
- Inventing 'every N days' figures for a crawler's cadence.
- Ignoring stale recency on important, frequently changing pages.
- Expecting errors and slow responses not to reduce recrawl frequency.
Privacy and accuracy notes
Recrawl analysis uses crawler tokens, paths, and timestamps only. No visitor identity is involved, and a crawler is not a person.
Related pages
- Measuring AI crawl coverage
AI crawl coverage is the share of your important URLs that declared AI crawlers have actually fetched in a window. Measuring it means joining a list of crawl-worthy pages to observed bot requests by token, then looking at which URLs were reached, how recently, and which were missed. It is a server-side measurement built from request logs, not from human analytics.
- AI crawl budget and server load
Each AI crawler spends a finite budget on your site and consumes real origin resources per request. Inefficient URL structures, parameter explosions, and uncacheable dynamic pages waste that budget and amplify load. Reducing wasted fetches lets the budget reach your important content while keeping CPU, database, and bandwidth use sustainable.
- AI visibility analytics
Track how recently each AI crawler last fetched your pages.
Sources and verification notes
- Google — crawl budget management for large sitesDocuments factors influencing crawl frequency; AI operators rarely publish exact intervals.
- Sitemaps protocol — lastmodAccurate lastmod helps crawlers prioritise recrawling changed URLs.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.