Price-comparison and shopping crawlers — overview
Price-comparison and shopping crawlers fetch retailer product pages to read prices, availability, and product details for comparison platforms. This overview explains how Idealo, PriceRunner, and Shopzilla operate, why they combine crawling with structured merchant feeds, and how retailers should set policy. They build offer-comparison datasets, not a general search index, so their crawling reflects offer-refresh cadence.
What this category is
Price-comparison crawlers gather retail offers so shoppers can compare prices across merchants. Idealo is strong in Germany and Europe; PriceRunner across the Nordics and Europe; Shopzilla within the Connexity/Kit comparison network.
These platforms typically combine two data paths: structured merchant feeds that retailers submit, and crawling of product pages to read or verify prices and availability. The result is an offer-comparison dataset, not a general web search index.
How to recognise and handle them
Identify these crawlers by their documented commerce user-agents, and treat them as bot traffic separate from human shoppers. A key nuance: if you submit a merchant feed, blocking the crawler does not stop offer ingestion through the feed.
For retailers, allowing these crawlers can increase offer visibility in comparison results. robots.txt remains a request to compliant crawlers, not an access control, so manage commerce crawling through feeds and documented tokens rather than expecting enforcement.
- Purpose: offer comparison for shoppers, not general search
- Combine merchant feeds with product-page crawling
- Blocking the crawler does not stop feed-based offer ingestion
How it appears in analytics and logs
Seeing price-comparison crawlers means shopping platforms are reading your product prices and availability for comparison listings. It is commerce bot traffic, not human shoppers and not search indexing; volume tracks how often your offers are refreshed.
Diagnostic use case
Understand price-comparison and shopping crawlers as a group, so retailers can classify commerce crawlers consistently and decide whether to allow them and feed offers.
What WebmasterID can help detect
WebmasterID classifies price-comparison crawlers server-side as commerce data collectors and groups them on the bot-intelligence surface, so price crawling stays separate from human shopper analytics.
Common mistakes
- Treating price-comparison crawlers as search engines indexing your catalogue.
- Blocking the crawler while still submitting offers via a merchant feed.
- Counting price-crawl hits as human shoppers in analytics.
Privacy and accuracy notes
These crawlers are identified by user-agent only. No visitor identity is involved. WebmasterID records each as a bot event, separate from human analytics, and never attaches it to a profile.
Related pages
- Idealo price-comparison crawler
Idealo is a major European price-comparison platform, particularly in Germany, that aggregates product offers so shoppers can compare prices across retailers. Its crawler fetches retailer and merchant product pages to read prices, availability, and product details for that comparison. It is a shopping/commerce crawler, not a search engine, and operates alongside merchant feeds rather than building a general web index.
- PriceRunner crawler
PriceRunner is a price-comparison platform, strong across the Nordics and Europe, that helps shoppers compare prices and offers across retailers. Its crawler fetches retailer product pages to read prices, availability, and product details, complementing structured merchant feeds. It is a commerce/shopping crawler rather than a search engine, gathering offer data for comparison rather than a public web index.
- Shopzilla crawler
Shopzilla is a long-running shopping-comparison brand (part of the Connexity/Kit network of comparison and retail-media services) that aggregates product offers and prices for shoppers. Its data collection combines merchant feeds with crawling of retailer product pages to read prices and availability. It is a commerce/shopping crawler rather than a search engine building a public web index.
- Bot intelligence
Deterministic categorisation of commerce and search crawlers.
Sources and verification notes
- Idealo — price comparison platformExample of a price-comparison crawler combining feeds and crawling.
- PriceRunner — price comparisonExample of a European price-comparison platform.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.