SafeDNS content-classification crawler
SafeDNS is a DNS-based web-filtering service that classifies sites into content categories so its customers can allow or block them. To build and maintain that categorisation, it fetches public pages to analyse their content. This is classification for filtering, not search indexing, and appears in logs as fetches from SafeDNS infrastructure.
What this means
SafeDNS provides DNS-level web filtering: organisations and families use it to allow or block categories of sites. For that to work, the service must know what category a site belongs to, so it fetches and analyses public pages to classify them.
This is classification for filtering, not search indexing. SafeDNS is not ranking your pages; it is deciding which content category your site falls into so its customers' policies can apply.
How it identifies itself
SafeDNS classification fetches come from its own infrastructure and may carry a self-identifying user-agent. Match on the documented identity where available rather than an exact version.
Because exact tokens and ranges are not exhaustively published and may change, this entry is marked partially verified; the classification purpose and SafeDNS infrastructure are the reliable signals. As with any user-agent, treat the string as a claim.
- Purpose: content categorisation for web filtering
- Fetches public pages to classify them
- Not search indexing of your content
How it appears in analytics and logs
A SafeDNS fetch means a web-filtering service is analysing a page to assign it a content category. It is classification automation, not search indexing or human audience.
Diagnostic use case
Recognise content-classification fetches from SafeDNS in logs, separate them from search indexing, and understand them as web-filtering categorisation.
What WebmasterID can help detect
WebmasterID classifies content-classification fetches server-side as bot/monitoring traffic, keeping filtering crawls out of human analytics.
Common mistakes
- Treating classification fetches as search indexing.
- Counting filtering crawls as human page views.
- Assuming the category a filter assigns reflects search ranking.
Privacy and accuracy notes
Identification uses the request user-agent and classification context only. No visitor identity is involved. WebmasterID records the fetch as a bot event, separate from human analytics.
Related pages
- Security scanners vs search crawlers
Security scanners (Censys, Shodan, BinaryEdge, Qualys and similar) probe hosts, ports, and application surface to assess exposure and find vulnerabilities. Search crawlers (Googlebot, Bingbot) fetch and index content to rank it. Confusing the two leads to wrong robots.txt decisions and misread logs: robots.txt governs content crawling, not port scanning, and scan traffic should never be counted as audience.
- Monitoring bots vs search crawlers
Monitoring bots (uptime and performance checkers such as Pingdom and UptimeRobot) fetch your pages on a schedule to confirm availability, not to index them. They differ from search crawlers, which build a search index, and from SEO crawlers, which gather competitive data. Telling them apart keeps synthetic checks out of human analytics.
- Censys and Shodan scanning crawlers
Censys and Shodan are internet-wide scanning services that map reachable hosts, open ports, and exposed services for security research and asset discovery. They are not search-engine crawlers indexing your content for ranking; they probe infrastructure. Their requests appear in logs as scanning activity from their published scanner identities, and they offer opt-out mechanisms for operators.
- Bot intelligence
Deterministic categorisation of crawlers, classifiers, and automation.
Sources and verification notes
- SafeDNSDNS-based web-filtering / content-classification service; exact crawler tokens and ranges not exhaustively published.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.