How to block the SafeDNS crawler
SafeDNS operates a crawler that fetches public pages to categorise sites for its DNS-based content-filtering service. It is a declared crawler with a documented robots.txt token. This page shows how to disallow it, with the caveat that site categorisation can also draw on sources other than a live crawl.
What this means
SafeDNS provides DNS-based content filtering and web security. To decide which category a site belongs to, it uses a crawler that fetches public pages. Blocking that crawler asks SafeDNS to stop fetching your content for categorisation.
A block does not necessarily change how your site is classified: categorisation services often combine crawling with curated category databases, user reports and third-party feeds. So disallowing the crawler reduces direct fetching but may not move your filtering category.
How to block it
Target the SafeDNS crawler token in its own group, matching on the stable token rather than a version string.
User-agent: SafeDNSBot Disallow: /
Because robots.txt is advisory, confirm in your logs that token-carrying requests stop. If you believe your site is mis-categorised by a DNS filter, the effective fix is usually a re-categorisation request through the provider, not a robots.txt rule.
- robots.txt token to target: SafeDNSBot
- Categorisation may use databases and reports, not only a live crawl
- Mis-categorisation is fixed via the provider, not robots.txt
How it appears in analytics and logs
A request carrying the SafeDNS crawler token is a content-categorisation fetch for DNS filtering, not a human visit. It is bot traffic. The user agent is a claim, so confirm behaviour in your logs.
Diagnostic use case
Ask SafeDNS's categorisation crawler to skip your site, and understand why your site's filtering category may not change as a result.
What WebmasterID can help detect
WebmasterID classifies categorisation crawlers like SafeDNS server-side and surfaces them, so you can see whether the crawler still reaches your pages after a robots.txt change.
Common mistakes
- Expecting a robots.txt block to change your DNS-filtering category.
- Trusting the user agent without confirming behaviour in logs.
- Counting categorisation crawler hits as human traffic.
Privacy and accuracy notes
Blocking SafeDNS relies only on the request user-agent token. No human identity is involved. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- How to block the Censys scanner
Censys runs internet-wide scanning that catalogs hosts and services for security research. Because it operates at the host/port level rather than fetching pages as a polite web crawler, robots.txt is largely ineffective. This page explains what Censys does and why firewall-level controls, not robots.txt, are the right response.
- robots.txt basics: what it does and what it cannot do
robots.txt is a plain-text file at your site root that tells compliant crawlers which paths they may request. This page covers the directives, how user-agent groups are matched, and the limits that trip people up: robots.txt is advisory, it does not hide pages from search, and it is not a security boundary.
- SafeDNS content-classification crawler
SafeDNS is a DNS-based web-filtering service that classifies sites into content categories so its customers can allow or block them. To build and maintain that categorisation, it fetches public pages to analyse their content. This is classification for filtering, not search indexing, and appears in logs as fetches from SafeDNS infrastructure.
- Bot intelligence
Deterministic categorisation of filtering and security crawlers.
Sources and verification notes
- SafeDNS — documentationSafeDNS documents its categorisation crawler; confirm the exact token against current docs.
- Robots Exclusion Protocol (RFC 9309)
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.