Censys and Shodan scanning crawlers
Censys and Shodan are internet-wide scanning services that map reachable hosts, open ports, and exposed services for security research and asset discovery. They are not search-engine crawlers indexing your content for ranking; they probe infrastructure. Their requests appear in logs as scanning activity from their published scanner identities, and they offer opt-out mechanisms for operators.
What this means
Censys and Shodan continuously scan the public internet to build searchable indexes of hosts and services — what ports are open, what banners services return, and which devices are reachable. Security teams use these to find their own exposed assets; researchers use them to study the internet at large.
This is fundamentally different from a search crawler. Censys and Shodan are not reading your articles to rank them; they are characterising your infrastructure's externally visible surface.
How they identify themselves
Both services publish information about their scanning and provide opt-out paths. Their probes can carry self-identifying user-agents or originate from documented scanner infrastructure, and each publishes guidance on recognising and excluding their scans.
Because exact tokens and IP ranges vary across their scanner fleets and change over time, this entry is marked partially verified. The reliable signal is the scanning behaviour plus each provider's published scanner documentation; corroborate with their opt-out and identification pages.
- Purpose: host/service discovery, not content indexing
- Each publishes scanner identification and opt-out guidance
- Exact tokens/ranges vary across scanner fleets over time
What to do about it
If you do not want to appear in these indexes, follow each provider's documented opt-out or exclusion process rather than relying on robots.txt, which governs content crawlers and does not control port scanning.
Treat scanning as a prompt to review your external exposure: if a service is reachable and indexed by a scanner, it is reachable by anyone. The scan itself is a visibility signal, not an attack.
How it appears in analytics and logs
A Censys or Shodan request is an infrastructure scan probing reachable services, not a content crawl for search. It is automated scanning traffic and should be treated as bot/recon activity, not audience or SEO crawling.
Diagnostic use case
Recognise internet-scanning probes from Censys and Shodan in logs, distinguish them from search indexing and SEO crawlers, and decide whether to request exclusion.
What WebmasterID can help detect
WebmasterID classifies recognised scanner probes server-side as bot/monitoring traffic and shows which endpoints they reached, so security scanning does not blend into human analytics or search-crawl coverage.
Common mistakes
- Expecting robots.txt to stop internet-wide port scanning.
- Treating scanner probes as search indexing that affects rankings.
- Counting scanning requests as human visits.
Privacy and accuracy notes
Scanner identification uses the request user-agent and published scanner information only. No visitor identity is involved. WebmasterID records the probe as a bot event, separate from human analytics.
Related pages
- Security scanners vs search crawlers
Security scanners (Censys, Shodan, BinaryEdge, Qualys and similar) probe hosts, ports, and application surface to assess exposure and find vulnerabilities. Search crawlers (Googlebot, Bingbot) fetch and index content to rank it. Confusing the two leads to wrong robots.txt decisions and misread logs: robots.txt governs content crawling, not port scanning, and scan traffic should never be counted as audience.
- BinaryEdge scanning crawler
BinaryEdge is an internet-scanning service that collects data on reachable hosts, open ports, and exposed services for security and threat-intelligence use. Like Censys and Shodan, it probes infrastructure rather than indexing your pages for search ranking. Its scanning appears in logs as automated probes, and the service provides information for operators who want to identify or exclude it.
- Qualys web application scanner
Qualys operates security scanning that assesses web applications and infrastructure for vulnerabilities and misconfigurations. Some Qualys scanning is authorised by the site owner (an internal security assessment); some is part of broader internet measurement. It is a security tool, not a search crawler, and its probes appear in logs as scanning rather than content fetching for ranking.
- Bot intelligence
Deterministic categorisation of crawlers, scanners, and automation.
Sources and verification notes
- Censys — about scanningInternet-wide scanning service; opt-out and scanner identification documented on site.
- ShodanInternet-wide host/service search; exact scanner tokens and ranges vary over time.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.