Categorise crawlers, search bots, and automation traffic
WebmasterID separates bot traffic from human traffic at ingest time. The dashboard surfaces AI crawlers, search-engine crawlers, and automation tools — each with timestamps, frequency, and per-site breakdowns. No invention; uncategorised bots stay uncategorised.
Closed taxonomy, deterministic classification
Every bot visit lands in one of four buckets. The 'uncategorised' bucket is reported honestly — we never label an unknown bot as 'probably AI'.
AI crawlers
GPTBot, ClaudeBot, PerplexityBot, Google-Extended, anthropic-ai, OAI-SearchBot, and similar surfaces. Visible in the dashboard with timestamps + frequency.
Search-engine crawlers
Googlebot, Bingbot, DuckDuckBot, and other indexers. Separated from AI crawlers so SEO + AI visibility stay decoupled.
Automation tools
Headless browsers, monitoring agents, and benign automation. Bucketed into 'automation' rather than misclassified as humans.
Uncategorised
When the request signature doesn't match a known pattern, the bot appears under 'other'. Never invented; we report what we see.
Operational use cases
Bot Intelligence informs SEO, infrastructure, and AI visibility decisions.
- Detect when GPTBot or ClaudeBot reaches a site for the first time and confirm the resulting AI referrals on /ai-referrals.
- Catch automation surges that would otherwise inflate analytics if they were grouped with humans.
- Cross-reference search-bot frequency with index coverage in Search Console.
- Spot AI crawlers respecting (or ignoring) your llms.txt policy.
Frequently asked
- How does WebmasterID separate bots from humans?
- Server-side rule matching against a maintained list of bot signatures. The tracker emits a normalised event; the ingest layer classifies it deterministically. Humans and bots are stored separately so dashboards never mix them.
- Do you store the request user-agent?
- No. We extract the bot-category signal and discard the raw value. The dashboard never displays the underlying request signature.
- What about AI-search bots that don't advertise themselves?
- If a crawler doesn't identify itself, it lands in 'other'. We don't fabricate AI categorisation from circumstantial signals. The 'other' bucket is honest.
- Does bot traffic count toward analytics totals?
- No. Page views and conversion events are filtered to human traffic by default. Bot visits appear in the Bot Intelligence view and on the AI Visibility page when the crawler is an AI surface.