Colly scraper user agent
Colly is a popular Go web-scraping framework. Its default user agent contains a colly token and version, so default-configured Colly crawls are identifiable, but operators commonly override the UA. Built on Go's HTTP stack, it is scraping automation, not a human visit.
What this means
Colly is a scraping and crawling framework for the Go language, used to build spiders that fetch and parse pages at speed. By default it sets a user agent containing the colly token and version.
That default identifies the framework, but Colly makes changing the user agent easy, and scraping projects routinely do — often to a browser-like string — so many Colly crawls do not carry the token.
How Colly identifies itself
Colly's default user agent contains the colly token and version. Match on the colly token substring rather than a fixed version. Colly is open-source and the default and override are documented.
Underneath, Colly uses Go's net/http stack, so when the user agent is stripped rather than replaced, traffic can resemble the generic Go-http-client default. Treat the colly token as a strong but optional signal.
- Default user agent contains the colly token plus version
- Built on Go's net/http; relates to the Go-http-client default
- User agent is commonly overridden by the scraper
Catching overridden Colly crawls
When the user agent is changed, identify Colly-style scraping behaviourally: fast, parallel requests, broad URL enumeration, no asset or JavaScript loading, and minimal browser headers. These patterns matter more than the string once the default is replaced.
For crawls that respect robots.txt you can disallow paths or set crawl-delay; for those that ignore it, behavioural detection and rate limiting are the practical controls.
How it appears in analytics and logs
A request whose user agent contains the colly token is a Colly-based Go scraper with its default UA. Because the setting is commonly overridden, much Colly traffic arrives under a custom or browser-like user agent.
Diagnostic use case
Spot default Colly scraping in logs, understand its relationship to Go's HTTP client, and recognise why the token is often replaced.
What WebmasterID can help detect
WebmasterID classifies the default colly token server-side as scraping automation and surfaces it on the bot-intelligence view, while noting that overridden user agents need behavioural signals to catch.
Common mistakes
- Assuming all Colly traffic carries the colly token — the UA is often overridden.
- Confusing stripped-UA Colly with generic Go-http-client traffic.
- Counting Colly scraper requests as human visits.
Privacy and accuracy notes
Colly detection uses only the user agent and request shape. No human identity is involved — it is a Go program. WebmasterID records it as a bot/automation event, separate from human analytics.
Frequently asked questions
- How is Colly related to the Go HTTP client?
- Colly is built on Go's net/http package. If a Colly scraper removes its user agent rather than replacing it, the traffic can look like the default Go-http-client UA.
Related pages
- Go-http-client user agent
Programs built with Go's standard net/http client send a default user agent of the form Go-http-client/x.y. Seeing it means a Go application made the request — common for backend services, CLIs, and integrations. It is honest automation, not a browser, and the default can be overridden. This page covers the pattern.
- Scrapy crawler user agent
Scrapy is a popular Python web-scraping and crawling framework. Out of the box its requests carry a default user agent containing the Scrapy token and version, but operators frequently override it, so a Scrapy token is a strong signal while its absence does not rule scraping out. It is automation, not a human visit.
- SEO crawler user agents
SEO platforms run their own crawlers to build backlink indexes and audit data. Bots such as AhrefsBot, SemrushBot, and DotBot identify themselves with a documented token and honour robots.txt. They are not search-engine indexers. This page explains the family and how to recognise and control it.
- Bot vs human
Separate framework-driven scraping from real human visits.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.