OPEN LINK ↗
// 27m agoOPENSOURCE RELEASE
Mintlify Open-Sources Agent-Traffic Dataset
Mintlify open-sourced agent-traffic, a zero-dependency dataset and classifier that distinguishes people, AI agents, crawlers, search engines, and other bots from HTTP requests. It combines user-agent patterns, headers, and published IP ranges to classify traffic and detect impersonators.
// ANALYSIS
Agent traffic is becoming a first-class observability problem, and Mintlify’s practical approach makes the issue accessible to any developer operating a documentation or web platform.
- –Covers coding agents, assistants, AI crawlers, search crawlers, HTTP clients, and browser impersonators
- –IP verification helps distinguish legitimate crawlers from services borrowing well-known user-agent strings
- –Markdown-preferring Accept headers expose agents hiding behind ordinary browser identities
- –Runs across Node, Bun, Deno, Cloudflare Workers, and Vercel Edge with zero dependencies
- –The MIT-licensed dataset gives developers a shared foundation for measuring machine readership beyond traditional analytics
// TAGS
agent-trafficagentobservabilitydata-toolsdatasetopen-sourcedevtool
DISCOVERED
27m ago
2026-11-10
PUBLISHED
1h ago
2026-11-10
RELEVANCE
8/ 10
AUTHOR
mintlify
