OPEN_SOURCE ↗
YT · YOUTUBE// 14d agoINFRASTRUCTURE
Cloudflare launches Pay Per Crawl for AI bots
Cloudflare’s AI Crawl Control now lets site owners price crawler access instead of only blocking bots. Requests can return HTTP 402 with machine-readable price headers, while Cloudflare handles billing as merchant of record.
// ANALYSIS
This is a clever attempt to turn AI scraping from a binary permission problem into a negotiated market. It’s not a universal fix, but it could become the default leverage point for publishers already sitting behind Cloudflare.
- –`crawler-price`, `crawler-exact-price`, and `crawler-max-price` create a real request/response handshake instead of a vague paywall.
- –Cloudflare acting as merchant of record removes a lot of operational friction for publishers who would never build billing themselves.
- –Adoption still depends on crawler operators supporting signed requests via Web Bot Auth, so the ecosystem has to cooperate.
- –The feature matters most where Cloudflare already fronts the site; outside that footprint, big crawlers can keep going around it.
// TAGS
pay-per-crawlcloudpricingapiautomation
DISCOVERED
14d ago
2026-03-28
PUBLISHED
14d ago
2026-03-28
RELEVANCE
8/ 10
AUTHOR
DIY Smart Code