BACK_TO_FEEDAICRIER_2
Cloudflare launches Pay Per Crawl for AI bots
OPEN_SOURCE ↗
YT · YOUTUBE// 14d agoINFRASTRUCTURE

Cloudflare launches Pay Per Crawl for AI bots

Cloudflare’s AI Crawl Control now lets site owners price crawler access instead of only blocking bots. Requests can return HTTP 402 with machine-readable price headers, while Cloudflare handles billing as merchant of record.

// ANALYSIS

This is a clever attempt to turn AI scraping from a binary permission problem into a negotiated market. It’s not a universal fix, but it could become the default leverage point for publishers already sitting behind Cloudflare.

  • `crawler-price`, `crawler-exact-price`, and `crawler-max-price` create a real request/response handshake instead of a vague paywall.
  • Cloudflare acting as merchant of record removes a lot of operational friction for publishers who would never build billing themselves.
  • Adoption still depends on crawler operators supporting signed requests via Web Bot Auth, so the ecosystem has to cooperate.
  • The feature matters most where Cloudflare already fronts the site; outside that footprint, big crawlers can keep going around it.
// TAGS
pay-per-crawlcloudpricingapiautomation

DISCOVERED

14d ago

2026-03-28

PUBLISHED

14d ago

2026-03-28

RELEVANCE

8/ 10

AUTHOR

DIY Smart Code