Cloudflare Browser Rendering adds sitewide crawl endpoint

This pushes Browser Rendering beyond screenshots and scraping into full-blown AI data ingestion infrastructure.

• –A single endpoint now handles discovery, rendering, and extraction across a whole site, which cuts a lot of custom crawler glue code.
• –Output in Markdown and structured JSON makes the feature immediately useful for LLM pipelines, not just browser automation workflows.
• –Scope controls, incremental crawling, and static-mode fetches show Cloudflare is aiming at production crawling workloads rather than one-off demos.
• –Respect for `robots.txt` and sitemap-driven discovery matters because AI crawlers are under growing scrutiny from publishers and site owners.
• –Availability on both free and paid Workers plans lowers the barrier for developers experimenting with RAG and web-scale content collection.