Meta crawler dominates AI training crawls

Meta is brute-forcing its way to data supremacy, willing to scrape at volumes that make OpenAI and Anthropic look like casual browsers.

• –**Aggressive Data Gathering:** Meta's crawler volume representing over 70% of the dataset signals an intense, resource-heavy effort to feed their next-generation models.
• –**Catching Up:** This massive surge in scraping supports the narrative that Meta is playing catch-up and leveraging its massive infrastructure to vacuum up as much web data as possible.
• –**Open Web Under Pressure:** The scale of Meta's operations will likely accelerate the adoption of anti-scraping tools and robot.txt blocks by publishers concerned about bandwidth and intellectual property.