Anthropic and OpenAI: Extracting More from the Web than Contributing

As AI continues to advance, the question arises: are tech giants merely extracting online information without reciprocating? Recent insights from Cloudflare suggest a growing imbalance in this web-related exchange.

The role of data remains one of the less highlighted facets of artificial intelligence's progression. Despite heavy investments in technology and personnel, major companies strategically sidestep discussions about obtaining essential human data.

Instead of purchasing high-quality data necessary for AI training and operations, many companies deploy web-crawling bots to gather information without cost.

Historically, this practice was balanced by tech companies redirecting users back to the original sources, effectively maintaining a system where data sharing was compensated by referrals, facilitating site revenue through ads and subscriptions.

However, the emergence of AI-driven solutions is disrupting this model. These technologies provide direct responses to queries, thereby reducing site visits and undermining the original content creators.

Cloudflare's Observations

With Cloudflare overseeing about one-fifth of global web operations, the organization began scrutinizing this pattern in 2025, noting how often major tech bots crawl websites versus referring traffic back.

The ratio between crawling and referrals serves as an ethical barometer for technology firms. A high ratio indicates disproportionate extraction of data relative to referrals.

In January's initial reports, Anthropic was prominently highlighted for its excessive crawling compared to its minimal site referrals.

Similarly, OpenAI displayed a worsened balance over this period, insinuating greater data extraction from the web with diminishing returns to website operators.

Financial Implications

Reports from late 2024 raised concerns as the extensive crawling by Anthropic and OpenAI led to surging expenses for some web operators. A notable instance involved a developer witnessing a client's cloud service bills double due to active bot traffic.

This scenario reflects a wider trend where AI companies' practices impose additional financial burdens on website owners, contradicting the originally mutualistic internet framework.

Inquiries to Anthropic about these practices remained unanswered, with previous statements indicating possible inconsistencies in Cloudflare's analysis.

Future of Web and AI Interaction

Variables influencing the crawl-to-refer metric exclude app-based interactions, a potential factor in altering these metrics if considered.

Conversely, Google's traditional search practices, which still highlight direct website links, help maintain a more favorable crawl-to-refer balance. Yet, the integration of AI elements in their search services indicates a shift in traffic dynamics.

Google has consistently emphasized its commitment to web sustainability, asserting its role in supporting internet traffic flows.

Observation of these developments continues, with Business Insider planning ongoing analysis of web usage trends.