The AIpocalypse is here for web sites as search referrals plunge

Google AI Overviews and other AI search services appear to be starving the hand that fed them.

Google's AI-generated summaries of web pages, officially released in May 2024, show up atop its search results pages so search users don't have to click through to the source website.

A year later, enterprise AI analytics biz BrightEdge reported that Google AI Overviews had generated more search impressions (up 49 percent), but click-throughs to the actual web sites dropped 30 percent

That means AI Overviews is leading more people to use Google Search to find answers to their queries. But those people are less likely to follow search results links that lead to the source website. Good for Google. Terrible for the ecosystem of web sites that had learned to depend on search referrals for buyers, readers, and viewers.

Kevin Indig, who writes about search engine optimization (SEO), marked the one-year anniversary of AI Overviews with a usability study. Based on data from the 70 individuals surveyed, he observed that when AI Overviews are absent, "outbound click rates rise to an average of 28 percent on desktop and 38 percent on mobile."

Ahrefs, an SEO site, in April said AI Overviews reduced clicks by about 35 percent.

Citing data provided by SimilarWeb (which SimilarWeb shared with El Reg, Barron's last week reported that search referrals to top US travel and tourism have fallen 20 percent year on year, while news and media sites saw search-driven traffic drop by 17 percent during that period.

Other categories of website also showed declining search referral traffic: e-commerce (-9 percent); finance (-7 percent); food/drink (- 7 percent); and lifestyle/fashion (-5 percent).

Meanwhile, AI search engine referrals have replaced only about 10 percent of traditional search referral traffic, according to SimilarWeb.

While the stats vary depending on who's measuring, the story is consistent: web publishers, who provided the content that trained these AI models, face dramatically diminishing visitors, which means lower advertising and subscription revenues, even amid overall growth in search impressions.

This may help explain the lawsuits web publishers have filed against AI firms.

The latest indication of this widely reported trend comes from Matthew Prince, CEO of Cloudflare, which has been developing tools to thwart bots that scrape web data for AI training.

Speaking on Thursday at an Axios event in Cannes, France, Prince reportedly said that ten years ago, Google's average ratio of pages crawled to visitors referred was 2:1. Six months ago, that ratio had increased to 6:1. Today, according to the Prince, it's 18:1.

While Prince did not clarify the breakdown of crawler duties (e.g. indexing, copying data for training, or something else), the suggestion is that AI companies are taking more content and giving less in return.

And that's allegedly the case for two other AI model makers that have implemented web search, OpenAI and Anthropic.

According to Prince, OpenAI's ratio of pages crawled to visitors referred has gone from 250:1 to 1,500:1 while Anthropic's ratio has gone from 6,000:1 to 60,000:1.

As we noted earlier this week, AI crawlers have become a burden for many websites, which end up bearing the cost to serve content to AI firms for commercial AI services.

Despite much ado about the potential for an AI-based challenger to disrupt Google, the Chocolate Factory still controls about 90 percent of the search market, according to BrightEdge. But it's eating all the cacao beans that made its business possible in the first place.

Google, OpenAI, Anthropic and Cloudflare did not immediately respond to requests for comment. ®