Around 80% of AI crawler activity supports AI model training rather than customer acquisition or referral traffic, according to Search Engine Journal.
Businesses, meanwhile, continue paying for the bandwidth, server capacity, and infrastructure needed to support that activity.
ForeFront Web, a digital agency specializing in web design and development, SEO, and digital marketing, says businesses should evaluate AI crawler traffic with the same scrutiny they apply to any other business investment.
The agency's Senior SEO Specialist, Gabrielle Schneier, says many organizations still measure AI crawler activity by volume instead of business return.
"Search has always rewarded publishers by sending visitors back to their websites, and AI crawler traffic deserves the same level of accountability," she says.
"But if it isn't generating referrals, visibility, or revenue, businesses need to understand what they're getting in return."
Cloudflare CEO, Matthew Prince, discusses how AI crawlers are reshaping referral traffic and why publishers are rethinking the value exchange between websites and AI platforms:
How AI Crawlers Increase Website Costs
Search Engine Journal’s research, which cites TollBit data, reports that AI bot traffic grew roughly 300% over the past year.
By the end of 2025, around one in every 31 requests across TollBit's publisher network came from an AI crawler.
Most of those requests support AI model training rather than search indexing.
And as that activity grows, businesses can face greater demand on bandwidth, servers, and website infrastructure, particularly across websites with large volumes of content.
Automated traffic can inflate website reporting, which can make it more difficult to distinguish customer activity from bot activity.
"Traffic itself doesn't tell you whether your website is performing," Schneier adds.
"Businesses need to know which AI crawlers are creating value and which are simply increasing costs."
In an interview with CBS News, Cloudflare's Stephanie Cohen explains why businesses are taking a closer look at AI crawlers as website costs continue to rise:
Why Most AI Crawler Traffic Doesn't Deliver ROI
Search engines index content so users can discover it through search results and visit the original website.
AI crawlers often collect the same content to train language models without creating the same referral opportunity.
Referral traffic has long been one of the simplest ways to measure the value of being crawled.
But AI-generated answers don't always produce the same outcome, even when they reference the original source.
In fact, training-related crawling increased from 72% of AI bot activity in July 2024 to 79% in July 2025, while search-related crawling fell from 26% to 17% over the same period, per Cloudflare.
Cloudflare also reported that, across the websites it measured, OpenAI generated roughly one referral for every 1,091 pages crawled in July 2025.
On the other hand, Anthropic generated one referral for about 38,066 crawled pages.
Schneier suggests that businesses should look beyond crawler volume and identify which AI platforms generate referrals, citations, or customer engagement and which primarily consume website resources.
"Being visible in AI results has value, but visibility alone doesn't pay the hosting bill," she explains.
"Businesses still need to know which platforms are producing measurable outcomes."
Ahrefs explains how AI search is reducing referral traffic and why businesses should measure search performance by commercial outcomes rather than visibility alone:
How Businesses Can Measure the ROI of AI Crawler Traffic
Measuring crawler requests alone no longer shows whether AI traffic is contributing to business performance.
Businesses should compare AI crawler activity with the outcomes it produces.
This also applies to referral traffic from AI platforms, citations in AI-generated responses, conversion data, server logs, and infrastructure costs, all of which provide a more meaningful picture than crawl volume alone.
The same analysis can identify which areas of a website attract the highest levels of automated activity.
Resource-intensive pages that receive frequent AI crawler requests without generating measurable business outcomes may warrant closer review.
"Businesses already measure the return from advertising, software, and cloud services, so AI crawler traffic shouldn't be treated any differently," Schneier says.
"If you can't measure what it's contributing, you can't measure whether it's worth supporting."
Ahrefs explores the AI metrics businesses should monitor to understand whether AI search is delivering meaningful commercial values:
How Businesses Can Build Smarter AI Crawler Policies
Businesses that want their content to appear in AI-generated answers may still choose to allow AI crawlers, but that doesn't mean every crawler should receive the same level of access.
Crawler policies should reflect commercial priorities alongside discoverability.
As such, businesses should understand which AI platforms consistently generate referrals or citations, identify where automated traffic is placing the greatest demand on website resources, and review crawler access as AI search continues to evolve.
Website performance, infrastructure costs, analytics, and marketing return all influence how businesses evaluate AI crawler activity.
"Every business has a different balance between visibility and cost," Schneier says.
"But the important thing is understanding where AI access supports your objectives and where it's consuming resources without delivering a measurable return."
Search engines have traditionally rewarded publishers with referral traffic in exchange for indexing their content.
AI platforms, however, don't always return the same value, which leaves businesses to decide whether the value they receive justifies the cost of supporting increasing crawler activity.
So if businesses continue paying the infrastructure costs behind AI crawling, what should they expect to receive in return?
