AI Crawl vs AI Traffic: What a 560,000-Request Study Changes for Your AEO Strategy

TL;DR: A study by Andy Crestodina (Orbit Media Studios), covering more than 560,000 AI bot requests across 74 websites, separates two behaviors that are often confused: crawling (an AI bot reads a page) and referral (a human clicks a link cited by the AI). Homepages get crawled roughly 15 times more than the rest of the site, but product and service pages generate the most real visitors, about 3 times more than blog posts. And a page buried more than 3 folders deep receives almost no AI referral traffic at all.

Writing for humans and for Google's algorithm has been the rule for twenty years. A third visitor now silently browses your pages: the AI search agent (GPT, Claude, Perplexity...). What is needed here is not a new theoretical framework, but a cold reading of the data: which pages do these bots actually read, and above all, which ones send back real visitors?

This article draws on the study "AI Crawls vs. AI Traffic" by Andy Crestodina, co-founder of Orbit Media Studios, published on July 8, 2026 and based on 560,695 AI bot requests analyzed across 74 websites (Cloudflare AI Crawl Control data). The results shake up part of the conventional wisdom of classic SEO and refine what we already know about AEO and GEO: being read by an AI and being recommended by it are two different things, measurable separately.

Methodology note: every figure quoted in this article was checked directly against Orbit Media Studios' original publication. We did not, however, have access to the raw Cloudflare dataset behind the study; cross-check with the source if you need to quote it in a high-stakes context.

What Exactly Does the Orbit Media Study Measure?

The study distinguishes two AI bot behaviors, based on Cloudflare reports:

• Crawling: an AI bot reads a page to learn from it or index it.
• Referral: a human actually clicks a link cited by the AI in an answer.

The 74-site sample covers varied sectors (professional services, B2B tech, manufacturing, nonprofits, e-commerce), which makes it possible to draw trends applicable to most sites, without being limited to a single type of business.

Why Does AI Crawl the Homepage So Much?

This is the study's most striking finding: AI engines crawl homepages roughly 15 times more than any other type of page.

Mathematically, a homepage, a single URL among hundreds, should receive only a tiny fraction of bot attention. That is not the case: it serves as a semantic pivot, the place where the AI goes to find the brand's core identity before forming an opinion about the rest of the site.

Cyrus Shepard, founder of Zyppy, quoted in the study, notes that many publishers neglect their homepage's potential, letting secondary pages (about, support) carry the brand message, when that is precisely where the AI looks for the information at the source.

How to Optimize Your Homepage for AI

If you were to optimize only one page, it would be this one. It must become a complete training manual for a language model, not just a visual front door:

• a short, sharp sentence summarizing your positioning;
• explicit details on what you do, how, for whom, and your differentiators;
• direct answers to the most common sales questions;
• tangible proof: numbers, awards, testimonials;
• case study excerpts with concrete results displayed directly, not just a link to another page.

Does Site Size Make a Real Difference?

The study finds a strong correlation (0.86) between a site's number of pages and the attention it receives from AI. More pages mechanically means more entry points to match user queries.

But size is not everything: some 50-page sites receive as much attention as 1,000-page sites, for three combined reasons: better semantic optimization of the pages, a brand already strong in the AI's training data, and overall marketing that generates external mentions. This last point ties directly to what we detail in our article on site architecture for topical authority: producing lots of content without structure is not enough; every page must reinforce the others.

Why Do Blog Posts Generate So Few AI Clicks?

Being read by AI is not the same as receiving traffic from AI. That is the whole point of the crawl/referral distinction the study establishes.

Blog posts are massively crawled by bots, but show roughly 20% fewer clicks than homepages. Crestodina calls this the Dark Library Effect: the AI reads the article, answers the user directly, cites your site, but nobody clicks.

Conversely, product and service pages generate about 3 times more referral traffic, per page, than blog posts. The intent changes: a user considering a purchase wants to see the site, test the experience, verify the company's credibility. The AI filters upstream, but the click remains necessary for the final decision.

That does not mean you should stop publishing. 47% of the pages analyzed in the study generated no direct traffic, which is not a failure in itself: crawling feeds the background knowledge the AI has of your expertise, even when it does not translate into an immediate click. It is the same principle we develop in our article on getting cited by Perplexity: every article counts over time, not just on its traffic at day 30.

What Is the Impact of URL Depth on AI Traffic?

AI willingly crawls a page buried at the 3rd or 4th folder level, but its willingness to recommend it collapses with depth. Crestodina calls this the architecture tax.

That is a strong signal in favor of a flat architecture. Your services and pillar content must stay within one or two clicks of the homepage: it was already a UX best practice, it is now a direct AI visibility factor.

Should You Block AI Bots via robots.txt?

Faced with declining classic organic traffic, blocking AI bots can seem tempting. The Orbit Media study advises against this approach: your site remains the only corner of the internet you fully control.

If you block it, the AI will still learn about your brand's existence through other sources (social networks, forums, press), but you lose the opportunity to train it on your own terms, with your own proof. This is consistent with what we observe on Perplexity: the engine relies heavily on classic search engine results and does not always respect robots.txt restrictions by default, so blocking does not necessarily prevent being mentioned; it just deprives you of control over the source.

Action Plan

1. Your homepage: condense your positioning, your proof, and your answers to sales objections into it.
2. Your product/service pages: they are what turn an AI citation into a visitor, then a customer.
3. Your architecture: bring your key content closer to the root to escape the depth tax.
4. Your blog: keep publishing without expecting immediate traffic on every article; it feeds what the AI knows about you over time.

At ForgR, this is the authority Marc and Clara build around your site by deploying a network of topical blogs optimized for SEO and GEO, while Gaïa monitors your actual visibility in generative AI answers.

FAQ

What is the difference between AI crawling and AI referral traffic?

Crawling is an AI bot reading a page to learn from it or index it. Referral is a human actually clicking a link cited by the AI in an answer. A page can be heavily crawled without ever generating a click, which is the case for the majority of blog posts according to the Orbit Media study.

Why is my homepage crawled so much by AI bots?

Because it serves as a semantic pivot: it is the place where the AI goes to find your brand's core identity before forming an opinion about the rest of the site. The Orbit Media study measures roughly 15 times more crawling on homepages than on other pages.

Should you stop publishing blog posts if they generate little AI traffic?

No. 47% of the pages in the sample generated no direct traffic, yet still feed the background knowledge the AI has of your expertise. The blog builds an authority measured over time, not just by the immediate click.

At what URL depth does a page become invisible to AI traffic?

Referral traffic drops sharply from 3 folders deep (about 25% of the baseline level) and becomes nearly zero at 4 folders and beyond. Better to keep key content within one or two clicks of the homepage.

Does blocking AI bots with robots.txt protect my content?

No, and it can work against you. The AI will still learn about your brand's existence through other sources (social networks, press, forums), but you lose the opportunity to train it with your own proof and your own content.

What is the source of these figures?

The study "AI Crawls vs. AI Traffic" by Andy Crestodina, strategist at Orbit Media Studios, covering more than 560,000 AI bot requests analyzed across 74 websites.

Sources

• Orbit Media Studios : AI Crawls vs. AI Traffic: What 560,000+ Requests Reveal About Where AI Actually Sends Visitors, a study by Andy Crestodina, published July 8, 2026, 560,695 AI requests analyzed across 74 sites (Cloudflare AI Crawl Control data).
• ForgR : AEO: structuring your content to get cited by AI, a complete definition of AEO and a content structuring method.