A 55-day proxy-log study from Alli AI, covering 24,411,048 requests across 78,000+ pages on 69 customer websites, found that OpenAI’s ChatGPT-User crawler made 3.6x more requests than Googlebot between January and March 2026. Search Engine Journal published the dataset on April 7. The headline number reframes how B2B teams should think about crawl budget for the rest of 2026.
The category-level numbers are starker. AI crawlers as a group (ChatGPT-User, GPTBot, ClaudeBot, Amazonbot, Applebot, Bytespider, PerplexityBot, CCBot) made 213,477 requests over the window, against 59,353 for traditional search crawlers. ChatGPT-User alone exceeded Googlebot, Bingbot, and Amazonbot combined.
Our read: the B2B crawl-budget conversation has bifurcated. For a decade it was one allow-or-deny line in robots.txt. From now on it is two questions, one for retrieval bots that decide whether your content shows up in ChatGPT’s answers tomorrow, and a separate one for training bots that fold your content into model weights next year. Most B2B robots.txt files have not made that distinction yet. Cloudflare’s Sept. 15 AI crawler deadline turns that distinction into an operational default for ad-supported pages, not a theoretical robots.txt choice.
Key Takeaways
- Alli AI’s January-March 2026 proxy-log study analyzed 24,411,048 requests across 78,000+ pages on 69 sites and found ChatGPT-User crawled 3.6x more than Googlebot.
- AI crawlers as a category made 213,477 requests vs 59,353 for traditional search crawlers, also 3.6x at the category level.
- ChatGPT-User alone exceeds Googlebot, Bingbot, and Amazonbot combined, the single largest crawl footprint on most sites in the study.
- ChatGPT-User is OpenAI’s retrieval crawler (live answers); GPTBot is the separate training crawler. They require different robots.txt directives.
- The B2B robots.txt default should distinguish marketing/product pages (allow all) from proprietary docs and gated content (deny AI crawlers, keep search crawlers).
What the 24M-Request Study Actually Found
The study sampled 55 days of proxy-layer logs from 69 sites on the Alli AI crawler-management platform. Two findings matter most for B2B operators.
First, ChatGPT-User is a retrieval crawler, not a training crawler. It fires in real time when a ChatGPT user asks a question requiring fresh web information. Retrieval-crawl frequency is the lever that controls whether your content appears in ChatGPT’s answers next week. Blocking ChatGPT-User does not just protect content from training; it removes you from live answer surfaces entirely. GPTBot, OpenAI’s training crawler, is a different bot with a different decision attached.
Second, response speed compounds the volume. ChatGPT-User responds in roughly 11ms at a 99.99% success rate, against Googlebot’s 84ms at 96.3%. AI crawlers can finish a content sweep in a fraction of the wall-clock time Google needs, with knock-on effects for edge-cache strategy and origin load.
Why Retrieval-Crawl Volume Matters More for B2B Than Training-Crawl Volume
For B2B, the more useful lens is which surface the crawl produces.
Training crawls (GPTBot, ClaudeBot in some configurations, CCBot) feed model weights that update on a multi-month cycle. The marketing benefit shows up in how AI tools describe your category in 2027, not in next quarter’s pipeline.
Retrieval crawls (ChatGPT-User, PerplexityBot, the live-fetch portion of Gemini and AI Mode) decide whether your content appears as a citation when a B2B buyer asks an AI tool a vendor question today. That is the surface where 61.7% of AI citations are ghost citations and where Gemini’s referral traffic to external sites grew 115% in three months. The retrieval-crawl volume gap is the upstream cause; the citation and referral patterns are downstream effect. Perplexity’s referral decline only matters if PerplexityBot can reach your content. The downstream citation gap 5W mapped in May 2026 quantifies how far the divergence has run: only one in five pages AI engines cite still appears in Google’s top 10 for the same query, down from roughly 70% in early 2024.
The B2B Robots.txt Decision Is No Longer Binary
Three patterns are emerging in B2B robots.txt files that handle the bifurcation cleanly.
Marketing and product pages: allow all crawlers. Pricing, product, comparison, blog, and resource pages are designed to be discovered. Blocking AI retrieval crawlers here is an own-goal; the ChatGPT answer that named a competitor instead of you is the invisible cost.
Proprietary docs and customer-data exports: deny AI crawlers, keep search crawlers. Internal knowledge bases, customer-portal documentation, raw data exports, and email-gated PDFs should not be in any AI training set. Block GPTBot, ClaudeBot, CCBot, and other training-only user-agents at the directory level.
Gated content middle ground: case-by-case. Customer success stories, gated reports, and webinar registration pages sit in the middle. Allowing retrieval crawlers but blocking training crawlers is a coherent default; the user-agent split is what makes it enforceable.
The CDN side matters too. ChatGPT-User’s 11ms response window means a single popular ChatGPT answer can fan out to thousands of near-simultaneous fetches against your origin. Aggressive edge-caching of marketing pages is no longer optional. Googlebot’s crawl-budget logic is well-documented; the AI-crawler equivalents are not.
What B2B Teams Should Do This Quarter
Audit robots.txt against the 8-bot AI crawler list. Confirm explicit directives for ChatGPT-User, GPTBot, ClaudeBot, Amazonbot, Applebot, Bytespider, PerplexityBot, and CCBot. Category-level default deny without per-bot reasoning is the most common mistake; default allow without scoping proprietary directories is the second.
Run a server-log split between AI and search crawler traffic. Filter the last 30 days of logs by the eight user-agent strings above and compare against Googlebot. If your AI-to-Google ratio diverges from the 3.6x reference, that is the starting point for diagnosis.
Decide your gated-content position before the next AEO conversation. Write down, in one paragraph, which content categories are allowed for AI retrieval, allowed for training, neither, or both. Without it, every new AI bot launch becomes a fresh internal debate. The diagnostic side of the same problem caught up in May: Ignite X’s Credibility Score scores B2B brands across six AI-visibility dimensions, and its digital trust signals dimension explicitly weights the robots.txt and crawl-policy hygiene this section recommends.
The crawl-volume shift is upstream of every other AI-search metric B2B teams are tracking. Getting the input policy right is the cheapest leverage in the AEO playbook for the rest of 2026.
Frequently Asked Questions
ChatGPT-User is OpenAI’s retrieval crawler; it fires in real time when a ChatGPT user asks a question requiring live web information, and what it fetches determines whether your content appears in ChatGPT’s answers. GPTBot is OpenAI’s separate training crawler that gathers content for future model training cycles. They have different user-agent strings and require separate robots.txt directives. Blocking one does not block the other.
For marketing pages, product pages, blog content, and anything you want B2B buyers to find when they ask AI tools vendor questions: no. Blocking ChatGPT-User removes your content from ChatGPT’s live answers entirely. For proprietary docs, customer-data exports, gated whitepapers, and content you do not want surfaced in AI answers: yes, block at the directory level using a Disallow directive scoped to the ChatGPT-User user-agent.
The Alli AI study of 24 million requests across 78,000+ pages on 69 sites between January and March 2026 found ChatGPT-User crawled 3.6x more than Googlebot on average. ChatGPT-User alone exceeded Googlebot, Bingbot, and Amazonbot combined. A separate 30-day study from Digital Applied across four B2B SaaS sites found the same multiple holds in B2B-specific verticals.
Yes, for marketing pages specifically. ChatGPT-User responds in roughly 11ms compared to Googlebot’s 84ms, which means a single popular ChatGPT answer can fan out to thousands of near-simultaneous fetches against your origin. Aggressive edge caching of marketing pages through Cloudflare, Fastly, or your existing CDN protects origin servers from spikes while keeping retrieval-crawl access fast.
