Are AI Crawlers Blocked on Websites? What's Really Happening in 2026

Yes, AI crawlers are increasingly blocked on websites, and blocking is now mainstream infrastructure behavior. The more useful question for a B2B SaaS team is whether you are blocking them on purpose or by accident, and what that choice costs you in AI search. This guide gives you the current data, the two blocking mechanisms, a crawler-by-crawler map, and a decision method you can run this week.

Are AI Crawlers Blocked on Websites? The Short Answer

Yes. AI crawlers are widely and increasingly blocked on websites, and blocking is deliberate on some sites and accidental on many others. On Cloudflare's network, GPTBot was the most-disallowed AI crawler in robots.txt in Q1 2026, and automated bots now make up the majority of web traffic.

The scale is easy to underestimate. In Cloudflare Radar data shared in June 2026, automated bots generated 57.5% of HTML web traffic, the first time machine requests outnumbered human ones. AI crawlers are a large and growing slice of that machine traffic, and a real portion of the web has decided to keep them out.

Blocking happens through two mechanisms, and knowing which one you use matters. The first is robots.txt, a text file that requests crawlers stay away. The second is the CDN, WAF, or server layer, where blocks are actually enforced by user-agent or IP. Compliant crawlers honor the first. Every crawler is subject to the second.

Here is the honest nuance: the answer to "are AI crawlers blocked?" is "yes, and it depends on who is blocking and why." A publisher blocking GPTBot to protect training data is making a different decision than a B2B SaaS company that quietly blocks every AI user-agent at the edge and then wonders why it earns no AI citations. The rest of this guide separates those cases and gives you a method to find out which one describes your site.

What Counts as an "AI Crawler"? (And Why the Label Misleads)

An AI crawler is any automated agent that fetches web pages on behalf of an AI system. The label misleads because these agents split into three purposes that behave very differently, and blocking one does not block the others.

AI crawlers fall into three groups: training crawlers (GPTBot, ClaudeBot, CCBot), AI-search crawlers (OAI-SearchBot, PerplexityBot), and user-initiated fetchers (ChatGPT-User). Each group carries a separate robots.txt token, so your access policy is a set of independent decisions rather than one switch.

The distinction has real consequences. Training crawlers collect content to build and refine models. They typically send little or no referral traffic in return, which is a large reason sites choose to block them. Across full-year 2025, training drove roughly 80% of AI bot activity, according to Cloudflare data reported by Digital Applied. AI-search crawlers fetch pages to answer live user queries, and they are the crawlers that make you eligible to appear in AI answers. User-initiated fetchers act when a person asks an AI assistant to read a specific URL.

Because these tokens are independent, blocking GPTBot (training) does nothing to your appearance in ChatGPT search, which depends on OAI-SearchBot. Treating "AI crawlers" as one decision is where most access mistakes begin.

The Named Crawlers and Their User-Agents (Reference Table)

These are the crawlers worth knowing by name, with the exact robots.txt token for each. Getting the token right is the difference between a block that works and a rule that does nothing.

One point corrects a common error. OpenAI runs three separate agents: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for user-initiated fetches. Per OpenAI's documentation, robots.txt may not apply to ChatGPT-User because a person triggered the request. Google-Extended is another frequent source of confusion. Per Google's Search Central documentation, it is a training opt-out control token only. It has no separate HTTP user-agent and does not affect Google Search inclusion or ranking.

Use this table as your source of tokens. The tokens are case-insensitive in practice, but match the published spelling to avoid silent misses.

How Websites Block AI Crawlers: robots.txt vs the Edge

Two layers do the work. robots.txt asks crawlers not to visit, which is voluntary. The CDN, WAF, or server actually enforces blocks by user-agent or IP, which is not voluntary at all. Any team serious about crawler control should know which layer its blocks live in.

robots.txt works as a request rather than a wall. Compliant crawlers like GPTBot and ClaudeBot honor it, and the file still has no enforcement mechanism, so real blocking happens at the CDN or WAF layer. A robots.txt block looks like this:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

That entry tells GPTBot and CCBot to stay away from the whole site while leaving other crawlers untouched. It works only because those crawlers choose to comply.

Edge enforcement is different. At the CDN, WAF, or origin server, you can filter by user-agent string, block by IP range, or apply rate limits that throttle automated clients regardless of what they claim to be. This is where blocks are actually enforced, and it is also where accidental blocks tend to hide. Cloudflare, for example, added a block-by-default posture and managed rules that can stop AI crawlers without anyone editing robots.txt. That convenience is exactly why the enforcement layer deserves a deliberate review rather than a set-and-forget setting.

Keep the mental model simple: robots.txt states a preference, and the edge enforces reality. When the two disagree, the edge wins.

Do AI Crawlers Actually Respect robots.txt?

Mostly yes for the major named crawlers, and with an important caveat: compliance is voluntary, so robots.txt alone is not a guarantee. Major AI crawlers state they respect robots.txt, but because compliance is voluntary and unenforceable, robots.txt is best treated as a stated preference rather than a technical block.

The major operators publish opt-out instructions and say their crawlers obey robots.txt directives. GPTBot, ClaudeBot, and PerplexityBot all fall into this group. That covers the well-behaved majority of named AI traffic, which is why a correct robots.txt entry does real work against them.

The gaps are worth naming plainly. robots.txt is a voluntary standard with no technical enforcement, so a crawler that chooses to ignore it faces no barrier from the file itself. There have been documented cases of crawlers bypassing robots.txt (a recurring concern raised in 2024 coverage), which is background rather than a current headline. And per OpenAI's documentation, ChatGPT-User may not apply robots.txt at all, because the request is user-initiated rather than an autonomous crawl.

This is the reason infrastructure providers moved toward enforcement they control. Cloudflare shifted toward edge measures in 2025, including block-by-default settings and AI Audit tooling, precisely because a request at the robots.txt layer cannot stop a crawler that declines to honor it. If a block has to hold, it belongs at the edge.

How Common Is It? AI Crawler Blocking by the Numbers (2025–2026)

AI crawler blocking is now mainstream, and the trend is quantified by Cloudflare and others through 2026. The numbers below are observed measurements of what sites are doing, reported by the sources named alongside each figure.

By August 2025, more than 2.5 million sites had fully disallowed AI training (Digital Applied, citing Cloudflare) and over 1 million Cloudflare customers had turned on AI-crawler blocking, per Cloudflare's own reporting. In the five months after Cloudflare's July 2025 block-by-default switch, customers blocked 416 billion AI scraping requests, a figure Matthew Prince shared in December 2025. That is the scale of the pushback in roughly half a year.

The composition of traffic explains the reaction. In Cloudflare Radar data reported by Digital Applied, automated bots reached 57.5% of HTML traffic in June 2026, and AI crawlers accounted for about 20.3% of verified bot traffic. At the robots.txt layer specifically, GPTBot was the most-disallowed AI crawler in Q1 2026, appearing in 5.52% of AI-crawler disallow rules, ahead of CCBot (5.08%), ClaudeBot (4.88%), Google-Extended (4.44%), and Bytespider (4.23%), per TechnologyChecker.io data via Digital Applied.

Publisher behavior shows the trade-off in action. Among news publishers, 79% block at least one AI training bot, yet only 46% block Google-Extended, per Digital Applied citing Cloudflare. Publishers that appear reluctant to block Google-Extended tend to be protecting Google Search visibility, since the decision to block a training token sits close to their organic-search lifeline. The pattern is a useful signal: sites block aggressively where the cost is low and hold back where a block might touch traffic they depend on.

Should You Block AI Crawlers? The Trade-Off for B2B SaaS

It depends on the crawler's purpose. Blocking training crawlers protects content with little downside, while blocking AI-search crawlers removes you from AI answers, which most B2B SaaS companies do not want. Blocking OAI-SearchBot removes a site from ChatGPT search answers, so a B2B SaaS company that wants AI citations should block training crawlers selectively while allowing AI-search crawlers.

The value exchange with training crawlers is lopsided, which is why blocking them feels reasonable. Cloudflare Radar data reported on crawl-to-refer ratios shows extreme imbalances: ClaudeBot has been observed above 11,000:1 (reaching roughly 70,900:1 in one June 2025 week), PerplexityBot around 118:1, Microsoft Copilot around 33:1, and Google around 4.9:1. Sites that block training crawlers are responding to that imbalance, where a crawler takes a great deal and returns almost no traffic.

The other side of the ledger is AI-search referral traffic, which has been growing. Adobe Analytics reported that AI-referred traffic to US retailers grew 393% year over year in Q1 2026, and Adobe cautions that this figure is retail-specific and does not generalize to other sectors. For B2B SaaS the takeaway is directional: the channel that AI-search crawlers feed is real and expanding, so cutting off those crawlers forecloses a growing source of qualified discovery.

For a PropTech or FinTech SaaS deciding what to allow, three neutral questions make the trade-off concrete:

1. Does your access policy match your AI-visibility goal? If you want to appear in AI answers, the AI-search crawlers must be allowed to reach your pages.
2. Are you separating training from search? Blocking training crawlers can protect content while leaving your AI-search eligibility intact.
3. Would a block touch a channel you rely on? The publisher pattern with Google-Extended shows how a single token can sit uncomfortably close to core traffic.

Are You Accidentally Blocking AI Crawlers? (The B2B SaaS Blind Spot)

Many B2B SaaS sites block AI crawlers without meaning to. It usually happens at the CDN or WAF layer, through login walls, or via JavaScript rendering, so the crawlers never reach the content the marketing team keeps publishing. For B2B SaaS, accidental AI-crawler blocking usually lives at the CDN, WAF, or bot-protection layer while robots.txt looks clean, so marketing teams often stay unaware that their content is unreachable.

Four mechanisms cause most of the accidental blocks:

• Aggressive WAF and bot rules. Bot-protection settings often treat unfamiliar automated user-agents as threats and block them by default, which quietly catches AI crawlers.
• Block-by-default toggles. A one-click infrastructure setting can disallow AI crawlers across the whole domain, and it stays on until someone reviews it.
• Login and paywall gating. Content behind authentication is unreachable to a crawler that cannot log in, so gated resources never enter AI answers.
• Heavy JavaScript rendering. When key content loads only after client-side JavaScript that a crawler does not execute, the crawler sees an empty shell.

There is directional evidence that this is common. A LightSite AI audit of roughly 2,870 mostly US and UK sites, weighted toward B2B SaaS with about 30% eCommerce, found that 27% blocked at least one major LLM crawler, usually at the CDN, hosting, or WAF layer rather than in robots.txt or the CMS. This is a self-reported vendor sample rather than an independent authority, so treat it as a directional signal of prevalence and hold the exact percentage loosely. In that sample, B2B SaaS sites were in the worst shape.

The root cause is an ownership gap. Marketing publishes content and tracks visibility, while engineering configures the WAF, CDN, and bot rules that decide whether crawlers can reach any of it. Neither team owns the seam between them, so a block set for good security reasons can silently erase the marketing team's AI-search eligibility.

How to Check Whether Your Site Blocks AI Crawlers (5-Minute Self-Audit)

You can confirm whether AI crawlers can reach your site in about five minutes. Check robots.txt, test a request with each crawler's user-agent, and review your CDN or WAF bot rules. To check if your site blocks AI crawlers, read /robots.txt for disallow rules on GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, then send a test request using each user-agent to see whether the CDN or WAF returns a 200 or a block.

Run these five steps in order:

1. Read your /robots.txt. Open yourdomain.com/robots.txt and look for Disallow rules tied to GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. This tells you your stated preferences. It cannot tell you what the edge actually enforces.
2. Send a test request with each user-agent. Use a command-line request that sets the crawler's user-agent, for example curl -A "OAI-SearchBot" -I https://yourdomain.com/. A 200 means the page is reachable for that agent. A 403, 429, or challenge page means something is blocking it. Repeat for each AI user-agent you care about.
3. Review your CDN, WAF, and bot-management dashboard. Check for AI-crawler categories, block-by-default toggles, and rules that filter unknown user-agents. This is where enforcement lives, so it is the step most likely to reveal an accidental block.
4. Confirm the content renders without JavaScript or login. View the page as raw HTML (disable JavaScript, or fetch the source) and confirm the substantive content is present. If the body only appears after client-side rendering or behind a login, crawlers may reach an empty page even when the status code is 200.
5. Decide allow versus block per purpose. With the results in hand, set your policy by crawler purpose: training, AI-search, or user-fetch. The self-audit tells you the current state. The purpose framework tells you the target state.

Keep the limits of each check in mind. robots.txt shows intent, the curl test shows edge behavior for a single request, and the dashboard shows the rules. Together they give you a reliable picture, and no single step does it alone.

What We Found When We Audited Our Own Crawler Access (First-Hand)

When PropSaaS Growth audited its own site, AI crawlers reached every page freely, and the brand still earned zero AI citations, which shows that unblocking crawlers is necessary and still not enough on its own. Our site was fully indexed and reachable by AI crawlers, yet a five-prompt audit across Claude and Perplexity returned zero brand presence. The gap was a citation-footprint problem sitting on top of clean crawler access.

The exercise is documented in our published self-audit, "I Ran My AEO Framework on My Own Brand. I Was Invisible." (May 30, 2026). Founder Gemma Smith ran a five-archetype prompt set across Claude and Perplexity against a site that was fully indexed and reachable by AI crawlers. The result was zero brand presence across all five prompts on both engines.

The conclusion we drew: the site is indexed, and the engines do not yet know to recommend it. Crawler access was clean, and citations were still absent. That gap is the whole lesson. Access is the entry ticket. Citation footprint is the game.

This is exactly why "am I blocked?" is step one rather than the finish line. Confirming that crawlers can reach you clears the first hurdle, and it says nothing about whether AI engines will cite you. A B2B SaaS team that unblocks its crawlers and then stops has done the necessary work without doing the sufficient work. The self-audit is where that distinction became concrete for us, and it shaped how we treat crawler access as the beginning of an AEO program rather than the end of one.

What This Means for B2B SaaS

Blocking AI crawlers is now a routine, per-purpose access decision. For B2B SaaS, the priority is making sure you are not blocking the crawlers that feed AI answers. For B2B SaaS, the highest-value AI-crawler action is confirming you are not accidentally blocking AI-search crawlers at the CDN or WAF layer.

Four principles carry the framework:

• Know the layers. robots.txt states a preference, and the CDN or WAF enforces it. Accidental blocks usually live at the edge.
• Separate training from search. Blocking training crawlers protects content. Blocking AI-search crawlers removes you from AI answers.
• Audit before deciding. Confirm the current state with a five-minute self-check before you change any rules.
• Measure after. Set your policy by purpose, then verify AI visibility on a schedule so a silent block cannot resurface unnoticed.

Action Steps

Here is the sequence to run this week. Run the self-audit, set per-purpose robots.txt tokens, align your WAF rules, then re-check AI visibility in 30 days.

1. Run the self-audit. Read /robots.txt, send a curl test with each AI user-agent, and review your CDN, WAF, and bot-management dashboard.
2. Set robots.txt tokens by purpose. Decide training, AI-search, and user-fetch access separately, and write the exact tokens (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) that match those decisions.
3. Align your WAF and CDN rules. Make sure the enforcement layer agrees with robots.txt, so you are not allowing a crawler in one place and blocking it in another.
4. Verify rendering. Confirm your key content is present in raw HTML, without depending on JavaScript execution or a login.
5. Set a 30-day AI-visibility recheck. Put a recurring review on the calendar so any new block gets caught early.

If you want a partner to run this end to end for a PropTech, FinTech, or B2B SaaS site, see our services. And if you want a fast, structured look at where your AI-search foundations stand, book a Foundation Audit.

The takeaway

So, are AI crawlers blocked on websites? Yes, widely and increasingly, through two layers: a voluntary request in robots.txt and real enforcement at the edge. The deliberate-versus-accidental frame is what matters for your site, because a block you never chose can be as costly as one you did. For B2B SaaS, the priority is clear: make sure you are not blocking the AI-search crawlers that decide whether you show up in AI answers. Confirm access first, then treat citation footprint as the real work. The one-line takeaway: check whether you are blocked, decide access by crawler purpose, and measure what changes.

Frequently asked questions