Your robots.txt Lets AI Agents In. Your Edge Might Not

Most agent-readiness advice starts and ends with robots.txt. If the file allows the agent, the site is open to agents. Or so the checklist says.

We wanted to know whether that holds at the network edge, where CDNs and bot-management products actually decide who gets a page. So we ran a pre-registered study, The Silent Wall, on 150 popular websites. We fetched the same public pages as three honest clients and compared what each one got back with what each site's robots.txt says.

The short version: declaring ourselves an AI agent didn't cost us access, but robots.txt was a poor guide to what the edge would do. On 16.8% of the sites whose robots.txt allowed our agent, the edge still challenged or blocked it.

What We Did

We took 150 domains from the Tranco top-sites list, 30 in each of five sectors: SaaS, e-commerce, finance, media and developer tools. A written rule fixed the selection before any data was collected.

For each site we did two things:

1. Read the policy. We fetched robots.txt once and parsed it with an RFC 9309 parser. For 14 vendor-documented crawler tokens (GPTBot, ClaudeBot, ChatGPT-User, Claude-User and others, plus Googlebot as a control) we recorded whether the file names the token and whether it allows /.
2. Observed the behaviour. We fetched the homepage and up to three public pages from the sitemap as three clients, in random order, from the same connection:

We never pretended to be OpenAI's, Anthropic's or anyone else's crawler. Edges verify those by IP range, reverse DNS or signatures. Imitating them would measure how sites treat impostors, and it isn't a defensible thing to do in a published study. Our agent declared itself honestly and linked to a public bot page with an opt-out.

Each response was sorted into pre-registered classes: ok, thin, challenge, block, rate_limited, error, other or self_excluded. We ran two full rounds. We checked 20 results by hand against the stored evidence, and all 20 matched the rules.

Finding 1: Saying "I'm an Agent" Didn't Cost Access

Homepages were challenged or blocked for:

• the declared agent on 16.0% of sites (95% CI 11.0–22.7);
• the unlabelled script on 18.7% (13.2–25.7);
• the headless browser on 18.7% (13.2–25.7).

None of the paired differences is statistically significant (McNemar p = 0.45 for agent vs. browser, 0.29 for agent vs. script). The second round repeated these results almost exactly: outcomes agreed for 96.7–98.0% of domains, depending on the client.

So, for a new, honestly labelled agent, the label itself wasn't what got it stopped. Edges that stopped our agent mostly stopped the other two clients as well. On 19 of the 33 sites that challenged or blocked anyone, all three were stopped.

Finding 2: The Policy Said Yes, the Edge Said No

Among the 137 reachable sites whose robots.txt allowed our agent at /, the edge still challenged or blocked it on 16.8% (11.5–23.9).

That gap is real, but it is mostly not aimed at AI. In nearly all of those cases the headless browser was stopped too. It got the page on only 2 sites (1.5%). These edges are stopping non-human clients in general, whatever robots.txt says.

A few responses stood out. 12 sites refused our agent even its robots.txt (HTTP 403), so the file meant to say what's allowed couldn't be read by the client it was written for. One site answered 402 Payment Required, and one returned a vendor-specific 498 block code.

Finding 3: Where You Host Matters More Than Who You Are

Blocking rates differed far more by sector and edge vendor than by client identity. For the declared agent:

By edge vendor, inferred from response headers, the rate ranged from 33.3% on Akamai-fronted sites (27 sites) to 0% on Vercel (7 sites). Within each group, all three clients were treated much the same.

These cells are small, between 3 and 46 sites, so read them as descriptive. They also describe how those vendors' customers configure their defences, not the vendors' products as such.

Finding 4: Policies Single Out Training, Not Agents

Of the 140 sites with a fetchable robots.txt:

• 23.6% disallow at least one AI training crawler at /. The most common were Bytespider (19.3%), CCBot (18.6%) and ClaudeBot (17.1%).
• 13.6% disallow at least one AI search crawler.
• 12.1% disallow at least one user-initiated agent such as ChatGPT-User or Claude-User.
• 0% disallow Googlebot.

The written intent is narrower than "block AI". About twice as many sites disallow a training crawler as disallow an agent that fetches a page because a person asked it to.

How This Fits With Other Measurements

Another study, published by PageLens on 12 October, ran a similar test on the same Tranco list. It took a different approach: it sent requests named after well-known AI crawlers such as GPTBot and ChatGPT-User. It found that 9.3% of the 740 sites it could reach turned those requests away while letting a browser and a made-up crawler name through.

The two results fit together. Using a famous crawler's name from the wrong network gets you treated as an impostor, and some edges block impostors while letting the real, verified crawler through. PageLens notes that from the outside this looks the same as blocking AI. A new agent that declares its own name doesn't carry that baggage: in our data it was treated like any other non-browser client.

For agent builders, the practical upshot is to identify yourself honestly, and to use a verifiable identity (published IP ranges or Web Bot Auth signatures) if you want to be treated better than an anonymous script.

What Site Owners Should Check

If you want agents to be able to read your public pages:

1. Test the edge, not just the file. Fetch a few public pages with a plain HTTP client and a headless browser, not only a desktop browser. If they get challenges or 403s, your robots.txt isn't the policy that counts.
2. Make sure robots.txt itself is reachable. If your edge blocks non-browser clients from reading robots.txt, well-behaved agents can't learn your rules.
3. Decide what you're blocking. If your aim is to stop training crawlers, say so for those tokens. Bot rules that block every non-browser client also block the user-initiated agents your customers send.
4. Allow verified agents explicitly. Most bot-management products can allow-list verified bots. That's the reliable way to let a known agent in while still stopping impostors.

We're adding a policy-versus-enforcement check to the Agent Readiness Score. It will flag sites whose robots.txt allows agents while the edge challenges or blocks them. It's planned for the next ARS release, and we'll document it on the methodology page when it ships.

Limitations

• One vantage point. All requests came from one residential connection in Nigeria. Edge decisions depend on IP reputation and geography. Comparisons between the three clients are controlled because they shared the IP, but the absolute rates may differ elsewhere. Nine domains were unreachable for every client from this location.
• 150 domains. Sector and vendor cells are small.
• Headless means headless. Our browser was identifiable as headless Chrome, so its results aren't a human visitor's.
• Verified crawlers aren't covered. We measured how sites treat a declared but unverified agent, which is what most independent agents look like today. We didn't measure how they treat OpenAI's or Anthropic's verified crawlers.
• Policy as written. We recorded robots.txt as written. OpenAI and Anthropic say their user-initiated agents may not follow it.
• One reviewer. The 20-result hand check was done by the person who wrote the harness.

Read the Full Study

• Report: The Silent Wall (2026)
• Pre-registration, data and harness: github.com/Glintbase/research
• Our crawler: GlintbaseResearch bot page

Disclosure: Glintbase sells agent-readiness scoring (ARS). This study was pre-registered before data collection, and deviations from the plan are logged in the pre-registration.