Chapter 6 — Be Reachable

This is the least glamorous lever, the one with the strongest evidence behind it, and the one where a single wrong line in a configuration file can remove you from a platform entirely without anyone noticing for months.

The bots that matter, and the one everyone confuses

The most commonly botched item in this whole field is the difference between the crawlers that gate training and the crawlers that gate citation. Block the wrong one and you either hand over training data you meant to withhold, or you delete yourself from a platform’s answers.

That last row deserves a paragraph of its own, because the misconception is everywhere.

There is no bot called Google-Extended fetching your pages. Googlebot crawls as it always has. The token tells Google whether the resulting content may be used to train and ground its Gemini products. Google’s documentation says plainly that it does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are features inside Google Search. So blocking Google-Extended does not touch them. Teams have spent quarters believing otherwise.

And a trap worth stating plainly, because it’s the most likely way a reader breaks their own site while trying to follow this chapter: robots.txt allows by default. You do not need to “explicitly allow” anything. Worse, creating a named group — User-agent: OAI-SearchBot — means that crawler reads only that group and ignores your User-agent: * rules entirely, silently unblocking everything you had disallowed globally.

The correct action is to verify these bots aren’t disallowed. Not to add groups for them.

The deadline on your calendar

If you sit behind Cloudflare, this section has a date in it.

In July 2026 Cloudflare replaced its single AI-bot toggle with three behavioral categories: Search (indexing your content to answer questions about it later), Agent (acting in real time for a person), and Training. Allow, block, or block only on pages with ads — per category, on every plan including free.

From 15 September 2026, for new domains, new sites on existing accounts, and free-tier customers: on pages that display ads, Training and Agent are blocked by default. Search stays allowed.

The trap is the interaction. Crawlers that combine search and training get blocked if training is disabled, because the most restrictive rule wins. A site that monetizes with advertising and accepts the new defaults can quietly lose AI-search visibility from mixed-purpose crawlers, with no error message and no obvious symptom.

Audit your zone settings.

The economics driving all this are worth a line, because they explain why access is getting harder. By mid-2026, 57 percent of web traffic was bots — the first time automated requests exceeded human ones. The crawl-to-referral ratios are extraordinary: some AI crawlers fetch thousands of pages for every visitor they send back, against roughly five to one for Google. Publishers noticed. The rules are being renegotiated in public.

Render it on the server

Here is the least contested finding in this entire field, measured independently, with no dissent:

Most dedicated AI crawlers do not execute JavaScript.

Vercel, measuring around 1.3 billion monthly AI crawler requests, found no major AI crawler running JavaScript. They fetch JS files. They never execute them. A separate analysis of 23 crawlers found 69 percent unable to execute JavaScript, with Googlebot, Bingbot and Gemini’s live fetch the exceptions.

So: anything you want cited has to be in the initial HTML response. Not after hydration. Not in a tab that loads on click. Not behind a lazy-load. Not in a JavaScript-injected accordion.

The first test is view-source, not the DevTools inspector — the inspector shows you the rendered DOM after JavaScript has run, which is exactly what these crawlers never see. The definitive test is curl with the bot’s own user-agent string, since edge logic and user-agent-based rendering can serve a crawler something quite different from what your browser gets.

Both measurements are a year or more old now, in the fastest-moving corner of this subject. Re-test rather than assuming.

Stop wasting the crawl

The same Vercel data found something cheap to fix. ChatGPT’s crawler spends nearly 35 percent of its fetches on 404s. Claude’s spends 34 percent. Googlebot spends 8 percent.

Roughly a third of the AI crawl budget spent on the average site is burning on dead URLs.

Pull your server logs, filter by AI user agent, sort by status code, fix the worst offenders. It’s an afternoon, and it directly increases the share of your real content that gets retrieved.

The new traffic class nobody has a policy for

A different kind of visitor has appeared, and most security teams have never discussed it.

Browser-based agents — Perplexity’s Comet, OpenAI’s Atlas, Claude’s Chrome extension — made up roughly 71 percent of observed agentic activity in a 2026 measurement. Nearly 70 percent of that activity touched product and search pages. Only 3 percent touched checkout.

The practical risk: these look like browsers, not declared crawlers. Aggressive bot mitigation blocks them. When it does, you aren’t blocking a scraper — you’re blocking a customer’s assistant halfway through a task.

Get your security and marketing teams in a room about this once, deliberately, before it happens rather than after.

On structure, and what the evidence actually supports

Now the part where I have to be more careful than most writing on this subject, because the evidence cuts against the received wisdom.

Retrieval works on chunks. That’s documented. But the largest controlled experiment found formatting and content structure had minimal effect across every model tested, and the peer-reviewed benchmark found conversational rewrites frequently hurt ranking.

So structure your content well because it makes retrieval mechanically plausible and because it’s good writing. Not because there’s evidence of a large visibility lift. Specifically:

Don’t orphan your sections. A block that opens “as mentioned above” is ambiguous alone. Restate the subject in the first sentence. Cheap, sound.

Front-load the answer. Citations peak in the first fifth of a page and fall to almost nothing in the last tenth.

Organize around real questions. The one well-evidenced structural finding is that question-shaped queries trigger AI Overviews nearly seven times more often. That’s about how people ask, not how you format — but building content around actual questions is how you meet those queries.

Don’t fund a reformatting project. If someone proposes restructuring four hundred pages into bullet lists for AI visibility, ask for the evidence.

One button not to press

Google added a Search Console toggle that removes your site from AI Overviews, AI Mode and Discover AI features while keeping you in classic search.

Three reasons to leave it alone. You can’t make the decision on data, since Search Console reports AI impressions but not clicks or CTR — you’d be trading an unmeasured benefit for an unmeasured cost. It may take more than you intend, since a meaningful share of trending news queries embed Top Stories carousels inside AI Overviews. And in a comparative system, removing yourself from the candidate set doesn’t reduce your competitor’s visibility. It increases it.

The reachability checklist

Check robots.txt against the bot table — verify, don’t add groups — 1 day

Audit Cloudflare and WAF settings before 15 September 2026 — 1 day

Confirm key content is in view-source, then confirm with curl and each bot’s user agent — 2 days

Move client-side-only content to server-side rendering — varies

Pull AI-bot logs; fix the worst 404s and redirect chains — 1 day

Confirm no noindex or nosnippet on pages you want in answers — 1 day

Review bot mitigation against agentic browsers — 1 day

Don’t press the opt-out button