AI Search Optimization: which system gets its evidence from where

AI Search Optimization is the work of getting a brand to appear in the answers of AI search systems. That means Google with AI Overviews and AI Mode, ChatGPT, Perplexity, Gemini and Microsoft Copilot.

Almost every guide on the topic starts with the text. There is a layer underneath that they skip. Each of these systems runs its own crawlers, and each crawler has a different job. Put the wrong line in your robots.txt and you lose visibility exactly where you meant to keep it.

In our experience this is where most quiet reach disappears. The writing is fine. The access is closed.

This article is for anyone who owns a website and wants to know what AI search actually requires. It settles the term first, then sums up what Google and Microsoft officially say about it, then covers the access layer and its crawlers.

The term covers two different things

AI Search Optimization means optimising for AI search systems. The same term also means using AI as a tool inside SEO work, so having a model build keyword clusters or generate meta descriptions.

Both meanings share the name. The search results page for the term mixes them, which is why guides on the subject regularly talk past each other.

This article covers the first one.

The difference from classic search engine optimisation sits in the output. Classic SEO works towards a ranked list of documents that a person then picks from. AI search systems assemble an answer out of several web sources and name those sources alongside it.

AI Search Optimization, GEO and AEO

The three terms do not sit side by side. AI Search Optimization is the umbrella for work on visibility in AI answers. Generative Engine Optimization describes the mechanics of how an answer is assembled and which levers have evidence behind them. Answer Engine Optimization describes the writing craft, meaning the passage that works as an answer.

The practical work does not split along those acronyms. It splits by system, because each system sources its evidence differently.

What Google says about AI features

No additional requirements

Google is more explicit here than the guide landscape suggests. Its documentation on AI features in Search, dated 10 December 2025, carries the line "The best practices for SEO remain relevant for AI features in Google Search (such as AI Overviews and AI Mode)."

The denial sits right next to it. "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." It goes on to say you need no new machine-readable files, no AI text files and no special schema markup.

Google's optimisation guide of 15 May 2026 explains why. The generative features are built on the same ranking and quality systems as Search.

Non-commodity content

At the same time Google does make content recommendations. One section of the guide is titled "Create valuable, non-commodity content for your audience".

Google describes commodity content there as material based on common knowledge that could have come from anyone. Its own example is "7 Tips for First-Time Homebuyers". Against that it sets the recommendation to offer a viewpoint of your own that stands out.

Both hold at once. Google gives recommendations and says in the same breath that they do not become an extra requirement. Turning this into "Google demands X for AI Mode" overreads the source.

What Microsoft says about AI features

Microsoft describes the same direction with different vocabulary. A post dated 6 May 2026 says the unit of value shifts from documents to "groundable information", meaning discrete supportable facts with clear provenance.

An earlier post from 12 February 2026 defines grounding as the system that connects AI to current, authoritative information.

Microsoft names four dimensions where this work departs from classic ranking. The quality of source attribution. Freshness. Coverage of high-value facts. And how contradictions between sources are handled.

Behind that sits the question of whether a single statement can be traced to a source at all. A system that is meant to cite needs identifiable facts, a traceable origin, and a rule for the case where two sources contradict each other.

What a page looks like that supplies exactly those checkable facts is covered in the article on the grounding page.

Grounding means something different at Google than at Microsoft

The term has two official uses, and they sit on different levels.

Google uses grounding as the umbrella for tying a model's answer to verifiable sources. Retrieval-augmented generation is one technique underneath it, alongside grounding with Google Search and with Google Maps. Anchoring an answer to retrieved evidence improves its quality, because the model is not writing from memory alone.

Microsoft means a layer inside its own system, sitting above crawling and indexing, that connects generative answers to current sources.

The access layer: which crawler works for which system

This is the part the guides skip. Vendors separate their crawlers by job, and the dividing line runs somewhere different at every one of them.

At OpenAI the important sentence sits right there in the documentation. "ChatGPT-User is not used to determine whether content may appear in Search." So a site that allows only the runtime fetcher and blocks OAI-SearchBot does not show up in ChatGPT's search at all.

The expensive mistake with Google-Extended

Plenty of guides recommend blocking Google-Extended in robots.txt to get out of AI Overviews. It does not work there.

Google describes Google-Extended as the control over whether content is used for training future Gemini models and for grounding in Gemini apps. The documentation states "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Elsewhere Google writes that AI is built into Search, and that robots.txt directives for Googlebot are therefore the control for access. For Google, AI Overviews and AI Mode are part of Search.

Put together, that leaves an uncomfortable choice. Anyone who genuinely wants out of AI Overviews would have to block Googlebot. That also means leaving ordinary Search.

An example that does not do what people think:

```

User-agent: Google-Extended

Disallow: /

```

Those lines keep content out of Gemini training. The page appears in AI Overviews unchanged.

To limit the display without giving up search visibility, use the snippet controls Google names for the purpose. This line in the head caps text reuse at 40 characters and drops image and video previews:

```

<meta name="robots" content="max-snippet:40, max-image-preview:none, max-video-preview:0">

```

Individual passages can additionally be excluded with data-nosnippet on the surrounding element, which helps with things like pricing that reads as misleading out of context.

robots.txt does not govern runtime fetches everywhere

When a user asks a question and the system then pulls a page, vendors treat that differently.

Perplexity states openly that its user-initiated fetcher generally ignores robots.txt. OpenAI writes for ChatGPT-User that the rules may not apply because a person triggered the action. Anthropic, by contrast, keeps Claude-User under robots.txt control.

A blanket claim that "robots.txt controls AI access" therefore fails for the runtime fetchers at two of three vendors.

Microsoft controls this through meta tags

Microsoft runs no AI crawler separate from bingbot. The control sits one layer later, on the page itself.

According to an announcement dated 22 September 2023, NOCACHE allows content to appear in generative answers, but only as URL, title and snippet. NOARCHIVE excludes the content from those answers. In both cases the page stays in regular Bing results.

That announcement dates from the Bing Chat era, the predecessor of Copilot. That the mechanism still holds unchanged is plausible and not confirmed for 2026 by this source.

The shared standard does not exist yet

This fragmentation is why the IETF is working on a single approach. The aipref working group is active.

The state of play is soberer than references to it suggest. The vocabulary draft was last revised on 27 April 2026. The draft defining the Content-Usage header field and the robots.txt binding expired on 28 October 2025. Neither has become an RFC.

The part site owners would actually deploy is the expired one. Anyone planning today is planning without this standard.

What does not survive checking

Two claims circulate on this topic that fail a cross-check. We looked into them because both came out of an SEO tool while this article was being prepared.

The swapped source

"Non-commodity content" is widely attributed to Microsoft. The term is Google's and is the literal section heading in its optimisation guide. A search across the Microsoft domains returns no hit for "commodity".

Microsoft describes a related requirement with its own term, "groundable information".

The reframed optimisation layer

Microsoft writes that grounding adds "a new optimization layer on top" of the existing infrastructure. Summaries turn that into a layer publishers are supposed to optimise.

What is meant is a layer inside Bing itself, above crawling and indexing. A publisher does not tune it.

How to approach the topic

In practice this order has worked.

Settle the access question first. Which crawlers are allowed, and does that match the intent? Decide separately for training crawlers and search-index crawlers.

Then do the content work, using the same means that already apply to Search. Google says itself that no special measures exist for this.

Finally, measure which system mentions the brand at all. How to do that without counting noise is covered in the article on prompt tracking. Which metrics hold up is covered in the article on AI visibility.

Conclusion: access first, then the writing

The content advice from the vendors is unspectacular. Google points to existing practice and to having a viewpoint of your own. Microsoft describes checkable facts with clear provenance. That fits in a sentence and needs no new rulebook.

The differences sit one layer down. Every vendor splits its crawlers differently, and the most widely repeated advice about Google-Extended does not do what it is meant to do. In practice we see a misplaced line in robots.txt more often than a weak text.