Two kinds of AI search to keep apart
When people say AI search they usually mean one of two different things. The first is a feature inside a search engine, such as Google's AI Overviews and AI Mode. The second is an assistant, such as ChatGPT or Claude, that can look up web pages when a user asks a question.
They behave differently and they publish different documentation. Mixing them up leads to advice that sounds precise but has no basis. In this post we only describe what the companies themselves have written, and we say so when something is not documented.
What Google documents about its AI features
Google's Search Central page on AI features says that to be a supporting link in AI Overviews or AI Mode, a page must be indexed and able to be shown in Google Search with a snippet. The page adds that nothing further is required and no other special optimisations are necessary. It also says indexing and serving are never guaranteed.
The same page explains that both features may use a technique called query fan-out. In practice Google issues a batch of related searches on subtopics while building an answer, so it can surface a wider set of links than a single classic query. Pages that answer the specific sub-questions in a topic, such as pricing, process and suitability, have more chances to be one of those links. That is our reading, not a Google promise.
What OpenAI and Anthropic say about their crawlers
Both companies document separate crawlers for separate jobs. Allowing one does not allow the others, and your robots.txt file controls them independently.
| Crawler | Operator | Documented purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search features; sites opted out will not be shown in ChatGPT search answers |
| GPTBot | OpenAI | Collects content that may feed the training of its generative AI models |
| ChatGPT-User | OpenAI | Visits pages when a user asks something; robots.txt rules may not apply to these user-initiated visits |
| Claude-SearchBot | Anthropic | Crawls the web to improve search result quality; disabling it may reduce visibility in search results |
| Claude-User | Anthropic | Fetches pages at the moment a Claude user asks something that needs them |
| ClaudeBot | Anthropic | Collects content that could contribute to training its models |
What nobody outside these companies can tell you
The documents above describe access and eligibility. They do not describe ranking or selection. Several questions remain open to anyone outside the companies.
- Why one page is cited over another: Neither Google nor the assistant makers publish the full selection logic, so any formula you are sold is a guess.
- How often a business will be mentioned: Answers vary by query and context, and Google notes that AI Overviews often do not appear at all.
- Whether mentions send customers: Google says clicks from pages with AI Overviews are higher quality, but that is its own statement and it does not give volumes.
- How stable any result is: These systems change, so a result seen today may not repeat next month.
What this means for a Canberra business
Local discovery has always depended on who is nearby and who looks trustworthy. Google's own help pages explain that local results weigh relevance, distance and prominence together, so a complete, accurate profile feeds all three. If an AI feature draws on the same underlying information, then wrong opening hours, a mismatched address or a thin service description will cost you in both places. We cannot confirm exactly how any assistant uses profile data, so treat accuracy as good practice rather than a tactic.
Local questions put to an assistant can be very specific. Someone might ask for an electrician who covers Gungahlin, or a cafe near the lake that opens early. A page can only be quoted correctly if it states the facts plainly: the services, the suburbs you really serve, your opening hours and how to make contact. If those details are vague or missing, a person or a system has to guess, and a guess can send the customer to a competitor.
Controlling crawler access without hurting search
Each crawler listed above can be allowed or blocked separately in robots.txt, so decide the search crawlers and the training crawlers as two different questions. A business that wants to appear in ChatGPT search answers should leave OAI-SearchBot allowed. OpenAI also says it can take about 24 hours after a robots.txt change for its search systems to adjust, so do not judge a change on the same day.
Anthropic cautions that blocking by IP address is not a reliable opt-out, because it can stop its crawlers from reading your robots.txt file at all. Use robots.txt rules instead. For Google, robots.txt rules for Googlebot control crawling for Search, and Google points to the Google-Extended control for limiting use of content for AI training and grounding in other Google systems. Ask your web developer to show you the final file before and after any change, and keep a copy.
Steps that are grounded in the documents
None of these steps depends on a claim we cannot verify. They are all part of standard search practice.
- Check that your pages are indexedUse Search Console to confirm important pages are indexed and not blocked by robots.txt or noindex.
- Allow the crawlers you wantDecide on purpose which search and training crawlers are allowed, and write it into robots.txt.
- Keep text on the pagePut services, pricing approach, areas served and contact details in real text, not only in images or PDFs.
- Answer specific questionsWrite sections that answer what a customer actually asks, in the order they ask it.
- Keep local details consistentMatch your name, address, hours and categories on the site and on your Google Business Profile.
- Watch the reportsGoogle counts AI feature traffic in the Web performance report. A separate Generative AI report lists how often your pages were shown.
What we would not do
A new topic attracts new claims. These are the lines we hold.
Reasonable
- Improving pages that people and search engines can already read.
- Reading Google's AI Overviews documentation before changing anything.
- Reviewing how your brand is described when you ask an assistant, as a check rather than a ranking report.
- Learning how answer engine work differs from ordinary SEO before paying for it.
Not reasonable
- Paying for a promise to be cited by ChatGPT or Gemini.
- Adding hidden text or files aimed at language models.
- Publishing hundreds of thin pages in the hope that some are quoted.
- Treating one screenshot of an answer as proof of a trend.
Questions to put to anyone selling AI visibility
A pitch about AI search should be easy to test. These questions are fair to ask any provider, including us.
- Which document supports that claim? Google's own guidance tells site owners to compare claims about generative AI optimisation with its official documentation.
- What exactly will change on my site? The answer should be a list of pages, text and settings, not a mystery process.
- What happens if nothing is cited? An honest provider explains that citation cannot be guaranteed and says what you still gain from the work.
- How will you show progress? Look for reports on indexing, enquiries and crawl access, not a screenshot of one answer.
Frequently asked questions
Do I need special markup to appear in Google AI Overviews?
No. Google's documentation lists nothing extra to do and no special optimisations beyond being indexed and eligible for a snippet. Search Console includes sites by default, so check that nobody has excluded yours. Standard SEO fundamentals still apply, including crawl access and having the important content available as text.
Can I stop AI assistants using my website?
In part. You can disallow specific crawlers in robots.txt, such as GPTBot or ClaudeBot for training. OpenAI says user-initiated visits by ChatGPT-User may not follow robots.txt. Google provides controls such as nosnippet to limit what appears in Search, and Google-Extended to limit AI training and grounding in other Google systems.
Will blocking AI training crawlers hurt my Google rankings?
The documents we read do not say so. Google describes its own controls separately, and OpenAI and Anthropic describe training and search crawlers as separate. Review each setting on its own. Keeping search crawlers allowed is what the companies say affects search visibility.
Is AI search replacing normal Google search?
We cannot say, and the official pages do not claim it. Google says AI Overviews appear only when its systems judge them additive to classic Search, so they often do not trigger. For now, classic results and AI features both draw on a site's ordinary SEO health.
Can you get my business recommended by ChatGPT?
We cannot promise a recommendation, and no honest agency can, because selection logic is not public. We can help make your site clear, crawlable and accurate, which is the only part under your control. See our work on ChatGPT visibility for what that involves.
How do I measure whether AI features send me traffic?
Search Console has a Generative AI performance report. It breaks AI Overviews and AI Mode impressions down by URL, location, device type and day. Google also counts this traffic in the Web search type of the main Performance report. For other assistants, look at referral sources in your analytics, and treat small numbers with caution.
Sources and further reading
- AI features and your website (Google Search Central): Eligibility, no special optimisation, query fan-out, controls including Google-Extended, Search Console reporting
- Overview of OpenAI crawlers: OAI-SearchBot, GPTBot and ChatGPT-User purposes, robots.txt behaviour and the roughly 24 hour adjustment time
- Does Anthropic crawl data from the web? (Claude Help Center): ClaudeBot, Claude-User and Claude-SearchBot purposes; IP blocking is not a reliable opt-out
- Google Business Profile Help: Tips to improve your local ranking: Relevance, distance and prominence in local results
- Do you need an SEO? (Google Search Central): Advice to compare claims about generative AI optimisation with Google's official guidance
- Search Console Help: Generative AI performance report (Search): Report contents and rollout to all sites by 31 August 2026.
- Search Console Help: Search generative AI control: Include is the default; Exclude removes a site from AI Overviews and AI Mode.