How ChatGPT, Google and other AI assistants decide which business to recommend
By Paul Jeevanesan A., Founder and CEO, StudAI One · Updated · 9 min read
The short answer
AI assistants recommend businesses from two sources: what the model learned during training, and pages it retrieves live through a search index or crawler when the question needs current information. You can influence the live part by letting the right search crawlers reach your site, stating clear facts about your business, and keeping your details and reviews consistent across the web. No one can guarantee a recommendation, because answers vary with the question, the user, the location and the time.
Key takeaways
- Search-backed answers depend on crawler access: Googlebot for Google, Bingbot for Bing and Copilot, OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity and Claude-SearchBot for Claude.
- Search visibility does not require allowing model training. OpenAI, Google and Anthropic each separate training controls from search controls.
- Google says no special schema or AI text files such as llms.txt are needed for its AI features, and Google-Extended does not affect Search ranking.
- Clear facts, consistent business details across the web, genuine reviews and crawlable pages are the parts you control.
Model memory vs live search: where do answers come from?
Model memory is what a language model absorbed from its training data, which stops at a cut-off date. Search-backed retrieval is when the assistant runs searches, reads current pages and writes an answer from them, usually with links. A recommendation such as 'a good physiotherapist near me' depends on current, local information, which is exactly what retrieval is for.
| Model memory | Search-backed retrieval | |
|---|---|---|
| Freshness | Fixed at the training cut-off | As fresh as the index or the live fetch |
| Links to sources | Usually none | Usually shows the pages it used |
| How you influence it | Slowly and indirectly, through what is published about you over time | By being crawlable, indexed and clearly written now |
| Main control | Training crawlers such as GPTBot and ClaudeBot, and the Google-Extended token | Search crawlers such as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot |
The second column is the one you can act on this month. It also explains why an assistant can give an out-of-date answer when it does not search: it is relying on memory.
Which crawlers feed which AI assistants?
Each company documents its own crawlers. The table summarises what each one does according to the official pages listed in the sources at the end of this guide.
| Crawler or token | Company | What it is for | If you block it |
|---|---|---|---|
| Googlebot | Crawling for Google Search, which also supplies AI Overviews and AI Mode | Your pages cannot be crawled for Search or its AI features | |
| Google-Extended | A robots.txt token, not a separate crawler, controlling use of content for training future Gemini models and grounding in some other Google systems | No effect on inclusion or ranking in Google Search | |
| Bingbot | Microsoft | Crawling for Bing; Copilot grounds web answers through the Bing search service | Less visibility in Bing and in Copilot's web answers |
| OAI-SearchBot | OpenAI | ChatGPT search | Your site is not shown in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches pages when a ChatGPT user's action needs them; not automatic crawling | OpenAI says robots.txt rules may not apply to user-initiated fetches |
| GPTBot | OpenAI | Training generative AI foundation models | Your content should not be used for training; search is a separate setting |
| PerplexityBot | Perplexity | Surfacing and linking websites in Perplexity search; not used for training foundation models | Your site is less likely to appear in Perplexity results |
| Perplexity-User | Perplexity | User-triggered fetches inside Perplexity | Perplexity says it generally does not follow robots.txt, because a user requested the page |
| Claude-SearchBot | Anthropic | Improving search result quality for Claude users | Prevents indexing of your content for search |
| Claude-User | Anthropic | Retrieving pages in response to a Claude user's query | Prevents retrieval in response to user queries |
| ClaudeBot | Anthropic | Collecting web content for model training | Signals that future materials should be excluded from training |
The practical point: search visibility does not require allowing training. OpenAI states (opens in a new tab) that each of its settings is independent, so you can allow OAI-SearchBot and disallow GPTBot. Google says (opens in a new tab) Google-Extended does not affect inclusion in Search and is not a ranking signal. Anthropic (opens in a new tab) separates ClaudeBot (training) from Claude-SearchBot and Claude-User. OpenAI notes that robots.txt changes can take about 24 hours to affect its search results.
What does Google say about its AI features?
Google's page on AI features and your website (opens in a new tab) is unusually direct. Its main points:
- There are no additional requirements or special optimisations needed to appear in AI Overviews or AI Mode.
- You do not need to create new machine-readable files, AI text files or markup, and there is no special schema.org structured data to add.
- To be shown as a supporting link, a page must be indexed and eligible to appear in Google Search with a snippet.
- You can limit what is shown with nosnippet, data-nosnippet, max-snippet or noindex.
- Traffic from AI features is included in Search Console's Performance report under the Web search type.
So an llms.txt file is not a route to better Google visibility and will not help your Google rankings. Publishing one does no harm, but your effort is better spent on the pages themselves.
Why clear facts and consistent details matter
When an assistant describes a business, it combines snippets from several pages. If your website lists one set of opening hours and a directory lists another, the answer may repeat the wrong one, or leave you out because the sources disagree. Consistency reduces that risk. Keep these identical everywhere:
- Business name exactly as customers see it, and the legal name where relevant
- Address or service area, phone number and email
- Opening hours, including holiday changes
- The services you offer and the places you serve, in plain words
- Who runs the business and how long it has operated, if you are happy to share it
For local searches, Google's local ranking help page (opens in a new tab) names three factors: relevance, distance and prominence. It says businesses with complete and accurate information are more likely to show in local results, and that more reviews and positive ratings can help local ranking. AI assistants do not publish how they weigh reviews, so treat genuine reviews as one useful signal, never something to buy or fake. Our guide to Google Business Profile and local SEO goes into detail.
Make the important pages crawlable and readable
Crawlable means a crawler is allowed to request a page and can read its content. Indexed means the search engine has stored the page and can show it. A page can be crawlable but not indexed, so check both.
- Check robots.txt and remove any rule that blocks a search crawler you want, including Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot.
- Check that key pages do not carry a stray noindex tag, a login wall or a cookie wall that hides the content.
- Put core facts in the HTML your server sends, so any crawler can read them whether or not it runs JavaScript.
- Do not put essential information only in images, videos or PDFs; repeat it in page text.
- Give each service, location and common question its own page with a descriptive title and a direct opening answer.
- Submit an up-to-date sitemap in Google Search Console and Bing Webmaster Tools.
A practical checklist
- Decide your policy: allow search crawlers, and choose separately whether to allow training crawlers.
- Write the robots.txt rules to match, then check them with the robots.txt report in Search Console and the tools in Bing Webmaster Tools.
- Confirm your main pages are indexed in Google and Bing.
- Open each key page with a two or three sentence answer to its main question.
- Audit your business details on your site, Google Business Profile, Bing Places, social profiles and the directories in your sector, and fix every mismatch.
- Ask satisfied customers for honest reviews, and reply to reviews helpfully.
- Keep structured data accurate and matching visible content; do not add markup for things the page does not show.
- Ask the assistants about your business each month with a fixed set of prompts, and log what they say and cite.
- Repeat the crawler and indexing checks after any redesign or platform move, because robots.txt and noindex settings are easily reset.
What you cannot control
You cannot control the wording of the question, the user's location or history, or when an AI company changes its systems. Two people asking the same thing can be shown different businesses. That is why the useful measure is a trend across a stable set of prompts, as described in our guide to generative engine optimisation, not a single screenshot.
Our free website and AI search check reads your public home page and reports technical checks, an AI-generated summary of the page and suggested fixes. If you want a full AI visibility audit, our SEO and AI visibility service can help.
Frequently asked questions
Does blocking GPTBot remove my website from ChatGPT?
Not from ChatGPT search. OpenAI treats GPTBot, which is for model training, and OAI-SearchBot, which is for search, as independent settings. If you block OAI-SearchBot, your site will not be shown in ChatGPT search answers.
Does Google-Extended affect my Google rankings?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. It controls whether content can be used to train future Gemini models and for grounding in some other Google systems.
Do I need special schema or llms.txt for Google AI Overviews?
No. Google says there is no special schema.org markup and no AI text file needed for AI Overviews or AI Mode. A page needs to be indexed and eligible to show in Search with a snippet.
Why does ChatGPT give wrong information about my business?
Usually because it is relying on model memory, or because the pages it found disagree with each other. Make sure your current details are on crawlable pages, OAI-SearchBot is allowed, and your listings elsewhere match your website.
How do I get Microsoft Copilot to mention my business?
Copilot grounds its web answers by sending queries to the Bing search service, so start with Bing: allow Bingbot, verify your site in Bing Webmaster Tools and check indexing. Its AI Performance report shows citations in Copilot. Nobody can guarantee a mention.
Sources
- Overview of OpenAI crawlers
- Publishers and developers FAQ (OpenAI Help Center)
- Google's common crawlers (Google for Developers)
- AI features and your website (Google Search Central)
- Data, privacy, and security for web search in Microsoft 365 Copilot and Microsoft Copilot (Microsoft Learn)
- Introducing AI Performance in Bing Webmaster Tools Public Preview (Bing Webmaster Blog)
- Perplexity crawlers (Perplexity docs)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? (Claude Help Center)
- Tips to improve your local ranking on Google (Google Business Profile Help)
Checked on 23 October 2026. Platforms and rules change; check the source before acting on anything time-sensitive.