Web-based measurement: how we scan AI chatbots

Below, we explain why we chose web search, what a single run looks like, and what the limitations of this method are.

Does Aelo measure AI chats against web searches?

Yes, in all four models and from the very first scan. Every question from frozen query set We send requests via the API with the search tool enabled, which each provider makes available to developers: web search on OpenAI and Anthropic, grounding via Google Search on Gemini, and on Perplexity, search is an integral part of every response.

The questions are neutral, i.e. they do not mention any brand names. That is the rule a neutral scan, which we describe in more detail in methodology. The model must specify the brand itself, and we are counting how many times it did so in its responses, in which positions, and which websites it cited in doing so.

Why does the measurement without a search only show part of the image?

Because without a search query, the model responds solely based on the knowledge acquired during training, and this only changes with a new version of the model. Such a measurement will show whether the brand was accurately represented online before the model was made available to users. It will not indicate whether the changes made to the website over the last month are being reflected in the model.

Meanwhile, users are increasingly turning to versions with a search function. OpenAI added a search function to ChatGPT in October 2024, Anthropic added it to Claude in 2025, Gemini uses Google’s search engine, and Perplexity has been a search engine with answers from the outset. Measuring knowledge derived solely from training would mean measuring something different from what the customer actually sees. We describe the brand selection mechanism in more detail in the text How AI chatbots choose the brands they recommend.

What does a single scan cycle look like?

Like a sequence of the same steps, repeated with every measurement. Each phrase, which you are monitoring, is broken down into three questions asked in the same way a customer would ask them. We freeze this set of questions at the first scan so that subsequent measurements can be compared.

We calculate each metric separately for each model. You can find the definitions in the glossary: reference, proportion of responses, citation index and sentiment.

Why is the tracking done via the API rather than via chat apps?

This is because, via the API, every query is sent to the model under the same conditions. The chat app remembers previous conversations, tailors responses to the logged-in user and takes their location into account. Two people asking the same question may therefore receive different lists of brands for reasons that have nothing to do with the brand itself.

The API does not have these variables: it does not know the chat history or the user’s account. We set the location ourselves, according to brand identity. If you fill in the geolocation field, the model searches the results for that location, so a local business may appear in local search results. When the geolocation field is left blank, the search does not specify a city, but rather the language and market based on Brand identity. Consequently, results from subsequent scans can be compared, and any change says something about the brand rather than about the person making the query. The same principle underpins the recommendation that when manual test Ask in a new conversation or via private message.

What did the search-based analysis reveal in the case of the lime manufacturer?

He demonstrated that updates made on the website are reflected in the models within weeks, rather than only with the release of a new model version. The „Węże” limestone quarry did not feature in any of the 18 questions in the survey conducted on 24 June 2026. After working on their own website, the models mentioned it in 17 out of 18 questions on 27 July and in 18 out of 18 questions on 10 September. Citations During this period, his domains increased from 6% to 50%, and then to 94%. A full description of the initial measurements can be found in the text From 0 to 17 mentions on AI in five weeks.

Such a rapid shift in citations would not have been apparent in a measurement without a search, as none of the models had acquired new knowledge from training during that period. The September measurement also showed how differently the models utilise the web: the brand was mentioned by Claude and Perplexity in 1,00% responses, by ChatGPT in 56%, and by Gemini in 44%. We describe the data in research.

What does this measurement not show?

• It does not show the response that a specific user will see in the app, based on their chat history and account. We derive the location from Brand identity, not from the user’s device.
• It does not provide a single, fixed answer: when faced with the next question, the model may search other sources, which is why the trend observed across several scans is what matters, as we explain in the text How often should visibility be measured?.
• It does not say which pages the model searched through and which it did not use. We can see the pages it referred to in its response.
• It does not guarantee a recommendation: the model may mention the brand’s website and still not recommend it, as we explain in the text Knows the brand, but doesn’t recommend it.

What does ‘measurement with search’ mean for your website?

That the page must be accessible to the bots used by AI chatbots at the time the query is made. OpenAI uses the OAI-SearchBot for searches in ChatGPT, Anthropic uses the Claude-SearchBot and Claude-User bots, Perplexity uses the PerplexityBot, and Gemini uses Google Search. If the robots.txt file blocks them, the model may still recognise the brand from its training, but it will stop citing the website in its search responses.

The second condition is that the content must be quotable: the answer in the first sentence of the section, specific figures and conditions, structured data which describe the company. That is what it’s all about AEO, that is, part of a broader GEO. If the model does not mention the brand at all, start with the text ChatGPT doesn’t know my company, and when he mentions the competition, from the text Why does AI recommend the competition?. The effects are delayed, as explained on the website Why do changes take time?.

How does Aelo help with web search analytics?

It does it for you at a steady pace. Module Visibility in AI It sends a fixed set of questions to four models using web search, records the answers and the web pages cited, and displays the results separately for each model. If the assessment reveals a gap, the task is forwarded to Tickets, and you can prepare the content below it in Studio. The way the model describes the brand garners Synapsis. We compare the plans in pricing.

Agencies that track multiple clients can find out more on the website Aelo for agencies, and companies that manage their own websites, on the website Aelo for small businesses.

Try it for free

FAQ

Sources

• OpenAI, web search tool, API documentation: https://developers.openai.com/api/docs/guides/tools-web-search
• OpenAI, Introducing ChatGPT search (October 2024): https://openai.com/index/introducing-chatgpt-search/
• OpenAI, Overview of OpenAI Crawlers, the OAI-SearchBot and GPTBot bots: https://developers.openai.com/api/docs/bots
• Anthropic, web search tool, API documentation: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
• Anthropic, Claude can now search the web (March 2025): https://claude.com/blog/web-search
• Anthropic, the ClaudeBot, Claude-User and Claude-SearchBot bots: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Google, Grounding with Google Search, Gemini API: https://ai.google.dev/gemini-api/docs/google-search
• Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/guides/bots
• Aelo reports for Wapiennik „Węże” dated 24 June, 27 July and 10 September 2026 (published with the client’s consent)