Ask ChatGPT the same question twice and you will usually get two different lists. That is by design. The model picks its words by probability, and when it searches the web it can pull different pages each time. For a business, this means one test proves almost nothing. What you can measure is how often you're named across many repeats. A handful of firms in each market are named nearly every time, and the reasons are visible.
In mid-October we tested this ourselves. We asked ChatGPT and Google's AI Mode 15 questions a business owner might ask when hiring an IT firm, in cities from Long Island to Houston. We asked most questions five times on each, which gave us 106 answers. We have since extended that to 10 runs each on ChatGPT, Google AI Mode, Gemini and Claude; what we found testing ten IT firms on four engines is on our MSP page.
The short version
The list changes almost every time. Independent research puts the odds of ChatGPT repeating the same list of brands at under 1 in 100.
But it doesn't change at random. In 6 of the 7 markets where ChatGPT named firms at all, one or two firms showed up in at least 4 of 5 answers. Everyone else rotated.
Big markets scatter, small ones settle. In one large Texas metro, ChatGPT named 13 different firms across 5 answers. On Long Island, both platforms together named only 5 across 8.
Sometimes nobody gets named. For one law-firm IT question, ChatGPT gave general advice and named no firms in 4 out of 4 tries.
One test is misleading. We saw a firm named third in one answer and missing from the next.
The firms named nearly every time had things in common. 8 of 11 had 60 or more Google reviews, and most had a page for each town they serve. Directory listings mattered less than we expected.
Does ChatGPT give the same answer to everyone?
No, and not to the same person twice either.
The largest public test of this was run by SparkToro and Gumshoe.ai. In their January 2026 study, 600 volunteers ran 12 prompts through ChatGPT, Claude and Google's AI 2,961 times. There was less than a 1 in 100 chance that ChatGPT or Google's AI would return the same list of brands twice. The odds of the same list in the same order were closer to 1 in 1,000.
Local searches behave the same way. BrightLocal's September 2026 study repeated about 200,000 local searches over 60 days. For the same prompt, ChatGPT's result sets overlapped by only 23% on average. Google AI Mode's overlapped by 21% and AI Overviews by 33%. Google Maps, for comparison, agreed with itself about 99% of the time.
So if a customer told you "ChatGPT recommended you," that tells you about one answer, not about the next customer's.
What we saw when we asked the same question over and over
Our questions were the kind a buyer would ask, such as "I run a small accounting firm in South Florida, who should handle our IT?" Where we could, we checked that real people ask these questions, using Keyword Planner volume, Google Autocomplete or "People also ask". We ran ChatGPT in temporary, unpersonalised chats so past conversations couldn't steer it. We ran Google's AI Mode with US settings. Then we wrote down every firm each answer named.
Five patterns came up again and again.
1. Repeat lists were rare. In one Wisconsin market, two answers in a row named the same four firms. Over the next three answers, five firms we hadn't seen before showed up. That was the closest we came to a repeat.
2. Each market had a stable core. In 6 of the 7 markets where ChatGPT named firms, at least one firm was named in 4 or 5 of the 5 answers. In upstate New York, one firm was named in 4 of 5 ChatGPT answers and all 5 Google answers. On Long Island, one firm was named in all 5 ChatGPT answers. Around the core, the other spots changed from one answer to the next.
3. Large markets scatter. In a large Texas metro, ChatGPT named 13 different IT firms across 5 answers, and Google named at least 8 across 4. Long Island was the opposite: only 5 firms in total, two of them named just once. SparkToro saw the same split. Small categories stayed fairly stable, while large ones spread out.
4. Sometimes the answer has no names in it. Asked which firm should run IT for a law practice in the Chicago suburbs, ChatGPT gave a checklist of what to look for in 4 out of 4 tries, without naming a single provider. Google's AI Mode did the same for a general small-business question in a western Chicago suburb, 5 times out of 5. For a Pittsburgh law-firm question, ChatGPT named no one in 2 of 5 answers. An answer with no names is an open slot. Nobody has given the model enough to be confident about.
5. One answer can mislead you either way. A firm north of Chicago was named third in one ChatGPT answer and missing from the very next one. Check once and you would have called it a win. Check again and you would have called it a gap. Neither result tells you how often it's named.
Why do the answers keep changing?
There are three reasons, and only one is under your control.
The model samples. A language model writes by repeatedly picking a likely next word, not always the single most likely one. That is what makes answers sound natural, and it means two runs can go down different paths from the first sentence.
The search underneath changes. When ChatGPT or Google's AI answers a local question, it looks things up on the web: business profiles, directories, review sites and company websites. Each run can pull a different mix of pages, and the answer is built from whatever came back. The two platforms also read different sources, which is why ChatGPT and Google's AI Mode often name different businesses.
Context changes. Location, memory of past chats and how the question is worded all shift the answer. BrightLocal found that even moving around inside the same town changed things: ChatGPT's results overlapped only 36% across points in one town. We removed what we could by using temporary chats and fixed wording. Your customers won't.
The first and third you can't control. The second you can. The pages and listings the model finds are the raw material for its answer. Firms that turn up in most of those sources keep getting named, however the dice roll.
If the list keeps changing, what should you measure?
Count how often you're named, not where you rank. A "position" in an AI answer means very little when the order changes almost every time.
SparkToro's data shows why. One West Coast cancer hospital was named in 69 of 71 ChatGPT answers, which is 97% of the time. It was the first name in only 25 of them. Measured by position it looks like it's moving around. Measured by mentions it's the clear winner.
BrightLocal found that 71–80% of local businesses showed up in half their searches or fewer. Being named "sometimes" is normal. What matters is whether you're in the core or in the rotation.
How many times do you need to ask?
It depends on what you want to know.
For a precise percentage, you need a lot of runs. SparkToro recommends at least 60 to 100 runs of a prompt before averaging. Nobody does that by hand, and it's why tracking tools exist.
For a yes/no gap check, five runs tells you a lot. Basic probability: if a firm were named in half of all answers, the chance it would miss five in a row is 1 in 32, about 3%. So 0 out of 5 means you're very probably named less than half the time. What five runs can't tell you is whether you're named 1 time in 10 or never. To answer that, you need more runs or more questions.
That's why we report "0 of 5" and never "you're invisible on ChatGPT".
What did the firms named nearly every time have in common?
We took the 11 firms that AI kept naming and checked what the model could find about each one. We compared them with the firm we had tested for, which was missing from the same answers. Here is what we found:
Reviews made the biggest difference. 8 of the 11 firms that kept getting named had 60 or more Google reviews. The middle count was 64, and the top one had 220. Four of the six firms that were left out had 4 reviews or fewer.
But reviews alone didn't decide it. Three of the firms that kept getting named had only 12 to 21 reviews. Two firms with 44 and 46 reviews were left out of answers they should have been in. In both cases they were missing what the question asked for. One had no page for the niche in the question (accounting firms). The other was based in a nearby town and didn't appear in the top Google Maps results for the town we asked about.
Most had a page for each town they serve. 7 of the 11 had separate pages for the towns or counties they cover, from 3 to 25 of them. In both the Houston and Long Island markets, the firm named in nearly every answer had over 20 town pages and more than 60 reviews.
A page for the niche helped, even without the exact city. In South Florida, both firms named for an accounting-firm question had a page on IT for accounting firms in South Florida. Neither page named the city in the question. In Pittsburgh, the firm named in every Google answer had a law-firm IT page and a case study about Pittsburgh law firms.
Directories mattered less than we expected. Only 4 of the 11 had any reviews on Clutch, the main directory for IT firms. Just one sat at the top of its city's Clutch list.
Ten of the 11 had at least one of the two: 60 or more reviews, or pages for the towns they serve. The odd one out was named in the same market as a firm with 64 reviews.
There was one clear exception. A firm in the Chicago suburbs was named in 4 of 5 answers even though its website doesn't name a single town. It does have 64 Google reviews and local chamber of commerce listings. When your own site says little, what others say about you can still get you named.
This is a pattern from 11 firms on one day, not proof of cause. AI answers also draw on sources we didn't check, such as Yelp, the Better Business Bureau and the news. But it fits BrightLocal's study: Google Business Profiles were the single biggest source behind local AI answers, at 28.6% of citations. Business websites made up 42%.
The lesson is plain. The model names the firms it can most easily justify: plenty of reviews, and pages that match the question's town and type of business.
We covered the single-snapshot version of this test for IT firms here, and the local SEO checklist covers the profile and review side in detail.
How to check your own business properly
Pick three to five buyer questions you can show real demand for. Use Keyword Planner, Google Autocomplete or "People also ask", not the words you'd use about yourself.
Ask each one five times on ChatGPT, in temporary chats, and five times on Google's AI Mode. Change nothing between runs.
Write down two things for each answer: whether you were named, and every other firm that was. Your result is "named in X of 5", not yes or no.
Find the core. Any firm named 4 or 5 times out of 5 is the one to study. Look at its reviews, its directory listings and the pages on its site that match the question.
Repeat on the same day each month, with the same questions in the same words. One month's result shows where you stand. The trend over several months shows whether your changes worked.
If the answer named nobody, don't relax. That spot is open, and the first firm to give the model something solid to point to will probably take it.
Want this done for your market?
Repeated-run testing is how every AI Search Visibility engagement starts. We run your buyer questions on both platforms, report how often you're named compared with the firms that keep showing up, and explain what their pages and listings have that yours don't. The first audit is free, and you get a written report rather than a sales call.
This article is written by
Founder & Lead Strategist at CognitiveAISEO. Marvin has spent 20+ years in search and 4 years as an account manager at Lenovo, managing a book of 500+ partner accounts: resellers, MSPs and integrators. His posts draw on CognitiveAISEO's own tests: the same buyer questions asked 10 times each on ChatGPT, Google AI Mode, Gemini and Claude.
