Key takeaways
- ChatGPT does not look up a fixed answer. It generates text and writes its own web searches each time, so the cited sources change between runs.
- Query fanouts are the biggest driver of citation variance. One prompt can trigger several searches, and the sub-queries differ from run to run.
- Answers are crowded. Promptwatch Data shows 47.6% of ChatGPT responses cite 10 or more domains, so one run tells you very little.
- Memory, model choice, phrasing and content freshness all add more variation on top.
- Stop treating a single run as truth. Measure across many prompts, repeated over time, and watch trends instead of snapshots.
The short answer
You ask ChatGPT "what's the best project management tool for a small agency?" and get five brands with a handful of cited sources. You ask again ten minutes later. Two brands swap places, one disappears, and the sources are mostly different.
Nothing is broken. This is how the system works. If you do SEO or brand work, it matters a lot, because it means the screenshot your boss sent you with "we're not mentioned!" is one draw from a distribution, not a verdict.
The rest of this guide covers where the randomness comes from, how big it is, and what to do about it.
Why the text itself varies
ChatGPT does not pull answers from a database of pre-written responses. It generates text token by token, based on probabilities learned in training. Several things shape what comes out.
Sampling and temperature
Language models pick the next word from a probability distribution. A setting called temperature controls how adventurous that pick is. Low values give focused, predictable output. High values give more varied output. As ScaleMath explains, you can only change temperature through the OpenAI API, not in the ChatGPT web interface. On the web you get whatever OpenAI configured, and it is not zero.
Even at low temperature, outputs are not guaranteed to be identical. Floating-point math, batching and infrastructure details can nudge results. So "will ChatGPT give the same answer twice?" has a boring answer: it might, but nothing guarantees it.
You will see a claim floating around that ChatGPT is consistent only 73% of the time over 10 asks. That figure comes from a MultipleChat page with no primary study behind it. I would not quote it. The direction is right, the number is unsourced.
System prompts and product versions
Free, Plus and custom GPT users all sit behind different base instructions. Same question, different wrapper, different answer. This is mostly a tone and formatting issue, but it also changes how willing the model is to search the web at all.
Why the citations vary even more than the text
Text variance is annoying. Citation variance is bigger, because citations come from a separate step: web search.
Query fanouts
When ChatGPT decides to search, it rarely uses your prompt as the search query. It breaks the prompt into several searches, called fanouts, each aimed at a different angle. Promptwatch Data's query fanout report says a single prompt can trigger 3 to 8+ searches.
Those sub-queries are themselves generated by the model, so they differ every time. Run one: "best crm for small business", "hubspot vs pipedrive pricing", "crm for startups reddit". Run two: "top crm software 2026", "easiest crm to set up". Different searches return different pages, so different pages get cited.
A few details from the same report are worth knowing:
- Average fanouts per response fell from 2.15 in early December to about 1.84 by early March.
- After a gap in the data, they came back in April at exactly 1.0 per response.
- Average query length dropped from about 117 characters to about 53 once fanouts returned in April.
So the mechanism itself changes over time. And on August 8, 2026, ChatGPT started using the site: operator at scale, which jumped from about 0.4% to about 17% of fanout queries overnight, with searches per response nearly doubling, according to Promptwatch Data. I only have the summary of that report, but the direction matters: the way ChatGPT searches is not stable, so neither are the sources.
The practical upshot is that your page only has to match one of the sub-queries, not the user's literal prompt. That is why headings that read like short search queries, and coverage of side questions like "X vs Y", pricing, alternatives and how-tos, tend to help.
Crowded source lists
When a response cites a lot of domains, small shifts in the search step reshuffle who makes the list. Promptwatch Data (Sep 28 to Oct 4, 2026, responses with at least one citation) found:
| Prompt type | Cite 10+ domains | Other notable figure |
|---|---|---|
| All responses | 47.6% | 66.1% cite at least 8; only 6% cite 3 or fewer |
| Organic (no brand named) | 52.5% | 71.8% cite at least 8 |
| Brand-specific | 21.3% | Most common count is 5 domains (14.9%) |
| Competitor comparison | 49.1% | Only 0.35% cite a single domain |
With that many slots and that many candidate pages, expecting the identical list twice is unrealistic. Brand-specific prompts are the stable end of the range, since there are fewer plausible sources for a question about one named company.
Freshness churn
Some of the difference between your Tuesday run and your next Tuesday run is not randomness at all. The cited set really does move. Promptwatch Data tracked pages ChatGPT cited on July 6, 2026 for 90 days:
| Content type | Still cited after 30 days | Still cited after 60 days |
|---|---|---|
| Landing pages | 78% | 69% |
| Docs | 76% | 67% |
| Comparisons | 72% | 63% |
| Listicles | n/a | 60% |
| Product pages, reviews, how-tos | n/a | 55% (half-life 74 to 77 days) |
| News | half gone by day 36 | 41% |
Almost a quarter of news pages were never cited again after day 0. If your topic is news-heavy, expect the biggest swings. If it is built on docs and landing pages, citations are stickier.
Why two people get different answers to the same question
Even if you could freeze the randomness, you and a colleague are not using the same ChatGPT.
- Memory and conversation history. Prior chats and saved memories change how the model reads your intent. A question asked inside a long thread about enterprise software is not the same question asked cold.
- Model routing. As Obsero notes, a complex research-style framing can trigger a reasoning model such as GPT-5.4 Thinking, which synthesizes more sources and may surface different brands. I have not verified that independently, but it matches what practitioners report.
- Phrasing. "Best CRM" and "which CRM should a 10-person agency use" are different searches after fanout. Small wording changes lead to large citation changes.
- Location and language. Results vary by country, which is why geographic tracking exists in most serious tools.
- Time. The model, the search index and the web all change. Model updates matter too: average citations per response dropped across GPT-5.3, GPT-5.4 and GPT-5-Mini around the March 4, 2026 rollout, per Promptwatch Data.
A video from FTA Global's Search Engineering Masterclass frames it well: traditional SEO ranks pages, AI search builds answers. When the answer is constructed fresh from context, retrieval and source confidence, there is no single result page to rank on.
Fake citations are a separate problem
People on Reddit and Quora often lump "different citations every time" together with "ChatGPT made up a source". They are different issues.
When ChatGPT answers from memory without searching, it can produce references that look like real citations but are not. It imitates the format of a citation without checking it exists. When ChatGPT uses web search, the links come from actual search results, which is why they are usually real but vary. If a citation has no clickable, live URL, verify it manually before trusting it.
What this means for measuring AI visibility
Here is where I think many teams go wrong. They run a prompt, see their brand missing, panic, run it again, see it present, relax. That is gambling, not measurement.
A better approach:
- Track many prompts, not one. Promptwatch's own advice from the unique-domains data is to measure across many organic prompts, because individual runs are noisy.
- Repeat runs over time. Treat each run as a sample. Report the share of runs where you appear, not whether you appeared once.
- Track the metrics that survive noise: visibility rate, share of voice, average position, and how you are described.
- Separate prompt types. Brand-specific prompts are relatively stable. Organic prompts are volatile. Do not average them together.
- Capture the fanouts. If you can see the sub-queries ChatGPT generated, you can see why a source was or was not picked.
- Watch trends, then annotate model changes. A sudden shift across all your prompts on one day is usually a platform change, not your content.
- Look at the real interface. API outputs and the consumer ChatGPT UI can differ, so tools that test the real UI give you a closer picture of what users see.
Tools that help
A spreadsheet and manual prompting will not give you enough samples. Dedicated trackers run prompts on a schedule and aggregate the results. Promptwatch is the one I would start with, because it tracks real UI responses, shows prompt volumes and query fanouts, and ties citations to crawler logs and traffic so you can see what to fix, not just what moved.
Other options worth a look, depending on budget and focus:
A rough comparison:
| Tool | Strength for noisy-citation problems | Watch out for |
|---|---|---|
| Promptwatch | Real UI tracking, fanouts, crawler logs, content agents to act on gaps | More depth than a solo marketer may need |
| Profound | Enterprise monitoring and analytics | Higher price point |
| Peec AI | Simple prompt tracking, easy to start | Mostly monitoring |
| Otterly.AI | Affordable brand monitoring | Basic feature set |
| AthenaHQ | Visibility monitoring | Monitoring-focused |
| Ahrefs Brand Radar | Convenient if you already use Ahrefs | Fixed prompt sets, shallower AI data |
For more options, the GEO software directory at bestgeosoftware.com lists far more trackers than fit in one article.
How to get cited more consistently
You cannot make ChatGPT deterministic, but you can raise the odds that you are in the pool on most runs.
- Write for sub-queries. Cover the adjacent questions: comparisons, pricing, alternatives, how-tos. Each is a fanout you can win.
- Use clear, query-like headings. They match the short searches ChatGPT generates.
- Pick durable content types. Landing pages and docs held citations longest in the half-life data. News fades fastest, so refresh it or pair it with evergreen pages.
- Refresh important pages. Citations churn over weeks, so stale pages drop out.
- Check that AI crawlers can read your site. If bots hit errors or blocked pages, you are out of the pool no matter how good the content is.
- Be present on third-party pages AI cites. With 10+ domains per answer common, being named on other sites counts.
A simple repeat-run test you can do today
Before buying anything, run a small experiment to feel the variance yourself.
- Pick one organic prompt with no brand in it.
- Open a fresh chat with memory off or a temporary chat.
- Run it 10 times, a few minutes apart.
- Log which domains and brands appear each time.
- Count how many appear in at least half of the runs.
You will likely find a small core of sources that show up repeatedly and a long tail that rotates. Your goal is to move from the tail into the core. That core is what a tracker measures at scale.
Gaps worth knowing about
Two things I could not back with a primary source: OpenAI's own documentation on non-determinism, and independent repeat-run overlap studies for citations. Treat claims about exact consistency percentages as folk numbers until someone publishes the method.
Bottom line
Different citations on every ask are the expected behavior, caused mostly by freshly generated fanout searches, crowded source lists and shifting content, with memory, model routing and phrasing layered on top. Stop judging visibility from one run. Sample many prompts, repeat them, and track share across runs. For wider AI search patterns, the Promptwatch Data hub is a good place to check how citation behavior is changing month to month.