The short answer
Does AI retrieval work the same way across languages? No. It is close to fully separated. The same commercial question asked in seven European languages returns almost entirely different sources.
How separated? A question agrees with its own rerun on 72% to 74% of cited domains. It agrees with the same question in another language on 4% to 11%. That gap is at least six times the noise in the instrument, and it held across two full passes run days apart.
How wrong was I? Completely, on the mechanism. I expected thin-corpus languages to fall back to English sources and .com. The opposite happened, and the language that most contradicts me is the thinnest one in the set. That part of the result barely moved between passes.
What this cost to find out: 1,344 answers, two passes, 10,329 citations, and one bug in my own analysis code that had the headline number wrong by 14 points until the second pass exposed it.
The core result: retrieval is separated by language
Both passes, run days apart, measured independently.
measured range, both passesnoise floor
Each bar spans the two measurements rather than averaging them. The gap between every language and the dashed line is the finding, and it is the part of the chart with nothing drawn in it.
Read the first row against the rest. A question agrees with itself on roughly three quarters of its domains. It agrees with its own translation on well under a tenth.
Every language is at least six times further from English than English is from itself, and in the widest case eighteen times. There is no version of this measurement, in either pass, where any language comes close to the floor.
Between 675 and 707 of the roughly 750 domains appeared in exactly one language. Three appeared in all seven in both passes.
Those three are appliancepartspros.com, repairclinic.com and siemens-home.bsh-group.com. Two US-centric parts retailers and one manufacturer. Everything else is local to a market.
Cross-market AI visibility is not one problem with seven translations. It is seven problems that share three domains.
Why the ranges, and why they are the honest form
Every per-language overlap moved between passes, all six in the same direction, all by roughly the same amount. That is worth explaining rather than hiding in an average.
At overlaps this small, the two sets share a handful of domains. A shift of three or four domains out of 120 changes the ratio by half while changing almost nothing about the finding. The absolute movement was between 0.026 and 0.043. The relative movement looks alarming and is not.
What did not move: every language stayed far below the floor in both passes, and the ordering of the smallest and largest barely changed. What did move: the exact value, and the middle of the ranking. So the ranges are what gets published, and anyone quoting a single decimal from this study is quoting noise.
The number everything hangs on, and the mistake I made computing it
Generation at temperature zero is close to deterministic. Retrieval is not. The search runs fresh, the index underneath moves, and the same prompt sent twice returns a meaningfully different set of pages.
So every cell was asked three times, and the first thing the analysis computes is how much a cell disagrees with itself.
Across 224 repeated cells in each pass, two runs of the same question, in the same language, on the same model, at temperature zero, agreed on 0.737 of their cited domains in the first pass and 0.720 in the second.
About a quarter of the domains turn over between two identical requests.
Without that number none of the language figures mean anything. A study that measures each cell once and reports a difference is reporting a number it cannot distinguish from its own instrument.
The bug
The first version of this analysis reported a floor of 0.601. That was an artifact of my own code and I would not have caught it without the second pass.
124 of the 672 repeat pairs in pass one were cases where neither run cited anything. Every one was GPT, whose search fires on a minority of calls. A naive Jaccard divides by an empty union and returns zero, so those pairs were scored as total disagreement between two measurements that never happened.
That is not a noise floor. It is rerun churn averaged with a count of how often one model declined to search, and it moved the headline number by 14 points.
The correction makes the finding stronger, from a 5.3x margin to 6.3x, which is exactly why it nearly survived. A number that flatters you is the one you check last. The analyser now excludes those pairs and prints the exclusion next to the figure so it cannot be quoted without it.
What a second pass measures that repeats cannot
The three repeats inside a pass measure how much the index wobbles minute to minute. Running the whole study again days later measures whether it moved. Those are different questions and only the second one tells you whether a figure published today means anything next week.
| Same cell, same day, different run | 0.737 and 0.720 |
| Same cell, days apart | 0.684 |
A question agrees with itself less across days than within a day, by about five points. The index moved, and it moved in a measurable, reportable amount.
I have not found this figure published anywhere in the AI visibility literature. It is the number every vendor claim should be measured against, because it says how large a change has to be before it is a change at all rather than the index breathing.
If your tracker reports a week-on-week movement smaller than this, it has not measured anything. Ask what their equivalent number is. Most cannot answer, because producing it costs a second full run and nobody budgets for measuring their own instrument.
The surprise: my hypothesis was wrong, and that held
I went in expecting corpus depth to drive the result. Thin corpus, fewer native sources, fallback to English and .com. It is the intuitive model and it is what the training-data argument predicts.
local country domain.com
Italic labels are my own estimate of training-corpus depth, which is the variable this was built to test. Poland and Portugal carry the same label and sit at opposite ends. Bars are pass 2; pass 1 read 83.9, 59.3, 59.6, 54.7, 33.2, 29.3 and 12.7, so nothing here moved by more than three points.
Poland and Portugal are both thin-corpus languages and they sit at opposite ends of the table. Around 84% against around 28%. If corpus depth drove local anchoring they would be neighbours. They are as far apart as the table goes, in both passes.
Note how little this table moved. Every market is within three points of itself across two passes days apart, and Poland moved by one tenth of a point. This is the most stable thing in the study, which is the opposite of what I expected from the figure that contradicted me.
So the mechanism is not corpus depth. My reading is that it is market structure: Poland has strong domestic e-commerce in this category, and the index reflects the market it indexes rather than the size of the language that describes it. That is a hypothesis this study cannot test, and I would rather name it as a guess than promote it to a conclusion the way I nearly did with the first one.
This is a better result than the one I expected. “Corpus size predicts fallback” would have been tidy, with an obvious remedy, and wrong. “Market structure predicts local anchoring” says the work has to be done market by market, which is less tidy and more useful.
The control worked, and it caught something
Two of the four models were included as a control rather than as subjects, and the control is the reason to trust the rest.
Claude and Gemini have no native web search through OpenRouter. Probed directly they return zero citations: zero on native, zero with no plugin, zero for Gemini even on the :online suffix. Routed through Exa, both return ten citations every time. So the study was split into two arms.
| Arm | Models | What it is |
|---|---|---|
| assistant | GPT with native search, Perplexity with built-in search | The study. What a person using that assistant sees |
| index | Claude and Gemini, both through Exa | The control. One index, two summarisers |
| Pair | Pass 1 | Pass 2 | |
|---|---|---|---|
| Claude / Gemini, both through Exa | 0.827 | 0.804 | above the noise floor, both passes |
| GPT / Perplexity, both searching for themselves | 0.067 | 0.058 |
Claude and Gemini agree with each other more than a question agrees with its own rerun. Two models, two labs, agreeing above the level of self-agreement is not convergence. It is one measurement reported twice. Exa retrieves and two models summarise what it returned.
Two models that actually search for themselves agree on about 6%.
The control was built to let the analysis show that convergence rather than assert it, and it did, in both passes, at the same magnitude.
It generalises past this study. A great many AI visibility trackers are built on OpenRouter or a comparable aggregator, and any of them serving Claude or Gemini through a third-party index is not measuring Claude or Gemini. It is measuring that index, and putting the model’s logo on the chart.
The test costs ten minutes on your own dashboard. Ask two models the same question, then ask one of them the same question twice. If the two models agree with each other more than either agrees with its own rerun, you have one data source wearing two hats.
Two columns I have closed off, so they cannot be read wrong
Cites/answer is meaningless here and should not be quoted. Claude and Gemini return 10.00. That is Exa’s max_results ceiling, not model behaviour. The analyser prints the reason next to the number rather than leaving it to be misread:
pinned at the max_results ceiling, so this number is the config
Any per-language average built on that column inherits the problem, because it mixes two models pinned at a ceiling with one whose search fires a fraction of the time.
Which domains get cited is the measurement. How many is not.
GPT’s zeros are real, and they are a decision, not a language property. GPT-5.2 with OpenAI’s own search enabled returned no citations on 76% of calls in the first pass and 86% in the second. That is not a collection failure. The index arm answered the same questions in the same languages with citations nearly every time, so the questions were answerable and the sources existed. OpenAI’s system decided not to search.
Note the size of that move: the share of calls where GPT searched at all fell from 24% to 14% between two passes days apart. Whatever governs that decision is not stable, and it is not something a site owner can influence.
For a brand whose buyers use ChatGPT this is the finding with money attached. Most of the time there is no retrieval step to be visible in, and what the model says about your category comes from training rather than from your website. That is an entity and reputation problem, which is a different budget line from technical GEO.
One table I do not trust yet
Search-fire rate broken down by language looks like a finding and is not one yet.
| UK | DE | FR | ES | IT | PL | PT | |
|---|---|---|---|---|---|---|---|
| GPT search fires, pass 1 | 33% | 25% | 25% | 25% | 25% | 13% | 21% |
Poland at 13% against the UK at 33% is exactly the kind of number that gets screenshotted. It rests on 24 calls per language. Thirteen percent is three calls out of twenty-four and thirty-three percent is eight.
Three against eight is a difference you can get by asking on a different afternoon, and the pass-level rate moving from 24% to 14% between passes says the whole column is unstable. The direction may be real and it is consistent with everything else here, but it needs more questions than this study asked before anyone quotes it.
I am including it because leaving it out would be worse. Someone will compute it from the published dataset within a day, and it is better that the first person to state its sample size is me.
What to do with this
1. Stop reporting AI visibility as one number. A single cross-market score averages seven retrieval environments that share three domains. Ask your tracker for per-language domain sets and look at the overlap yourself.
2. Audit your tooling’s plumbing before you trust its output. Ask which retrieval path each model uses. If two models agree suspiciously well in your own dashboard, run the rerun test above. The AI visibility checklist covers what to ask a vendor.
3. Work market by market, not language by language. The strongest predictor of local anchoring here was market structure, and it was the most stable figure in the study. That means local commerce presence, local citation sources, local retailers, and it means the English playbook does not transfer even to markets whose language is well represented in training.
4. For ChatGPT, assume the search step is usually absent. Between 76% and 86% of calls here returned no citations at all. Retrieval-time optimisation only pays when retrieval happens.
5. Do not block the crawlers you want. Everything above assumes the retrieval bot could reach the page. On multi-market platforms a single country domain quietly blocking a retrieval bot is a visibility hole no content work will close, and it is the most common misconfiguration I find. Audit robots.txt market by market against an inventory of AI crawlers.
6. Measure your own noise floor, and then measure it again next week. Run the same prompt twice in a day and diff the domains: that is your within-day churn. Run it again days later: that is your drift. Here they were 0.73 and 0.68. Any reported change smaller than the gap between those two is the index breathing, not a result.
What was measured
| Category | Home appliance spare parts |
| Markets | United Kingdom, Germany, France, Spain, Italy, Poland, Portugal |
| Questions | 8, across five intents, written natively in each language |
| Models | GPT-5.2, Perplexity Sonar, Claude Sonnet 5, Gemini 3.1 Pro |
| Repeats | 3 per cell |
| Passes | 2, days apart |
| Calls | 672 per pass, 1,344 total |
| Citations | 5,236 and 5,093 |
| Distinct domains | 749 and 759 |
| Temperature | 0 |
The languages sit on a deliberate gradient of corpus size, because that was the variable I thought I was testing. English and German are thick, French and Spanish and Italian are substantial, Polish and Portuguese are comparatively thin.
The category is a stand-in and says so. Appliance spare parts is structurally the same problem as the catalogue I work on daily: fitment driven, commodity, cross-border, long tail, a handful of pan-European retailers competing with local ones. It is deliberately not my employer’s market, so this is independent research rather than work done on company time about company competitors. No employer data touches it.
Questions were written natively, not translated. A literal translation asks something no native speaker would type, and the retrieval that follows answers a question nobody asked.
Limits, stated plainly
One category. The method transfers, these specific numbers do not.
Language and market are conflated. They move together here by design, so this study cannot separate “Polish the language” from “Poland the market”. Separating them needs the same language in two markets. Portuguese in Portugal and Brazil would do it, and Brazilian domains already appear in the Portuguese results, which is a hint worth chasing and the obvious next study.
Four models, one aggregator. Everything came through OpenRouter, so some of what is reported as model behaviour may be aggregator behaviour. That is exactly the failure the control found in two of its own arms. Direct-API replication would settle it.
Corpus depth is asserted, not measured. I placed seven languages on a thick-to-thin gradient from general knowledge of training corpora, not from a token count. The hypothesis it was built to test failed anyway, and it failed on the two languages I was most confident about, so the conclusion does not depend on the ranking being precise. It is still an assumption and it should be named as one.
Two passes is the minimum, not a lot. Everything reported as a range here is a range across two measurements. A third pass would narrow them. The figures I am confident in are the ones that barely moved across the two I have.
FAQ
Does AI retrieval work the same way in every language? No. Across two full passes, the same question asked in seven European languages produced citation sets that overlapped with the English one by 4% to 11%, against a measured noise floor of 72% to 74%. Every language was at least six times further from English than English was from itself.
Can I audit AI visibility in English and assume it transfers? No. Between 675 and 707 of roughly 750 domains appeared in exactly one language, and only three appeared in all seven. An English audit describes the English retrieval environment and very little else.
Does a smaller training corpus mean a language falls back to English sources? Not in this data, and this was the hypothesis the study set out to confirm. Poland and Portugal are both thin-corpus languages and sat at opposite ends of the local-domain table, at about 84% and about 28%, in both passes. Corpus depth did not predict local anchoring. Market structure is the better explanation, though this study cannot prove it.
Why does the same prompt return different sources each time? Because generation and retrieval are different systems. At temperature zero the text is close to deterministic, but the search runs fresh against an index that moves. Two identical requests here agreed on 72% to 74% of their cited domains, and the same question days apart agreed on 68%.
How large does a change have to be before it is real? Larger than your own drift. Here a question agreed with itself 73% of the time within a day and 68% across days. Anything smaller than that gap is the index moving, not your visibility changing.
Are AI visibility tools measuring the models they name? Sometimes not. Two models in this study agreed with each other on 80% to 83% of cited domains, above their own rerun baseline, because both were served by the same third-party index rather than by their own native search. Two models that searched for themselves agreed on about 6%. Ask your vendor which retrieval path each model uses.
Does ChatGPT always search the web before answering? No. With OpenAI’s own search enabled, GPT-5.2 returned no citations on 76% of calls in the first pass and 86% in the second. The control arm answered the same questions in the same languages with citations nearly every time, so the sources existed. The decision not to search was the system’s, and its rate is not stable.
Is this study reproducible? Yes. The harness, the questions and the configuration are published, and the per-citation dataset for both passes is released alongside the article. The method does not depend on the category.
The takeaway
Retrieval is not one environment with seven translations. It is seven environments that share three domains, and the distance between them is at least six times the noise in the measurement.
I was wrong about the mechanism, and the correction is the more useful result: it is not the size of the language, it is the shape of the market. That means the work does not scale by translating a playbook.
And I was wrong about my own noise floor by 14 points until a second pass made me look. Both corrections are in this article rather than quietly edited out, because a study arguing that most published figures are unmeasured has to show its own working first.
Before acting on any of this, measure your own drift. The most useful number here is the one that says how wrong a measurement can be while looking exactly like a result.
Emad Sharaki is a Senior SEO & GEO Strategist with 12+ years in search. He leads AI Visibility work at AUTODOC SE, tracking brand representation across six major language models and 35+ European markets, and maintains an enterprise database of 38+ verified AI crawlers used to govern crawler policy across seven European markets. He has spoken at WordCamp Porto, Porto WordPress Meetup and WordPress Day for E-Commerce, and has been quoted in Search Engine Journal. If you are starting on this, there is a learning path for AI search.
Method and data
The harness is public and the method does not depend on the category or the questions.
export OPENROUTER_API_KEY='sk-or-...'
node run.mjs --probe # verify model ids and the working search shape
node run.mjs --dry-run # matrix size and cost, sends nothing
node run.mjs # the run, resumable
node analyze.mjs # the numbers and the dataset
node run.mjs --out out-pass2 # the second pass, days later
node analyze.mjs --out out-pass2
node reconcile.mjs # what moved between themProbe first. OpenRouter’s documentation contradicts itself about how web search is enabled, so all three shapes are implemented and --probe reports which one actually returns citations on your account today.
The second pass needs --out. Pointed at a finished directory the runner correctly finds nothing to do and exits in a second, which reads exactly like a second pass that ran and agreed with the first.
Dataset: citations.csv for both passes, one row per citation, published alongside this article. OpenAI’s native search appends ?utm_source=openai to the URLs it returns. It is left in place as evidence of which search path produced the row.
If you find an error in any number above, tell me and I will correct it and say so. I found two in my own, and both are in the article.