{"id":"e8e73105-df37-470a-8ecf-dd06ce9e0c71","arxiv_id":"2608.30052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a live generative search interface, the query language gates whether local suppliers are named and the exit IP selects which country's market is served, with the two factors acting independently.","lead":"A controlled probe of a live generative search interface shows that the language of the question decides whether local suppliers are named at all, while the exit internet address decides which country's market is served. The two factors act on different things, which matters for anyone measuring AI visibility or selling into national markets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The translated prompts are never quoted; if any prompt encodes a country or locale, the claimed language/IP separation is not established.","rationale":"The reader's conditional verdict is well supported, and the paper is unusually careful: it discloses discarded runs, the post-hoc case-sensitivity correction, a failed prediction, the model ambiguity, and a direct conflict of interest. The statistical-resolution concern identified by the reader is real but is not the most load-bearing issue for the central claim. With the reported counts, the local-versus-English contrasts are large overall, and the boundary cells are even more extreme when considered jointly; six-run noise would weaken magnitude claims but would not by itself destroy the existence of a language effect. The regulation-versus-supply interpretation in §4.6 is also fragile, but it is an explanatory add-on rather than the core claim about language and IP separability. The prompt-specification gap is more fundamental: the entire experiment operationalizes “query language” through untranscribed stimuli. If the stimuli differ in content rather than only in language, then the language gate and the IP market-selection result are confounded at the point of manipulation. Because the paper does not quote the prompts, a reader cannot verify this from the manuscript alone, and the deposit was not checkable in this review. This does not imply the result is wrong; it means the central claim currently rests on an unverified control condition. That is exactly the kind of condition that should keep the paper at CONDITIONAL rather than ACCEPT. My proposed deposit inspection is a single check that settles the concern: clean prompts support the claim, country-bearing prompts refute it. Agreement with the reader is partial because I share their measurement-validity concern but locate it in stimulus specification rather than in the six-run sample size.","tokens_in":10508,"tokens_out":13709,"duration_ms":135359,"concrete_test":"Open the Zenodo deposit at DOI 10.5281/zenodo.22181306 and extract the exact user prompt from the collected run records for each of the eleven cells and all six prompt categories in all languages used. Check whether any prompt contains a country name, demonym, or locative expression naming Estonia, Germany, Norway, Türkiye, or any other market. If all prompts are country-free and semantically parallel translations, the language/IP separation is supported as claimed; if any prompt encodes a country or locale, the central claim fails and the affected cells must be re-run with country-free prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central manipulation is query language, implemented as translated prompts. The manuscript reports category names and model-output excerpts but never quotes the actual user query strings in any of the six languages. Every quoted phrase in §4.4, such as “Türkiye’de serbest çalışan” and “Eğer Almanya’da serbest çalışan olarak çalışıyorsan,” is generated answer text, not input text. If the non-English prompts contained a country name, demonym, or locative phrase, then the local-language cells would be requesting a national market by content, not merely by language. The Istanbul-versus-Berlin Turkish pair would still be informative only if the Turkish prompt is country-free; without the prompt text, the paper does not establish that. The data availability section points to a Zenodo deposit containing run records and collectors, which would resolve this, but the deposit could not be verified during review, and the manuscript itself omits the stimulus. This is a control problem rather than a statistics problem: six-run confidence intervals affect precision, whereas a country-bearing prompt would collapse the causal claim. The concern is about incomplete reporting, not author conduct; if the deposit shows country-free prompts, the main empirical contrast survives.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper reports a controlled observational audit of a commercial generative search interface. Over two days the author collected 234 usable runs in eleven cells defined by surface (logged-out web UI or API), verified exit IP, and query language, with six identical runs per cell. The treatment category is accounting software for freelancers; the control category is project-management software for a small marketing agency. The main empirical findings are: identical replays change the top recommendation on four of six prompts at the same rate in the web UI and in both API configurations; in the accounting category, official-language queries name domestic suppliers in 23 of 24 runs while English queries from Istanbul and Tallinn name none; holding Turkish fixed and moving the exit IP from Istanbul to Berlin moves the named supplier set from Turkish to German firms while the answer stays in Turkish (replicated with Russian from Tallinn); Russian from Estonia sits between Estonian and English; and the project-management control shows no domestic supplier in any language. The paper concludes that query language selects which national supplier set the system is willing to name, exit IP selects which market is treated as the user's, and the two mechanisms act separately.","tokens_in":10715,"tokens_out":13066,"duration_ms":117560,"significance":"Should the claims hold, the paper is a meaningful contribution to the search-auditing and generative-recommendation literatures: it generalizes the sock-puppet logic of classical personalization audits to generative interfaces, demonstrates substitution of whole national supplier sets rather than mere reordering of a shared result list, and provides a useful instability measurement that bears on how such audits should be designed. The manuscript has real strengths: the repeated-run design; explicit verification of exit IP per arm; transparent disclosure of discarded runs, the re-collection, and the case-sensitivity correction; a between-collection consistency check; a coded control category; a public data deposit; and an unusually candid competing-interests statement. The central uncertainty is not internal arithmetic but reporting and resolution: the query stimuli are never quoted, and several load-bearing cells consist of six runs against a system shown to be highly non-deterministic.","major_comments":[{"comment":"The manuscript never reports the full query strings that operationalize 'query language'. The Turkish and Russian phrases quoted in Section 4.4 are answer text, not input. Because the central manipulation is the prompt, any country-denoting content in the non-English prompts would make 'language' a proxy for an explicit market request, and the Istanbul-versus-Berlin Turkish contrast would no longer separate language from exit IP. Please put the verbatim prompts for all six languages in the paper, link each cell to the corresponding run records in the Zenodo deposit, and state explicitly that no prompt contains a country name, demonym, city, or national institution. This is a reporting-control issue rather than a statistical one, but it is load-bearing for the paper's main claim.","section":"§3.1–§3.2, §4.4"},{"comment":"The six-run-per-cell design underdetermines some of the verbal conclusions. In the two non-boundary countries, the local-vs-English comparison has reported p=0.0152 for Norway and p=0.0606 for Germany, so the language gate is not uniformly established at conventional significance in each independent replication. Moreover, §4.1 shows the top recommendation changes on four of six prompts under identical replays, and §6 concedes that a truly 90%-dominant answer would be observed as stable in only about 47% of six-run samples. The 0-of-6 versus 6-of-6 cells in Estonia and Türkiye are individually strong, so I am not asking for a different design; I am asking that the language match the evidence: report all per-cell exact tests, distinguish boundary cells from measured contrasts, and replace categorical phrases such as 'decides whether local suppliers appear at all' with a probability-respecting formulation unless a larger re-collection is added.","section":"§4.2, Table 2, §6"},{"comment":"The conclusion that the operative variable is national regulation rather than national supply cannot be identified from one treatment category and one control category. The paper itself states that the third category needed to separate regulation from supply was not collected. The project-management control also differs from the accounting category in several correlated features, including transactionality, the presence of localized legal obligations, and customer type. The observed pattern is consistent with a regulatory account but does not establish it. I recommend weakening the relevant statements in Section 4.6, the Discussion, and the Conclusion to a compatibility claim.","section":"§4.6"}],"minor_comments":[{"comment":"The test sentence reports two p-values for the same comparison; specify the exact contingency table, the event being tested (winning top recommendation versus brand presence), and the one-sided/two-sided convention used.","section":"§4.2"},{"comment":"The sentence about six-run detection rates does not define the event being detected (a change from the modal answer, a stable winner, or at least one appearance); rewrite it with the event explicitly stated.","section":"§6"},{"comment":"Case-sensitive matching is disclosed, but the exact matching strings for brands whose names are also ordinary Turkish words are not given; including them would let readers assess false-negative risk in addition to the false-positive examples already discussed.","section":"§3.2"},{"comment":"The API row has no exit IP; state explicitly that the API arm is a separate no-geography condition rather than a fourth surface for the IP comparison.","section":"Table 1"},{"comment":"Standardize the spelling of 'Türkiye' across the abstract and body text.","section":"Abstract and text"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper does something genuinely useful: it measures how query language and exit IP separately affect which suppliers a generative search interface recommends, using controlled runs against ChatGPT. The core result — language gates whether local suppliers appear at all, and IP decides which market's suppliers — is real and, as far as I can tell, new. The substitution pattern (0/6 vs 6/6 in multiple cells) is striking, and the replication on two unrelated language/country pairs bolsters it. Credit where due: the author discloses discarded runs, a post-hoc case-sensitivity fix, a failed prediction, and a direct conflict of interest. The negative control (project management) correctly disconfirms their own expectation and points toward regulation rather than supply, though that explanation rests on a two-category comparison with the decisive third cell uncollected.\n\nThe soft spots are real and the author mostly names them. Six runs per cell is thin — the mid-tier result (Russian from Estonia, 4/6 local) has Fisher's exact p ~0.06 against English, and the paper's opening causal framing ('decided before the model reasons') asserts an implementation ordering the measurement cannot support. The stress-test concern about unquoted prompts is fair: the manuscript never shows the actual query strings. If the non-English prompts contained country names, the language/IP separation collapses. The author points to a Zenodo deposit that would resolve this, but the deposit wasn't verifiable during text review. For the paper's own claims, this is the weakest structural gap, not a statistics problem.\n\nWho is this for? Researchers in generative-engine-optimization, search audit, and multilingual information access. It reads like a careful field report, not a theoretical advance. The measurement protocol and the disclosure norms are a model for this kind of work.\n\nRecommendation: send to peer review. The empirical pattern deserves scrutiny and replication with larger samples and reported prompts. The author should be asked to include the actual query strings (or confirm they are country-free) and to tone down the causal language. But this is worth a referee's time.","headline":"Earnest audit of language and location effects in generative search — small sample but clean design; worth engaging but needs replication and tempered claims.","tokens_in":11227,"tokens_out":1046,"duration_ms":12052},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Query language, not location, decides whether generative search names local suppliers.","keywords":["generative search","multilingual information access","recommendation localisation","query language","exit IP","non-determinism","measurement validity","market gating"],"falsifier":"Run the same experiment with 30 runs per cell: if the Estonian English cell ever returns a local supplier in 2 or more of 30 runs, or if the Turkish Berlin cell ever returns a Turkish supplier in 10 or more of 30, then the near-total substitution observed is not a stable property but a statistical fluke of small samples.","tokens_in":11765,"feed_emoji":"🔍","tokens_out":3060,"duration_ms":29074,"temperature":0.7,"pith_summary":"This paper demonstrates that in a generative search interface, the language of a commercial query—not the user's geographic location—determines whether local suppliers appear in the answer. When asked in the local official language, a local brand won 23 of 24 runs across four countries; asked in English from the same connections, local brands appeared in 0 of 12 runs in two of those countries. The paper also shows that language and exit IP act on separate things: holding the query language fixed and moving only the IP moves the market whose suppliers are named, while the answer stays in the query language. A control category with domestic suppliers but no national regulation showed no language effect at all, pointing the explanation at regulation rather than supply. The findings imply that English-language visibility measurements miss entire domestic markets.","feed_headline":"Language, not location, decides which brands AI search names","feed_subtitle":"English queries suppress local suppliers even when the system knows where you are, a 234-run probe shows.","key_machinery":"The experimental design manipulates query language and exit IP independently across 234 runs with six identical runs per cell, using a logged-out ChatGPT web interface and the OpenAI API. The central measurement is brand presence against a pre-compiled per-market supplier list, with case-sensitive matching to avoid false positives from common words. The separation of language and location is demonstrated through two unrelated language–IP pairs (Turkish from Berlin, Russian from Tallinn) where the supplier set follows the IP while the answer language stays fixed.","core_discovery":"The paper's central claim is that in a generative search interface, query language acts as a gate that selects which national supplier set the system is willing to name, while the exit IP selects which national market that set belongs to, and the two factors are separable. Concretely, when the query language matches the country's official language, a local supplier won 1 of 24 runs; asked in English on the same connections, local suppliers appeared in 0 of 6 runs in Estonia and Türkiye. Holding Turkish fixed and moving the exit IP from Istanbul to Berlin replaced Turkish suppliers with German suppliers while the answer remained in Turkish. A minority language (Russian in Estonia) occupied a middle tier, naming local suppliers in 4 of 6 runs and global ones in all 6. A negative control in a category with domestic suppliers but no national regulation showed no language effect, shifting the explanation from national supply to national regulation.","pith_inferences":["The language gate likely extends to other legally localised categories (e.g., tax filing, legal services, healthcare), where domestic suppliers might similarly disappear under English queries.","If the gate is implemented through retrieval, then optimising English-language content about a domestic supplier may be insufficient to make it appear in English answers—the gate may be a hard filter rather than a ranking signal.","The three-tier ordering (official language > minority language > English) suggests that the interface uses query language as a proxy for the user's national market, with English triggering a default global market regardless of the user's actual location.","A practical extension would test whether a logged-in account with a registered local address overrides the language gate; the current design used logged-out sessions only."],"forward_implications":["Any organization measuring its visibility in generative search answers solely in English is effectively blind to domestic competitors and the market those customers represent.","Visibility measurements must report both query language and exit IP (or state that an API has no egress), because both factors independently determine the recommended supplier set.","The finding that English acts as a global-market signal rather than a neutral control means that English-language recommendations cannot be used as a baseline for cross-market comparison.","The control category result implies that the language gate is not a universal effect but is tied to whether the product category is nationally regulated, such as accounting software subject to local VAT and filing rules.","The high run-to-run instability (top recommendation changed on 4 of 6 prompts) means that any single observation of a generative recommendation is unreliable; measurement requires repeated runs."],"supporting_citations":[{"why":"Defines generative engine optimisation and provides the measurement premise that a single answer is not evidence.","marker":"Aggarwal et al. 2024"},{"why":"Documents non-determinism in LLM settings, supporting the finding of run-to-run instability.","marker":"Atil et al. 2024"},{"why":"Establishes the sock-puppet audit method for search personalisation, which the paper's design borrows.","marker":"Hannák et al. 2013"},{"why":"Shows that geolocation drives modest personalisation in web search, contrasted with the substitution observed in generative search.","marker":"Kliman-Silver et al. 2015"},{"why":"Finds brand bias in LLMs (global positive, local negative), which predicts global dominance in English but not the language-gated local substitution.","marker":"Kamruzzaman et al. 2024"},{"why":"Calls for multi-prompt evaluation of LLMs, supporting the use of six identical runs per cell.","marker":"Mizrahi et al. 2024"}],"fun_headline_variants":["AI search names local brands by query language, not location","English queries suppress local brands in AI search, study shows","Query language, not exit IP, decides which brands AI search names","Use local language to see local brands in AI generated search","Language gate selects AI search suppliers; location picks market"],"cache_read_input_tokens":0,"weakest_assumption_plain":"Six identical runs per cell are sufficient to detect a 0-of-6 versus 6-of-6 contrast as a signal rather than as draws from overlapping noisy distributions, given that the system's top recommendation changed on four of six prompts across repeated identical runs.","fun_headline_variants_meta":{"raw":{"variants":["AI search names local brands by query language, not location","English queries suppress local brands in AI search, study shows","Query language, not exit IP, decides which brands AI search names","Use local language to see local brands in AI generated search","Language gate selects AI search suppliers; location picks market"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"cost":0.000234248,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000234248,"upstream_inference_prompt_cost":0.000123368,"upstream_inference_completions_cost":0.00011088},"total_tokens":1482,"prompt_tokens":1086,"completion_tokens":396,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":313}},"tokens_in":1086,"tokens_out":396,"cost_source":"openrouter_usage","duration_ms":5365,"temperature":1.0,"reasoning_tokens":313,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T04:58:15.407127+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-alltime-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same experiment with 30 runs per cell: if the Estonian English cell ever returns a local supplier in 2 or more of 30 runs, or if the Turkish Berlin cell ever returns a Turkish supplier in 10 or more of 30, then the near-total substitution observed is not a stable property but a statistical fluke of small samples.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines generative engine optimisation and provides the measurement premise that a single answer is not evidence."},{"cited_title":"Non-Determinism of \"Deterministic\" LLM Settings","cited_arxiv_id":"2408.04667","evidence_quote":"Documents non-determinism in LLM settings, supporting the finding of run-to-run instability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that geolocation drives modest personalisation in web search, contrasted with the substitution observed in generative search."},{"cited_title":"\"Global is Good, Local is Bad?\": Understanding Brand Bias in LLMs","cited_arxiv_id":"2406.13997","evidence_quote":"Finds brand bias in LLMs (global positive, local negative), which predicts global dominance in English but not the language-gated local substitution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Calls for multi-prompt evaluation of LLMs, supporting the use of six identical runs per cell."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}