{"id":"1df25431-9a22-4929-8b49-59bf9f2283f6","arxiv_id":"2609.18729","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AI chatbots vary drastically in how they recommend products and cite sources, and their APIs cannot substitute for auditing the consumer-facing interfaces.","lead":"Researchers audited product recommendations from ChatGPT, Gemini, and Google AI Overviews using 117 real-world consumer queries. They found that ChatGPT often presents recommendations as personal picks, that repeated queries yield different products and sources, and that the APIs do not reflect what consumers see on the web.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Unverified model version for ChatGPT interface threatens the interface–API comparison that motivates the central claim about API non-proxiness.","rationale":"The reader’s weakest assumption was query representativity, which is a valid concern but less load-bearing for the central argument: even with a narrow query set, the findings of variation across systems and repetition are statistically robust and thoroughly documented. The model-version unceratinty for ChatGPT is more critical because it directly undermines a key specific claim (RQ3) that APIs are not proxies for interfaces—a claim with immediate practical implications for auditors and regulators. However, the paper itself flags this limitation and treats interface and API as distinct conditions throughout, so the overall verdict should remain CONDITIONAL. The paper’s strongest contributions—demonstrated cross-system variability in recommendations and sources, and the instability of repeated queries—do not depend on this assumption. Therefore, no verdict adjustment is needed, but future replications should independently verify the interface model.","tokens_in":24134,"tokens_out":2088,"duration_ms":20446,"concrete_test":"Conduct a controlled replication where the ChatGPT interface model is identified (e.g., by prompting for system version with a known leak technique or via OpenAI’s documentation during the same time period). Then re-run the same 117 queries thrice on the interface and an API using the verified same model. If domain overlap rises substantially (e.g., >30%), the paper’s RQ3 conclusion should be downgraded from 'APIs do not reproduce the consumer interface'. To 'interface–API matching depends on model alignment, and access method alone does not account for the difference'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The logged-out ChatGPT interface self-reported gpt-5.6-luna bei generation, but the authors cannot independently verify this (Section 'Audit Methodology'). The interface–API comparison (RQ3) therefore compares an unknown interface model to a known API model (gpt-5.6-luna). If the interface actually uses a different model—e.g., a smaller, cheaper, or more safety-tuned variant—then the observed low domain overlap (12.0%) could reflect model mismatch rather than access-method incompatability. The paper's central claim that 'APIs do not reproduce the consumer-facing systems' (Conclusion) rests heavily on this comparison. If model confound were removed, the overlap might increase enough to weaken the practical advice that auditors must always use consumer interfaces. The paper acknowledges the limitation but does not refute it, leaving the RQ3 evidence partally compromised.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper conducts an empirical audit of AI-generated product recommendations across five access conditions: ChatGPT (interface and API), Gemini (interface and API), and Google AI Overviews. It curates a dataset (ConsumerQ) of 2,528 real commercial-advice queries, filters 117 physical-product queries, and collects 1,536 responses. The findings show that (i) ChatGPT uses a first-person voice and labels products as 'best' far more often than Gemini or AI Overviews; (ii) repeated queries yield variable recommendations and source lists; (iii) displayed sources differ drastically across systems (e.g., ChatGPT and Gemini share only 5.4% of domains on average); and (iv) APIs do not reproduce the consumer interfaces (mean domain overlap 12.0% for ChatGPT, 14.8% for Gemini). The paper concludes that audits of AI-mediated commercial advice must use repeated observations of consumer-facing interfaces.","tokens_in":24394,"tokens_out":3119,"duration_ms":30855,"significance":"If the findings hold, they have direct implications for consumer protection, regulatory enforcement under the EU Digital Services Act, and methodology for auditing generative AI. The work provides a novel, real-query dataset (ConsumerQ) and a rigorous auditing framework that pairs interface and API requests close in time. The empirical demonstration that APIs systematically diverge from consumer interfaces is a practically important caution for researchers and regulators who rely on API access. The paper is likely to influence both the auditing literature and policy debates about transparency obligations for large AI platforms.","major_comments":[{"comment":"The logged-out ChatGPT interface self-reports gpt-5.6-luna, but the authors cannot independently verify the model version. The RQ3 comparison (interface vs. API) is central to the claim that 'APIs do not reproduce the consumer-facing systems.' If the interface actually uses a different model (e.g., a cheaper or more safety-tuned variant), the observed low domain overlap (12.0%) could partly reflect model mismatch rather than access-method incompatibility. The paper acknowledges this limitation but does not refute the possible confound. To strengthen the conclusion, the authors should either provide evidence that the interface model matches the API model (e.g., through external probing or provider documentation) or more explicitly discuss how sensitive the interface–API divergence is to this assumption.","section":"Audit Methodology"},{"comment":"The 117 physical-product queries used for the main analysis are filtered using exclusion criteria (no price figure, no named product, at most one constraint) that yield a simplified and homogeneous set. The paper acknowledges this limitation but still reports precise numeric overlap estimates (e.g., 5.4% domain overlap between ChatGPT and Gemini) without discussing how the filtering might affect the generalizability of those numbers. For example, queries with named products or price constraints could produce different source sets or overlap patterns. The authors should add a brief discussion of whether the observed variability between systems and repetitions is likely to be larger or smaller for more complex or constrained queries.","section":"Physical product recommendations and Codebook B.3"}],"minor_comments":[{"comment":"The y-axis label 'share of queries for which the three repeats are not equal' is ambiguous. Please clarify that it means the proportion of queries where at least one of the three repeats differs on the indicated property.","section":"Figure 3"},{"comment":"The column heading 'Top 10' could be misinterpreted. Consider renaming to 'Top-10 domain share (%)' and adding a brief footnote explaining that it is the percentage of all domain occurrences accounted for by the ten most frequent domains.","section":"Table 1"},{"comment":"The source-type classification using gpt-5.5 was not validated against human annotations. The paper marks this as exploratory in the text, but Figure 7 presents the results without reiterating that caveat in the caption. Please add a note that these distributions are exploratory.","section":"RQ2 – Source type classification"},{"comment":"The term 'Jaccard Overlap' is non-standard; the more common term is 'Jaccard similarity' or 'Jaccard index.' Consider adopting the standard term.","section":"Throughout"},{"comment":"The paper mentions using 'Patchright with Chrome' but does not provide a citation or brief description. For reproducibility, please add a reference or a short explanation of its relationship to Playwright.","section":"Audit Methodology"},{"comment":"Minor typographical consistency: 'monetising' is used in the Abstract but 'monetizing' appears elsewhere (British vs. American spelling). Pick one style and apply throughout.","section":"Abstract"}],"recommendation":"minor_revision","confidential_remarks":"The unverified ChatGPT model version is the most likely point of pushback from reviewers; the authors handle it responsibly in the limitations but could strengthen the paper by adding a robustness check (e.g., if future provider documentation confirms the model, or by using model fingerprinting). The paper is a strong empirical contribution to the audit literature and very timely given the DSA context. I do not see any novelty or scope issues for cs.CY."},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is a solid audit paper that does what it says on the tin. It curates a dataset of real commercial-advice queries, then runs a careful head-to-head comparison of five conditions (ChatGPT interface/API, Gemini interface/API, AI Overviews). The core finding—that different systems recommend different products with different framing, and that APIs systematically diverge from their corresponding consumer interfaces—is well-supported by the data and should be useful to regulators, consumer groups, and researchers auditing under the DSA.\n\nThe methodology is the paper's real strength. Pairing interface and API requests within 0.2 seconds, using fresh browser profiles and residential proxies, and repeating each query three times gives the comparisons high internal validity. The authors also document their annotation pipeline transparently and report inter-annotator agreement where it matters.\n\nSoft spots: The 117-query sample is the main limitation, and the authors own it. It covers only physical product queries, filtered for generality, so the results may not generalize to services, software, or location-dependent advice. Three repetitions also capture only a slice of the variability. The stress-test concern about the ChatGPT model version is noted but, in my reading, not load-bearing: the paper never assumes model equivalence between interface and API, and treats the access methods as distinct conditions. The low domain overlap (12.0%) remains informative even if the underlying model differed—it still tells auditors they can't rely on the API to reproduce whatever the interface is doing. Still, a future audit that could independently verify the interface model would strengthen that leg of the argument.\n\nBottom line: This paper deserves peer review. It makes a concrete, reproducible empirical contribution, and its limitations are honestly stated. I'd recommend publishing with minor revisions and a request for broader query sampling in follow-up work.","headline":"Well-executed audit with honest limitations; the central claim that APIs don't proxy consumer interfaces holds up despite minor uncertainties about model versioning.","tokens_in":24925,"tokens_out":940,"would_cite":true,"duration_ms":11789,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"AI product recommendations are not stable answers to a shopping query: different systems recommend different products with different framing, and provider APIs do not reproduce what consumers actually see.","keywords":["AI audit","product recommendations","commercial advice","ChatGPT","Gemini","AI Overviews","source attribution","Digital Services Act"],"falsifier":"Submit the same 117 queries through the ChatGPT consumer interface and the ChatGPT API with the exact model and search-grounding parameters pinned on both sides, and recompute domain overlap: if the overlap approaches 100% rather than the reported 12.0%, the interface-API divergence is a configuration artefact rather than an inherent difference between the surfaces. A second check: repeat one query thirty times on a single interface; if the cumulative set of distinct cited domains plateaus within the first few requests, the claim that repeated queries keep revealing new sources would need to be abandoned.","tokens_in":25890,"feed_emoji":"🛒","tokens_out":11317,"duration_ms":70706,"temperature":0.7,"pith_summary":"This paper tries to establish that AI-generated product recommendations are not a stable answer to a shopping question: the same real query asked of ChatGPT, Google Gemini, and Google's AI Overviews produces different products, different framing, and different cited sources, and even repeating the identical query to one system can change the result. That instability matters because consumers increasingly treat chatbots as buying advisors, both OpenAI and Google are monetising those answers with advertising, and the European Commission has just designated ChatGPT as a Very Large Online Search Engine under the Digital Services Act, making independent auditability of such advice a regulatory requirement. Using 2,528 real user queries (the ConsumerQ dataset), the paper submits 117 physical-product queries three times through five audit conditions, namely the ChatGPT and Gemini consumer interfaces, their APIs, and AI Overviews, and measures how often the resulting recommendations and sources agree. The central negative result is that provider APIs are not a proxy for the consumer-facing product: identical queries submitted moments apart share only 12.0% of displayed domains between the ChatGPT interface and its API and 14.8% for Gemini. The paper's conclusion is methodological: neither isolated responses nor API observations can be assumed to represent the commercial advice consumers actually encounter, so audits must sample repeated responses, observe consumer-facing conditions, and record which source layer is being measured.","feed_headline":"Same query, different picks: AI shopping advice is unstable","feed_subtitle":"Audit of 117 real shopping queries shows ChatGPT, Gemini, and AI Overviews disagree — and APIs miss what consumers see.","key_machinery":"The load-bearing object is the paired audit condition: each of 117 verbatim consumer queries is submitted to five conditions, namely the logged-out ChatGPT and Gemini consumer interfaces, their developer APIs, and Google Search's AI Overviews, with every interface request paired to an API request a median of 0.17 seconds later, and with three repetitions per condition spread hours apart from a fixed Netherlands location using fresh browser profiles. Agreement is quantified by the Jaccard overlap $J(A,B) = \\frac{|A \\cap B|}{|A \\cup B|}$, the share of the union of two sets that both contain, applied to recommended-product sets and to cited domains and pages. The pairing carries the argument: because interface and API receive identical queries moments apart, their divergence cannot be blamed on query wording or short-term search drift and must be a property of the conditions as configured. A second mechanism is the separation of observable source layers, namely displayed citations, pages returned by web search, and grounding sources, which allows the paper to show that different access methods expose different stages of the same retrieval pipeline.","core_discovery":"The central claim is that AI-mediated commercial advice is a property of the whole system, not of any single response. Across 117 real shopping queries repeated three times, the consumer-facing systems disagreed systematically in how they frame advice: ChatGPT expressed a first-person preference ('my pick is...') in 79% of product-recommending responses and labelled a product as 'best' in 73%, while Gemini did so in 7% and 43% and AI Overviews in 2% and 48%. The recommended products themselves drifted across repetitions, most strongly for ChatGPT (mean Jaccard overlap of 0.178 between repeats), so two answers to the same question differed about as much as answers from two different systems. Displayed sources were shared even less: ChatGPT and Gemini had a mean domain overlap of 5.4% for the same query, with no common domain in 76.7% of comparisons, and repeated queries kept surfacing new domains (ChatGPT's observed domains per query grew from 2.65 after one request to 5.56 after three). The APIs did not reproduce their interfaces: interface and API shared 12.0% of domains for ChatGPT and 14.8% for Gemini, with systematic compositional differences (manufacturer domains dominated in the ChatGPT API; youtube.com and reddit.com in the Gemini API) and with the ChatGPT API exposing a search-return layer of 37.38 pages per observation against only 3.22 pages in the final citations. The paper concludes that neither isolated responses nor API observations can be assumed to represent the commercial advice consumers encounter, and that the unit of analysis for audits must become the system in use rather than the canonical answer.","pith_inferences":["If the reported instability extends to the higher-stakes purchase categories in the paper's own codebook, namely health, safety, and financial commitments, the practical cost of a variable recommendation rises sharply, strengthening the case for treating these outputs as regulated commercial advice rather than mere information.","A testable extension the authors did not run: repeating the audit from several EU jurisdictions or with language-localised queries would show whether recommendations and source sets shift by country, which is precisely the evidence regulators enforcing the Digital Services Act would need.","The provider-specific source profiles (techradar.com and tomsguide.com for ChatGPT; reddit.com and youtube.com for Gemini) look consistent with the content-licensing and data-access deals the paper mentions, which predicts that source sets will stay stable within a provider over time even as the exact cited pages change, a distinction future audits could test directly.","As advertising expands inside these assistants, a before-and-after comparison of first-person framing and source composition around each ad rollout would test whether monetisation measurably changes recommendation behaviour; the paper's method of paired, repeated observations is ready-made for that study."],"forward_implications":["Audits of AI commercial advice should sample repeated responses and report the variance, because a single response captures only part of the source set a consumer may encounter: ChatGPT's observed domains per query grew from 2.65 after one request to 5.56 after three.","Regulatory and legal evidence gathering should rely on consumer-facing observations rather than API calls, since the APIs diverge from the interfaces both in which domains appear and in how sources are composed within a response.","The framing of advice is itself part of the product: the same query arrives as a personal endorsement from ChatGPT, a list of alternatives from Gemini, or a source-panel summary from AI Overviews, and this framing affects how consumers read the certainty of buying advice.","Because 76.7% of ChatGPT-Gemini comparisons share no domain at all, source disagreement across providers is structural rather than incidental, implying that retrieval and licensing choices shape the advice consumers see as much as the underlying product facts.","The appropriate unit for evaluating these systems shifts from 'is this answer right?' to 'have we characterised the distribution of answers, framings, and sources a consumer could receive?'"],"supporting_citations":[{"why":"Supplies LMSYS-Chat-1M, one of the two real-user conversation corpora from which the ConsumerQ queries are drawn.","marker":"Zheng et al. 2024"},{"why":"Supplies WildChat-1M, the second real-user conversation corpus used to construct ConsumerQ.","marker":"Zhao et al. 2024"},{"why":"Largest measured study of ChatGPT use; motivates the audit by showing product-recommendation conversations are a real share of usage.","marker":"Chatterji et al. 2025"},{"why":"The DSA designation of ChatGPT as a Very Large Online Search Engine, which makes independent auditability of consumer-facing advice a live regulatory question.","marker":"European Commission 2026"},{"why":"Prior empirical audit of generative search response consistency and sources that this paper extends from general queries to commercial advice.","marker":"Grossman et al. 2026"},{"why":"Prior evidence that API benchmark scores do not transfer to chatbot interfaces, extended here from benchmarks to product recommendations and displayed sources.","marker":"Wang et al. 2026"},{"why":"Prior measurement of ChatGPT as a recommender across interfaces, motivating the interface-versus-API comparison design.","marker":"Schatto-Eckrodt et al. 2025"},{"why":"Experimental evidence that consumers over-rely on LLM-generated advice, which is why the framing and instability of recommendations matter.","marker":"Spatharioti et al. 2023"}],"fun_headline_variants":["ChatGPT says 'my pick' in 79% of shopping queries","Same query, different answers: AI advice drifts","API results miss what consumers see in AI shopping","AI shopping advice: unstable across repeats and systems"],"cache_read_input_tokens":0,"weakest_assumption_plain":"All comparisons rest on 117 physical-product queries that were filtered from real user logs by excluding anything with a price figure, a named product, a location, a date, or more than one constraint; if real shopping questions are messier than this filtered set, the measured differences may not describe what consumers actually ask.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT says 'my pick' in 79% of shopping queries","Same query, different answers: AI advice drifts","API results miss what consumers see in AI shopping","AI shopping advice: unstable across repeats and systems"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"cost":0.000236208,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000236208,"upstream_inference_prompt_cost":0.000135408,"upstream_inference_completions_cost":0.0001008},"total_tokens":1532,"prompt_tokens":1172,"completion_tokens":360,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":294}},"tokens_in":1172,"tokens_out":360,"cost_source":"openrouter_usage","duration_ms":5217,"image_pages":[],"temperature":1.0,"reasoning_tokens":294,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-17T06:13:38.960190+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"falsifier":"Submit the same 117 queries through the ChatGPT consumer interface and the ChatGPT API with the exact model and search-grounding parameters pinned on both sides, and recompute domain overlap: if the overlap approaches 100% rather than the reported 12.0%, the interface-API divergence is a configuration artefact rather than an inherent difference between the surfaces. A second check: repeat one query thirty times on a single interface; if the cumulative set of distinct cited domains plateaus within the first few requests, the claim that repeated queries keep revealing new sources would need to be abandoned.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The DSA designation of ChatGPT as a Very Large Online Search Engine, which makes independent auditability of consumer-facing advice a live regulatory question."},{"cited_title":"o der, J. T.; K \\","cited_arxiv_id":null,"evidence_quote":"Prior measurement of ChatGPT as a recommender across interfaces, motivating the interface-versus-API comparison design."}],"review_version":1}