{"id":"2525ea68-630f-4f0b-b379-fc6a3f9d5aad","arxiv_id":"2508.00838","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"LLM search responses frequently omit citations for pages they used, with Gemini and Sonar leaving about 3 relevant URLs uncited per query while GPT-4o's smaller gap appears to reflect selective disclosure.","lead":"A study of 14,000 real LLM search logs quantifies how often models cite the web pages they actually use, finding that some models leave several relevant URLs uncited per query. The authors argue that transparency about search traces, not technical capacity, is the main missing ingredient.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution-gap estimates treat LMArena logs as complete records of retrieval and citation; if logs are selectively disclosed, the gap magnitudes and cross-model ranking are unidentified.","rationale":"The reader's weakest assumption identified exactly this issue: LMArena logs may be incomplete or selective, and the same problem the paper attributes to GPT-4o could affect other models. My stress-test agrees and makes the concern more specific: the abstract's own 'selective log disclosures' explanation is an admission that absence of a citation in the log does not imply absence of attribution in the interface. This is load-bearing because every headline number (34%, 24%, 92%, 3 uncited sites, efficiency 0.19–0.45) is a difference between observed fetch/citation counts and relevant URL counts; if the observation channel is lossy in a model-dependent way, the gap and ranking are not identified. I recommend CONDITIONAL rather than REJECT because the concern is directly testable: a controlled replay with full logging can measure false-negative rates and either confirm or refute the model ranking. Until that check is run, the empirical claims should not be treated as established. No ad hominem intended; the critique is about the measurement channel, not the authors.","tokens_in":753,"tokens_out":6353,"duration_ms":65658,"concrete_test":"Replay a random sample of at least 1,000 queries from the LMArena corpus through each model's official API or an instrumented browser session with search enabled, capturing all tool-call URLs and all rendered citations. Apply the same relevance-labeling pipeline to both the harness output and the corresponding LMArena log, then compute per-model false-negative rates: the fraction of harness-observed relevant fetched URLs absent from the log and the fraction of harness-observed citations absent from the log. If these rates are near zero for all models, the log-based attribution gap is confirmed; if they are non-negligible and differ across models, the gap magnitudes and ranking must be re-estimated with missingness corrections.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Gemini and Sonar leave ~3 relevant websites uncited per query while GPT-4o's small gap is a logging artifact. This inference depends on treating the LMArena logs as complete and faithful transcripts of (a) every URL the search tool actually fetched and (b) every citation or clickable source shown to the user. The paper treats absence of a fetch/citation in the log as evidence that no fetch/citation occurred. Yet the abstract itself invokes 'selective log disclosures' to explain GPT-4o's tiny uncited gap, conceding that logs can omit real retrieval activity. If the same selective omission affects Gemini or Sonar, the reported uncited-gap overstates their attribution failures, and the citation-efficiency ranking (0.19 vs 0.45) conflates logging policy with retrieval design. Without per-model false-negative rates for logged fetches and citations, the negative binomial hurdle model's central parameter is unidentified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes approximately 14,000 LMArena conversation logs of search-enabled LLMs to measure an 'attribution gap' between the relevant web pages a model fetches and the citations it actually provides. The authors report that Gemini frequently answers without citations (92% of answers with no clickable source), Sonar fetches many pages but cites only a fraction, and a negative binomial hurdle model estimates that the average Gemini or Sonar query leaves about 3 relevant sites uncited, whereas GPT-4o's small uncited gap is attributed to selective log disclosure rather than better attribution. Citation efficiency is reported to vary from 0.19 to 0.45 across models. The paper concludes with a recommendation for transparent LLM search architectures with standardized telemetry. The review is based on the abstract only, as the full text is not available.","tokens_in":902,"tokens_out":2203,"duration_ms":25881,"significance":"If the findings withstand scrutiny, this paper would provide a valuable empirical baseline for a real and growing problem: the lack of attribution to sources in AI-generated search answers. The use of naturalistic LMArena logs is a strength because it captures real user behavior rather than synthetic benchmarks, and the paper makes a practical contribution by quantifying the gap in a way that could inform platform accountability. The reported cross-model variation in citation efficiency is a potentially important result, suggesting that attribution behavior is a design choice rather than a technical necessity. However, the identification of the gap depends heavily on the completeness and representativeness of the logs and on the validity of the 'relevant URLs read' measure, both of which are not verifiable from the abstract. The analysis is therefore potentially significant but currently not fully substantiated.","major_comments":[{"comment":"The claim that GPT-4o's tiny uncited gap 'is best explained by its selective log disclosures' creates an identification problem: if logs can omit real citations or fetches for GPT-4o, the same omission could affect Gemini and Sonar. The paper treats absence of a fetch/citation in the log as evidence that no fetch/citation occurred, yet for GPT-4o it treats absence as possibly a logging artifact. Without a per-model false-negative rate for logged fetches and citations, or an independent validation of log completeness, the reported uncited-gap magnitudes and the citation-efficiency ranking (0.19 vs 0.45) conflate logging policy with retrieval design. This is load-bearing for the central claim of cross-model differences, and the abstract does not indicate how the issue is addressed.","section":"Abstract, second paragraph"},{"comment":"The central measure is 'relevant URLs read,' but the abstract does not describe how relevance is determined from conversation context. If relevance is labeled by the authors without a validated protocol, inter-annotator agreement, or a defined boundary between relevant and irrelevant fetched pages, then the uncited-gap estimates may reflect labeling bias rather than model behavior. The full text must provide a concrete annotation scheme and reliability statistics; the current abstract gives no basis to assess whether the outcome variable is measured consistently across models.","section":"Abstract, first paragraph"},{"comment":"The negative binomial hurdle model is mentioned but not specified in terms of covariates or identification. In particular, how is the 'selective log disclosure' effect for GPT-4o estimated? If the model includes a logging-artifact parameter for GPT-4o that is not identified from external data, then the subsequent comparison of uncited gaps across models is circular: the conclusion that GPT-4o's small gap is an artifact would be built into the model rather than tested. The abstract needs to clarify the model specification and the source of identification for any disclosure parameter.","section":"Abstract, second paragraph"}],"minor_comments":[{"comment":"The phrase 'exploitation patterns' has a normative tone; consider using a more neutral term such as 'observed patterns' or 'usage patterns' to avoid implying intentionality by the models.","section":"Abstract, first paragraph"},{"comment":"The scope of the dataset ('approximately 14,000 conversation logs') should be dated and described in terms of model versions, retrieval tool configurations, and time period, since LLM behavior changes quickly and the findings may not generalize beyond the specific systems tested.","section":"Abstract, first sentence"},{"comment":"The recommendation of 'full disclosure of search traces and citation logs' would be more actionable if paired with a measurable standard, such as a minimum set of telemetry fields or a machine-readable citation format; the abstract currently states the goal without operationalization.","section":"Abstract, final sentence"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important topic, but the abstract alone is insufficient to assess the validity of the central quantitative claims. The identification problem around selective log disclosure is a serious concern that must be resolved with additional evidence in the full text, such as validation against known ground-truth retrieval logs or sensitivity analyses that bound the effect of missing log entries. I recommend major revision rather than rejection because the central idea is defensible and the issue is potentially fixable, but the current presentation does not yet support the strong cross-model claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the data: roughly 14,000 real LMArena conversations, cross-model, same queries, with observed counts of fetched and cited URLs. That is a useful empirical map and a basis for accountability standards. The paper also does something right by modeling the gap as a hurdle process rather than just reporting means, and the citation-efficiency range (0.19–0.45) makes the point that retrieval design, not model capability, drives the shortfall.\n\nThe soft spot is exactly what the stress-test note says. The abstract explains GPT-4o's tiny uncited gap by 'selective log disclosures.' That is an admission that the logs do not record every fetched URL or every citation. If that is true for one model, the completeness assumption for the other models is not automatic; the gap magnitudes and the model ranking could be logging artifacts rather than behavioral differences. The paper needs per-model false-negative rates for logged fetches and citations, or at least a sensitivity analysis. Relevance labeling is also a black box from the abstract: who decides a fetched URL was 'relevant,' and is that decision made blind to whether it was cited? That could bias the gap measure. No confidence intervals or model diagnostics are visible, which matters for a measurement study where the entire claim is a set of point estimates.\n\nThose concerns are proportionate. I do not think the central finding is obviously wrong. The paper is alert to selective disclosure in one case, which suggests the authors are not naive about the data, but they have not yet shown the reader that the other estimates survive the same critique. The abstract is too thin to verify any of this; the full methods will decide.\n\nThis paper deserves a serious referee. The question is important, the dataset is real, and the analysis is at least thoughtfully framed. My own verdict would be skeptical until the log-completeness issue is resolved, but that is a question for the full text, not a reason to desk reject. If the methods address false-negative rates and relevance labeling, the paper could be a solid contribution to platform accountability research.\n\nFor you: worth reading the full version when it appears, and I would cite it if the measurement validity holds up. But I would not rely on the specific numbers until then.","headline":"A real-world measurement of LLM attribution gaps with a valuable dataset, but the headline numbers hinge on a log-completeness assumption the paper itself concedes is violated for one model.","tokens_in":1343,"tokens_out":1096,"would_cite":false,"duration_ms":14086,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Web-enabled LLMs routinely answer queries without citing the relevant web pages they consumed.","keywords":["attribution gap","LLM search","citation behavior","conversation logs","web-enabled language models","citation efficiency","search telemetry","content credit"],"falsifier":"If a provider released its own server-side traces for the same conversations and the attribution gap for GPT-4o was still near zero—and similarly small for Gemini and Sonar—the paper's central conclusion that large uncited gaps are hidden by selective disclosure would fail. Conversely, if server-side traces reveal even larger uncited fetch counts, the gap estimates in the paper are conservative.","tokens_in":592,"feed_emoji":"🔍","tokens_out":3586,"duration_ms":36734,"temperature":0.7,"pith_summary":"This paper tries to establish that web-enabled large language models systematically use web content without crediting it, and that this attribution gap is a design choice rather than a technical necessity. Analysing roughly 14,000 real LMArena conversation logs from search-enabled systems, the paper documents three patterns: Gemini and GPT-4o often produce answers without fetching any online content, Gemini almost never gives clickable citations, and Perplexity's Sonar visits about ten relevant pages per query while citing only three or four. The result matters because the same queries produce citation-efficiency scores ranging from 0.19 to 0.45 across models, meaning retrieval design, not hard technical limits, determines how much credit web sources receive. If true, the finding raises the pressure on search-engine providers to disclose their search traces and citation logs.","feed_headline":"Gemini and Sonar leave 3 relevant sites uncited per query","feed_subtitle":"14,000 real conversations show the gap is design choice, not tech limit.","key_machinery":"The load-bearing object is the attribution gap, defined as the difference between relevant URLs fetched by the model and URLs it actually cites. The measurement is carried by a negative binomial hurdle model, which first models whether a response produces any citations and then models how many citations it produces; this lets the authors separate 'no citation at all' from 'cites too few.' The data are roughly 14,000 LMArena conversation logs with search-enabled systems, with fetched URLs labelled as relevant or not from conversation context. This machinery turns an abstract concern about borrowing without credit into a comparable, per-query number across models.","core_discovery":"The paper's central claim is that an attribution gap—the difference between the relevant URLs a model reads and the URLs it actually cites—is large and systematic in current web-enabled LLM search engines. Using LMArena conversation logs, the paper finds that Gemini and Sonar leave roughly three relevant websites uncited on the average query, while GPT-4o's much smaller apparent gap is best explained by its selective log disclosures rather than by superior attribution. A negative binomial hurdle model separates the decision to cite anything at all from the number of citations given, showing that citation efficiency varies from 0.19 to 0.45 across models on identical queries. The paper concludes that high-volume, low-credit retrieval—fetching many pages while crediting few—is the dominant pattern for several major systems.","pith_inferences":["The paper's selective-disclosure caveat about GPT-4o may generalize: if logs are incomplete for any model, the measured gaps and model ordering are best read as lower bounds on the true attribution problems.","One testable extension is to measure whether uncited fetch volume predicts downstream harms like traffic diversion or content scraping, not just credit loss.","Citation-efficiency scores could become a practical consumer signal if providers published them per query type, nudging systems toward fewer high-credit citations rather than many invisible fetches."],"forward_implications":["If the claim holds, head-to-head benchmarks should not rank GPT-4o as the best-attributing model, because its small uncited gap may be an artifact of selective logging.","Citation efficiency becomes a design metric: models range from 0.19 to 0.45 extra citations per additional relevant page, so large gains are available without new retrieval ability.","Content producers can expect high-volume, low-credit systems to keep sending disproportionate traffic value to a few cited pages while many read pages go uncredited.","Standardized telemetry and full disclosure of search traces would let users and regulators audit which pages a model actually used, making attribution gaps independently verifiable."],"supporting_citations":[],"fun_headline_variants":["Gemini and Sonar skip crediting 3 URLs per query","34% of Gemini answers skip web search entirely","92% of Gemini replies lack clickable sources","Perplexity reads 10 pages, cites only 4 per query","Attribution gap: design choice, not tech limit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings assume the conversation logs accurately record every URL each model fetched and cited, and that the authors can correctly label which fetched pages were relevant from the conversation context.","fun_headline_variants_meta":{"raw":{"variants":["Gemini and Sonar skip crediting 3 URLs per query","34% of Gemini answers skip web search entirely","92% of Gemini replies lack clickable sources","Perplexity reads 10 pages, cites only 4 per query","Attribution gap: design choice, not tech limit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1740,"prompt_tokens":927,"completion_tokens":813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":732}},"tokens_in":543,"tokens_out":813,"duration_ms":9238,"temperature":1.0,"reasoning_tokens":732,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:05:29.967064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a provider released its own server-side traces for the same conversations and the attribution gap for GPT-4o was still near zero—and similarly small for Gemini and Sonar—the paper's central conclusion that large uncited gaps are hidden by selective disclosure would fail. Conversely, if server-side traces reveal even larger uncited fetch counts, the gap estimates in the paper are conservative.","supporting_citations":[],"review_version":1}