{"id":"05c80630-37d1-4115-bcda-7e8beaca6b19","arxiv_id":"2412.13612","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new evaluation framework finds that even the strongest LLMs still hallucinate a large share of references when asked to write literature reviews, with accuracy varying across disciplines.","lead":"Researchers built an automated test that asks five large language models to generate references, abstracts, and literature reviews, then checks those outputs against real papers and human-written Annual Reviews texts. Even the best model, Claude-3.5-Sonnet, produced invented or wrong references about half the time in the reference-generation task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic Scholar coverage gaps may inflate reported hallucination rates and confound the cross-discipline ranking; cross-validate with OpenAlex/Crossref.","rationale":"After reading the full manuscript, I agree with the reader that the weakest link is the external verification oracle. The paper's own limitations section flags incomplete retrieval. The human validation is the strongest counter-evidence, but with n=100 and no description of how the sample was drawn, it cannot rule out a systematic, discipline-correlated coverage bias. Other potential issues (the six-discipline/five-discipline inconsistency, NLI direction, missing error bars) are real but less central: they affect secondary metrics or presentation, not the reference-existence judgment that drives the main claim. The concern I raise is load-bearing because the primary numerical result (51.59% precision) and the secondary cross-discipline ranking both depend on Semantic Scholar's completeness. If the proposed OpenAlex/Crossref cross-validation shows large disagreement, the paper's headline should be softened to 'LLMs produce many references that cannot be verified' rather than 'LLMs hallucinate references,' and the discipline-level analysis should be re-run controlling for database coverage. If the cross-validation shows agreement, the CONDITIONAL verdict can be upgraded. Since this is a verification-dependent empirical claim and the authors themselves flag the limitation, keeping the CONDITIONAL verdict is appropriate.","tokens_in":14685,"tokens_out":8587,"duration_ms":78540,"concrete_test":"From the released dataset, take the full set of LLM-generated references for the Reference Generation task (or a stratified random sample of 1,000 per model, balanced across the six reported disciplines). For each reference, query the OpenAlex and Crossref APIs using the generated title and first author, in addition to Semantic Scholar, and apply a matching rule analogous to Eq. 1 (title similarity ≥0.8 plus one matching field, or ≥3 fields if the title fails; accept the reference as real if any of the three databases confirms it). Recompute per-model and per-discipline precision. If Claude-3.5-Sonnet's precision rises above 65% or the Chemistry-vs-Mathematics ordering reverses, the Semantic Scholar coverage confound is confirmed; if all estimates move by less than 3 percentage points, the reported metrics are robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—that the best model (Claude-3.5-Sonnet) achieves only 51.59% precision on generated references—depends entirely on Semantic Scholar as the oracle for whether a reference is real. The Limitations admit that 'this may have resulted in incomplete reference retrieval.' Any real reference missing from Semantic Scholar is automatically labeled hallucinated by Eq. 1. Three consequences are load-bearing. First, the 51.59% figure could be a substantial underestimate: St (title search rate) is 64.82% for Claude, so over a third of generated titles are not found at all; some of these could be real but unindexed, especially older or field-specific work. Second, the cross-discipline result (Mathematics highest, Chemistry lowest) is confounded: if Semantic Scholar indexes mathematics literature more completely than chemistry, the observed precision gap would reflect database coverage, not model capability. The authors even report a positive correlation between citation counts and precision, which is plausibly a coverage artifact. Third, the human evaluation (100 references, 86% agreement, kappa 0.71) is too small and too thinly described to rule out this bias; a 14% disagreement rate can move the headline precision by several points. The qualitative conclusion that hallucination is nonzero would likely survive, but the exact numbers and the discipline ranking would not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces an automatic evaluation framework for LLM literature review writing, covering three tasks: reference generation, abstract writing, and review composition. Using 1,105 literature reviews from 51 Annual Reviews journals (2023) and five LLMs, it measures reference hallucination via Semantic Scholar verification (precision, recall, F1, title search rate) and evaluates text quality via similarity metrics, ROUGE, NLI-based factual consistency (TRUE and GPT-4o as judges), and Key Point Recall (KPR) with GPT-4-extracted key points. The central finding is that even the best model, Claude-3.5-Sonnet, achieves only 51.59% precision on reference generation (Table 1), and that performance varies significantly across disciplines.","tokens_in":14922,"tokens_out":6889,"duration_ms":62648,"significance":"If the results hold, the paper provides a practically important, actionable conclusion: vanilla LLMs should not be trusted to produce bibliographies without external verification. The framework is a useful contribution with public dataset/code, clear task decomposition, and multidimensional metrics. Strengths include the use of an external database (Semantic Scholar) to ground the hallucination evaluation, a (small) human validation study, and statistical tests of cross-discipline differences. However, the exact precision numbers and the discipline ranking are vulnerable to Semantic Scholar coverage gaps, and the NLI-based metrics in Eqs. (6) and (8) contain argument-order issues that call their current results into question.","major_comments":[{"comment":"The reference-verification oracle is Semantic Scholar alone, and Eq. (1) classifies every reference that does not match as hallucinated. The Limitations paragraph concedes that 'this may have resulted in incomplete reference retrieval.' Table 1 shows St = 64.82% for Claude-3.5-Sonnet, meaning over a third of generated titles are not found at all; some of these are likely real but unindexed references. This systematically deflates Precision and St. The cross-discipline ranking (Figure 4a, Table 4) is also confounded: the positive correlation between precision and average citation counts (r = 0.40 and 0.51) is precisely the pattern expected if database coverage drives match rates. To support the headline 51.59% figure and the discipline ordering, the authors should cross-validate a stratified sample (especially all not-found references) against OpenAlex or Crossref and report the rate of real references that are missed by Semantic Scholar.","section":"§3.3, Eq. (1), Table 1, and Limitations"},{"comment":"The NLI direction in Eq. (6) is inverted relative to standard factual-consistency evaluation. As written, Entail_pi = 1 if the LLM-generated abstract entails the original human abstract, whereas factual consistency of a generated summary is normally assessed as the source (human abstract) entailing the generated claim. This reversal, combined with the use of GPT-4o (an evaluated model) as the NLI judge, likely explains the surprisingly high GPT-4o-based Entail scores (92–97% in Table 2). The authors should either justify the direction with a human pilot or reverse the arguments to θ_NLI(a_i, a_g_i), and report agreement between the NLI judge and human annotations for the abstract task.","section":"§3.3, Eq. (6), Table 2"},{"comment":"The KPR formula in Eq. (8) does not measure what the text claims. It computes θ_NLI(x, c_g_i), i.e., whether each key point x entails the generated review c_g_i, but coverage should be measured as whether the generated review entails or contains the key point, i.e., θ_NLI(c_g_i, x). As written, KPR scores in Table 3 and Figure 4d cannot be interpreted as semantic coverage. Additionally, the key-point extractor (GPT-4) and the NLI judge (GPT-4o) include an evaluated model, introducing circularity. Please correct the argument order and re-run the analysis, and consider an independent judge for robustness.","section":"§3.3, Eq. (8), Table 3"},{"comment":"The human validation of the hallucination metric is too thin to rule out the Semantic Scholar coverage bias. Only 100 LLM-generated references were annotated by three raters (86% agreement, kappa = 0.71), and no breakdown is reported for references whose titles were not found by Semantic Scholar. A 14% disagreement rate can shift the headline precision by several points, and the sample does not allow estimation of the false-hallucination rate among St-negative cases. The authors should enlarge the sample, stratify it by title-search outcome, and report precision after excluding or reclassifying non-verifiable references.","section":"§4.5"}],"minor_comments":[{"comment":"The abstract and contributions state that the evaluation framework requires 'no human involvement,' but the 80% title-match threshold in Eq. (1) is said to be 'determined through human evaluation' and §4.5 reports a human validation study; please qualify the claim.","section":"§1 and §3.3"},{"comment":"The text says 'we categorize 51 journals into five disciplines' but then lists six categories (Biology, Chemistry, Mathematics, Physics, Social Science, and Technology); also the reported article counts (460+90+50+113+299+94 = 1106) do not match the stated total of 1,105.","section":"§4.4"},{"comment":"The caption states 'four LLMs,' but the table lists five models; the same wording appears in Appendix D, which says 'We select four LLMs for task evaluation.'","section":"Table 3 caption"},{"comment":"Eq. (5) defines s^(n)_pi with a sum over r_g^n ∈ R_g^i, which is notationally inconsistent because s^(n) is already indexed by n; please clarify the index structure.","section":"§3.3, Eq. (5)"},{"comment":"One-way ANOVA is used with highly unequal group sizes (e.g., Mathematics n=50, Biology n=460) and no mention of assumption checks; report effect sizes or non-parametric tests for robustness.","section":"§4.4"},{"comment":"Several reference entries are incomplete or contain typos (e.g., Chen et al. 2024 lacks venue; 'Kukliansy' should likely be 'Kukliansky'); please proofread the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core message—that LLM-generated bibliographies contain high rates of hallucination—is likely correct in direction, but the paper overstates the exact numbers without addressing Semantic Scholar coverage and contains two NLI argument-order errors that undermine the abstract and KPR metrics. These are fixable with re-analysis and additional validation, which the public code should enable. The paper fits cs.CL scope and should be considered after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a useful, well-scoped evaluation framework for LLM-generated literature reviews, with a genuine new dataset, and the main qualitative finding—that current LLMs still hallucinate a large share of references—is credible. But the exact numbers and the cross-discipline ranking are softer than the paper suggests, because the oracle is Semantic Scholar and the authors admit its retrieval is incomplete.\n\nWhat's genuinely new: 1,105 review articles from 51 Annual Reviews journals across five (well, six—see below) disciplines; three tasks (reference generation, abstract writing, review composition); and a combination of existing metrics—Semantic Scholar title/metadata matching, NLI, similarity, ROUGE, KPR—into one harness. The human evaluation on 100 references (86% agreement, kappa 0.71) is a real check, and they report per-dimension accuracy and discipline-level ANOVAs. The code and data are in an anonymous repo, so the work is reproducible in principle, though I couldn't verify that without access.\n\nSoft spots, in rough order of severity. First, the reference precision depends entirely on whether Semantic Scholar has the paper. Eq. 1 labels a reference hallucinated if it doesn't match a candidate retrieved from S2. For Claude, St is 64.82%, so over a third of generated titles don't turn up at all; some of those are likely real but unindexed. The 14% human/automatic disagreement could move the headline precision by several points, and the discipline ranking (Math highest, Chemistry lowest) is plausibly a coverage artifact—the positive correlation between citation count and precision is exactly what you'd expect if popular papers are better indexed. Second, the NLI direction in Eq. 6 is odd: it counts as consistent when the generated abstract entails the original, which seems backwards, and no validation is reported. Third, the paper says \"five disciplines\" but lists six. Fourth, there are no error bars or confidence intervals; just point estimates. The GPT-4o-as-judge overlap is a minor circularity, since they also use TRUE, so it doesn't bother me much.\n\nNone of this kills the central claim—even with generous corrections, the best model is nowhere near reliable enough to trust for bibliographies. But it does mean the headline 51.59% is a floor, not a point estimate, and the discipline differences shouldn't be interpreted as real until cross-validated with OpenAlex or Crossref.\n\nI'd send this to review, with a request for major revision: cross-check references against at least one other API, fix the NLI direction, add uncertainty quantification, and clean up the discipline count. For readers working on LLM evaluation or scholarly-writing tools, it's worth a look in its current form; for the rest, the abstract captures the takeaway.\n\nBest.","headline":"Useful benchmark for LLM literature-review evaluation, but the headline hallucination rates are only as good as Semantic Scholar's coverage; the qualitative result is solid, the exact numbers are not.","tokens_in":15451,"tokens_out":2830,"would_cite":true,"duration_ms":23749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even the best LLM fabricates nearly half its generated references","keywords":["LLM hallucination","reference generation","literature review","automatic evaluation","factual consistency","semantic coverage","citation verification","Annual Reviews dataset"],"falsifier":"Manually verify a random sample of the references the pipeline labels as hallucinated using independent bibliographic databases and publisher records; if more than a small fraction turn out to be real publications under slightly different titles or metadata, the reported hallucination rates are inflated.","tokens_in":14466,"feed_emoji":"📚","tokens_out":9339,"duration_ms":75634,"temperature":0.7,"pith_summary":"This paper asks whether large language models can be trusted to automate the writing of literature reviews, and it finds that they cannot yet be trusted to produce reliable bibliographies. The authors build an automatic benchmark from 1,105 human-written review articles across five disciplines and test five LLMs on three tasks: generating references, writing an abstract, and composing a full review with citations. The central finding is that hallucinated references remain common even for the strongest model: Claude-3.5-Sonnet, the best performer, achieves only 51.59% precision on standalone reference generation, meaning nearly half of its citations do not match verifiable publications under the paper's matching rule. Accuracy improves markedly when models write a review and cite sources within the same output, but even then the best precision reaches about 59%. These results matter because researchers routinely use vanilla LLM outputs to compile bibliographies, and the paper shows that unverified reference lists from these models are not reliable.","feed_headline":"Even the best LLM fabricates nearly half its generated references","feed_subtitle":"An automated benchmark on 1,105 human-written reviews shows invented citations remain common even for top models.","key_machinery":"The load-bearing mechanism is an automatic evaluation pipeline anchored on a gold-standard corpus: 1,105 human-written literature reviews crawled from 51 Annual Reviews journals, each providing title, keywords, abstract, context, and reference list. For reference accuracy, every generated citation is checked against Semantic Scholar via Eq. 1's two-part rule, requiring either a title match of at least 80% plus one matching metadata field, or at least three matching non-title fields; this yields precision, recall, F1, and a title-search rate. For text quality, the pipeline uses NLI models (TRUE and GPT-4o) to score factual consistency, embedding cosine similarity and ROUGE for semantic coverage, and Key Point Recall, which extracts key points from human reviews with GPT-4 and scores entailment of each point in the model-generated review. The mechanism makes the evaluation fully automatic and reproducible.","core_discovery":"The paper's central claim is that state-of-the-art large language models still hallucinate a substantial share of references when asked to produce literature reviews, and that this failure is measurable, systematic, and discipline-dependent. Using human-written Annual Reviews articles as the gold standard, the framework verifies each generated reference against Semantic Scholar through a two-condition matching rule: a title matched at 80% plus at least one other metadata field, or at least three non-title fields matching. Across five models, the best standalone reference precision is 51.59% (Claude-3.5-Sonnet), with only 64.82% of generated titles retrievable from an external search engine. When models write the review text and the citations together, precision rises across models (Claude-3.5-Sonnet to 59.06%), which the authors attribute to the mutual constraint between cited text and reference metadata. Factual consistency and semantic coverage against human-written abstracts also vary by model and discipline, with mathematics yielding the most accurate references and chemistry the least in the reference-generation task.","pith_inferences":["If the paper's verification database is incomplete, the measured hallucination rates are upper bounds; re-running the verification against independent bibliographic databases would separate database misses from model fabrication.","The positive correlation between citation counts and reference precision suggests models are most accurate for famous, heavily indexed papers, so hallucination rates may be higher for niche topics and non-English literatures.","The same benchmark could be applied to retrieval-augmented generation pipelines to quantify how much retrieval grounding closes the gap, though this paper deliberately evaluates only vanilla LLMs.","Because the dataset is fixed to 2023 reviews and the authors note possible training-data overlap, extending the benchmark to recent papers would measure whether newer model generations improve on hallucination."],"forward_implications":["Standalone reference generation from a vanilla LLM should be treated as unverified: even the best tested model produces a verifiable citation only about half the time.","Generating references inside a written review improves precision for most models, with Claude-3.5-Sonnet rising from 51.59% to 59.06%, suggesting citation-grounded writing constrains hallucination.","Errors are not evenly distributed: reference accuracy differs significantly across disciplines, with mathematics highest and chemistry lowest, and models show different disciplinary strengths.","Automatic hallucination detection aligns with human judgment: on 100 references, the automated method agreed with majority human annotation at 86% accuracy with a kappa of 0.71.","Even in the grounded review-composition setting, the best precision stays below 60%, so LLM-produced bibliographies will need external verification before academic use."],"supporting_citations":[{"why":"Supplies the search-API verification approach for generated reference titles that the paper adapts into its title search rate.","marker":"Agrawal et al. (2024)"},{"why":"Prior measurement of hallucination rates in LLM-generated systematic reviews that frames the reference-generation task.","marker":"Chelli et al. (2024)"},{"why":"Earlier manual validation showing ChatGPT generates invalid DOIs and references, motivating automated hallucination detection.","marker":"Athaluri et al. (2023)"},{"why":"Evidence that grounding generated text with real citations reduces hallucination, used to interpret why review composition improves precision.","marker":"Gao et al. (2023)"},{"why":"Provides the TRUE benchmark and NLI approach used to score factual consistency of abstracts.","marker":"Honovich et al. (2022)"},{"why":"Introduces Key Point Recall, the metric used to measure how much of human-written reviews the generated reviews cover.","marker":"Qi et al. (2024)"}],"fun_headline_variants":["Top LLMs still fabricate nearly half their references","Even best LLM review writers invent citations","AI literature reviews: 48% of references hallucinated","LLM-generated reviews cite fake papers, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline hallucination rate assumes the verification database and the paper's matching rule catch every real reference; if a real reference is missing from the index or fails the matching rule, the model is falsely counted as hallucinating.","fun_headline_variants_meta":{"raw":{"variants":["Top LLMs still fabricate nearly half their references","Even best LLM review writers invent citations","AI literature reviews: 48% of references hallucinated","LLM-generated reviews cite fake papers, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2500,"prompt_tokens":918,"completion_tokens":1582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":1520}},"tokens_in":534,"tokens_out":1582,"duration_ms":11819,"temperature":1.0,"reasoning_tokens":1520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:57:05.721297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually verify a random sample of the references the pipeline labels as hallucinated using independent bibliographic databases and publisher records; if more than a small fraction turn out to be real publications under slightly different titles or metadata, the reported hallucination rates are inflated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the search-API verification approach for generated reference titles that the paper adapts into its title search rate."}],"review_version":1}