{"id":"88cd60a3-9464-4de5-b9b7-eb8f340f76a0","arxiv_id":"2501.10483","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"ArxEval reports that 15 small language models hallucinate frequently on jumbled and mixed arXiv titles, but internal data errors and missing baselines prevent the quantitative rankings from being trusted.","lead":"This paper introduces ArxEval, a benchmark that asks 15 small open-source language models to interpret scrambled or merged arXiv paper titles, then checks whether their responses and DOIs match real papers. It reports low abstract-recovery similarity and mostly invalid or mismatched DOIs, but the benchmark's internal errors and missing baselines keep the quantitative results from being trusted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"0% title-match rate likely an artifact: the paper never specifies title normalization; exact-string comparisons would make any formatting difference count as hallucination.","rationale":"The reader rejected the paper for multiple internal inconsistencies, including dataset-size contradictions, Table 5 counts that do not add up, and the absence of released code or data. My stress-test focuses on the strongest claim itself: the 100% title-mismatch rate. This claim is surprising because it asserts that across 15 models and 497 valid DOIs, not a single title matched. Genuine hallucination rates that extreme are possible, but the paper gives no details about the matching procedure, and a perfect 0% across all models is highly suggestive of a systematic string-comparison artifact. The concern is concrete and testable: normalize strings and re-run the comparison on the same outputs. If matches appear, the headline claim collapses; if they still do not, the claim is strengthened. Either way, the paper as written lacks the data and code needed to verify the result, so the reader's REJECT verdict stands. I disagree with the reader's choice of weakest assumption: the ill-posedness of the Mixed Titles task primarily affects the denominator of the 17.75% valid-DOI rate and the 530 expected DOIs, but the 100% title mismatch could still hold even under an ill-posed task. The load-bearing assumption for the 100% claim is the title-matching evaluation itself, not the task's grounding.","tokens_in":13884,"tokens_out":5618,"duration_ms":53663,"concrete_test":"Request the raw model outputs and evaluation script from the authors. For each of the 497 valid DOIs in Table 5, re-run the title matching with normalization: lowercase, strip punctuation and whitespace, collapse LaTeX escapes (e.g., \\'a to a), and optionally use fuzzy matching or alias resolution. If any normalized title matches the DOI's official title, the 0% figure is refuted. Also compute the match rate under both strict and loose comparison to quantify sensitivity.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim that \"for every valid DOI retrieved, the associated title was incorrect 100% of the time\" rests on the title-comparison step described in Section 5 and Table 5. The paper does not report how the model-generated title and the API-retrieved official title were compared: no normalization, no string-matching method, no fuzzy matching, and no manual review are described. Because model outputs are free text gathered from prompts like \"only mention the Title and the DOI,\" the parsed title field is likely to contain trailing punctuation, case differences, LaTeX escapes, or subtitle truncation. Official API titles can have the same variations. If the comparison is a raw exact string equality, all 497 valid DOI/title pairs could fail not because the model hallucinated the title but because of formatting. That would make the 100% claim an evaluation artifact rather than a substantive finding. The conclusion that \"Every model also completely failed to generate the corresponding DOI for the title they generated\" depends entirely on this matching procedure, so this is the most load-bearing assumption behind the strongest claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ArxEval, a pipeline for evaluating hallucination in language models on scientific literature. It introduces two tasks: Jumbled Titles, where a model is given a shuffled paper title and asked to describe the paper, and Mixed Titles, where a model is given a shuffled concatenation of two titles and asked to return exactly two papers with their DOIs. Fifteen open-weight models are evaluated; the paper reports aggregate similarity scores for Jumbled Titles and DOI validity/title-match rates for Mixed Titles. The two central quantitative claims are that valid DOIs are generated only 17.75% of the time and that for every valid DOI retrieved, the associated title is incorrect 100% of the time across all models.","tokens_in":14017,"tokens_out":5958,"duration_ms":55316,"significance":"If the quantitative claims held, the paper would provide useful evidence on reference hallucination in LLMs, and the comparison across fifteen models could inform practical choices about model size. The work has genuine strengths: it validates DOIs against external repositories (Crossref, DataCite, UnPaywall, OpenAlex), covers a broad set of models, and openly discusses limitations such as quantization. However, the load-bearing conclusions currently rest on an unspecified title-comparison procedure and on inconsistent dataset and result numbers. The qualitative signal, that models invent plausible references from scrambled inputs, is credible and consistent with prior work, but the paper's specific quantitative claims are not supported as written.","major_comments":[{"comment":"Section 5 states that for every valid DOI retrieved, the associated title was incorrect 100% of the time, and Table 5 reports 0.00% Matching Titles for all models. The paper never specifies how the model-generated title was compared with the API-retrieved title: no normalization, no string-matching method, no fuzzy matching, and no manual review are described. Because model outputs are free-text responses to a prompt asking to 'only mention the Title and the DOI,' parsed titles are likely to differ from official titles in superficial formatting such as case, punctuation, whitespace, or LaTeX escapes. Without a specified matching protocol, the 0.00% rates are uninterpretable, and the claim in Section 6 that every model 'completely failed to generate the corresponding DOI for the title they generated' is unsupported.","section":"Section 5, Table 5"},{"comment":"Section 3 states that 176 categories were used with 3 titles from each category, giving 528 titles; Section 4.1, however, says 'We select 5 titles from each category.' These assumptions produce different dataset sizes. They also do not match the Mixed Titles count: pairing 528 titles yields 264 mixed titles, not the 265 reported in Table 3 and used in Section 6 (where 265 mixed titles imply an expected output of 530 DOIs). The inconsistency must be resolved because the expected-output denominator drives the validity-rate interpretation.","section":"Section 3 vs. Section 4.1, Table 3"},{"comment":"Table 5 contains internally inconsistent rows. For Orca-2 (7B), Total DOIs is 176 but DOIs Found (20) plus DOIs Not Found (86) equals 106, and the reported 18.87% is 20/106 rather than 20/176. The Llama-3 row lists 8 DOIs not found although 112 total minus 9 found is 103. The Section 6 statement that 'valid DOIs were generated only 17.75% of the time' does not match any aggregate calculable from Table 5: summing the rows gives 497 valid out of 2,254 total (22.05%), while the unweighted mean of the per-model validity percentages is about 19.6%. The authors should report the exact formula used and correct the table.","section":"Table 5, Section 6"},{"comment":"The Mixed Titles task assumes that a shuffled concatenation of two titles has a determinate correct answer: exactly two real papers and their two DOIs. The prompt 'Tell me 2 papers related to this' on a word-salad input is ill-posed; a model that responds with one relevant paper or states that the input is not a valid title is penalized as hallucinating. This conflates prompt compliance and question clarity with hallucination, and it affects the benchmark denominator (530 expected DOIs). The authors should justify the determinate-answer assumption or, at minimum, provide a breakdown of responses by number of DOIs and analyze partial responses separately.","section":"Section 4.2, Section 6"}],"minor_comments":[{"comment":"The introduction says fifteen models are evaluated but lists only eleven; Table 4 includes fifteen. Complete the list or correct the count.","section":"Section 1"},{"comment":"The Gunning Fog scores in the text (17 for Jumbled Titles, 19 for Mixed Titles) differ from Table 3 (17 and 20). Align the text and the table.","section":"Section 3, Table 3"},{"comment":"The Llama-3 row shows '8 [91.96%]' for DOIs Not Found; this should be 103 [91.96%] to be consistent with the total and found counts.","section":"Table 5"},{"comment":"Algorithm 1 does not show how categories are selected or how many titles are taken per category, despite Section 4.1 stating '5 titles from each category'; clarify the sampling procedure.","section":"Algorithm 1, Section 4.1"},{"comment":"Percentages for DOIs Not Found use inconsistent denominators across rows; for Orca-2 (7B) the not-found percentage is based on 106 rather than the reported total of 176. Use a single consistent denominator.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CL as an evaluation study, but the current version is not publishable as is: the central quantitative claims depend on an unspecified title-comparison step, the dataset description is internally inconsistent, and Table 5 contains arithmetic errors. These issues are fixable with a rerun or reanalysis of the evaluation, so I do not recommend rejection; however, the revision must include a precise description of the matching protocol, corrected numbers, and a discussion of the Mixed Titles task premise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth knowing about only as a cautionary example. The qualitative direction is credible—small models do invent plausible references when given scrambled paper titles—but the quantitative benchmark as reported is not trustworthy. The headline claim that valid DOIs are matched to correct titles 0% of the time is unsubstantiated because the paper never specifies how titles were compared. No normalization, no fuzzy matching, no manual review. With free-text model outputs, exact-string comparison would make any formatting difference look like hallucination. That could turn a real effect into an artifact.\n\nWhat is new is narrow: two perturbed-title tasks (Jumbled and Mixed) that I have not seen in the cited prior work, plus a fifteen-model comparison with DOI validation through Crossref, DataCite, UnPaywall, and OpenAlex. That is a reasonable effort, and the jumbled-title setup is an original way to probe hallucination, even if it is a simple combination of existing ideas.\n\nThe soft spots are serious and load-bearing. Table 5 has rows that do not add up: Llama-3 shows 9 found plus 8 not found with a total of 112; Orca-2 7B shows 20 and 86 with a total of 176. The dataset description is internally inconsistent—Section 3 says 3 titles per category (528 total), Section 4.1 says 5 per category. The Mixed Titles task is ill-posed: a shuffled concatenation of two titles has no determinate correct answer. Scoring every output against an expected 530 DOIs assumes the model should produce exactly two papers from a word salad. Low validity rates here may measure prompt compliance more than hallucination. No code, data, or seeds are released, and there are no error bars.\n\nWho gains from this? A reader wanting anecdotal examples of hallucinated references might find the qualitative illustrations useful. No one should cite this as a reliable comparative benchmark.\n\nRecommendation: this is not ready for referees. The matching procedure, the arithmetic, and the dataset count all need to be fixed, and the artifacts should be released. If those are addressed, it could become a minor but usable benchmark.","headline":"Plausible qualitative signal, but the benchmark's central quantitative claim is unverified and the main results table has arithmetic errors.","tokens_in":14579,"tokens_out":4555,"would_cite":false,"duration_ms":38877,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that across fifteen open language models, every valid DOI generated in its Mixed Titles task was paired with a wrong title, and that only 17.75% of generated DOIs existed.","keywords":["hallucination evaluation","scientific literature","language models","citation generation","DOI validation","title retrieval","embedding similarity","prompt adherence"],"falsifier":"Take a random sample of the Mixed Titles responses, rescore the titles against the validated DOIs using fuzzy string similarity, and have human raters judge whether one-DOI or zero-DOI answers to a shuffled word salad are reasonable; a large jump in the title-match rate or many accepted non-compliant answers would show the reported 100% and 17.75% figures depend on exact matching and on the fixed two-paper expectation.","tokens_in":13649,"feed_emoji":"📚","tokens_out":7635,"duration_ms":74296,"temperature":0.7,"pith_summary":"This paper tries to measure how often open language models fabricate scientific references when they handle altered paper titles. It proposes a benchmark with two tasks built from real preprint titles: one scrambles a single title and asks the model to describe the paper, the other shuffles two titles together and asks the model to return the two papers with their DOIs. The headline result is that only 17.75% of generated DOIs were real, and every real DOI was paired with an incorrect title across all fifteen models. A reader should care because academic and educational use of these models depends on citation-level accuracy, and this evidence says that accuracy is not there yet.","feed_headline":"15 models got every title wrong even when DOIs were real","feed_subtitle":"A benchmark using scrambled real titles finds open models cannot reliably cite scientific papers","key_machinery":"The carrying mechanism is the ArxEval pipeline, a two-task benchmark over 528 real titles from 176 preprint categories. The Jumbled Titles task shuffles the words of a title, asks the model to describe the paper, and scores the response against the true abstract with cosine similarity, BERTScore, and semantic textual similarity. The Mixed Titles task shuffles two titles into one string, asks for two paper titles plus DOIs, checks each DOI against bibliographic databases, and then compares the model's title for each valid DOI against the database title. The 100% title-mismatch finding is computed against exact title equality after DOI validation.","core_discovery":"The central claim is that current open language models systematically hallucinate when retrieving and generating scientific literature under input distortion. In the Jumbled Titles task, the best model averaged 0.585 similarity, and models clustered in a moderate band rather than recovering the papers. In the Mixed Titles task, none of the fifteen models produced the requested two correct title–DOI pairs; valid DOIs appeared only 17.75% of the time on average, and every valid DOI was associated with a title that did not match the database record. The paper presents these results as evidence that factual consistency in domain-specific scientific prompting is a critical unsolved problem.","pith_inferences":["The authors do not test fuzzy title matching; a rerun that allows paraphrases or near-matches could show how much of the 0% title accuracy is exact-match strictness rather than fabrication.","Because the Mixed Titles prompt asks models to answer a word salad as though it names two papers, low output counts and partial answers may partly reflect a model treating the query as ill-posed, which would make the 17.75% figure a measure of instruction-following as well as hallucination.","The same benchmark could be run with retrieval augmentation or web search available to the model, which would separate failures of stored knowledge from failures of generation and citation formatting."],"forward_implications":["Users cannot safely copy bibliographic references from these models without checking every DOI against a database.","A real DOI is not proof of a reliable citation, because the accompanying title was wrong in every verified case.","The two-task protocol gives a reusable way to compare future open models on scientific retrieval and generation fidelity.","Larger parameter counts did not reliably improve performance, so scaling alone is unlikely to fix citation hallucination."],"supporting_citations":[{"why":"It supplies the real paper titles and abstracts used to build both evaluation tasks.","marker":"[20]"},{"why":"It provides the sentence-embedding model used to compute cosine and semantic textual similarity for the Jumbled Titles task.","marker":"[34]"},{"why":"It provides the BERTScore metric used alongside cosine and semantic textual similarity in the Jumbled Titles evaluation.","marker":"[35]"},{"why":"It documents the baseline finding that models generate fabricated citations, which the Mixed Titles task extends to title–DOI pairs.","marker":"[31]"},{"why":"It provides an existing domain-specific hallucination detection benchmark that frames the need for a new evaluation pipeline.","marker":"[25]"},{"why":"It provides an existing biomedical hallucination benchmark whose scoring approach the paper positions itself against.","marker":"[26]"},{"why":"It supports the premise that models hallucinate even when asked about known facts, motivating the two tasks.","marker":"[21]"}],"fun_headline_variants":["15 models, zero correct title–DOI pairs","All 15 models failed every real DOI–title match","Benchmark: open LLMs hallucinate even with valid DOIs","No open model correctly cited a single real paper","Real DOIs don't fix LLM citation hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Mixed Titles task assumes a word salad made from two titles has exactly one right answer—those two papers with two DOIs—so any other response is scored as hallucination; if models instead treat the question as ill-posed, the headline rates measure prompt compliance rather than hallucination.","fun_headline_variants_meta":{"raw":{"variants":["15 models, zero correct title–DOI pairs","All 15 models failed every real DOI–title match","Benchmark: open LLMs hallucinate even with valid DOIs","No open model correctly cited a single real paper","Real DOIs don't fix LLM citation hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1279,"prompt_tokens":789,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":405,"tokens_out":490,"duration_ms":5429,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:30:11.346678+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the Mixed Titles responses, rescore the titles against the validated DOIs using fuzzy string similarity, and have human raters judge whether one-DOI or zero-DOI answers to a shuffled word salad are reasonable; a large jump in the title-match rate or many accepted non-compliant answers would show the reported 100% and 17.75% figures depend on exact matching and on the fixed two-paper expectation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It documents the baseline finding that models generate fabricated citations, which the Mixed Titles task extends to title–DOI pairs."},{"cited_title":"Delucionqa: Detecting hallucinations in domain-specific question answering, 2023","cited_arxiv_id":null,"evidence_quote":"It provides an existing domain-specific hallucination detection benchmark that frames the need for a new evaluation pipeline."},{"cited_title":"Dahl: Domain-specific automated hallucination evaluation of long-form text through a benchmark dataset in biomedicine, 2024","cited_arxiv_id":null,"evidence_quote":"It provides an existing biomedical hallucination benchmark whose scoring approach the paper positions itself against."}],"review_version":1}