{"id":"e57a2eb8-46dc-4137-acfc-1f5656b3dc27","arxiv_id":"2607.09774","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"AtomGPT's hybrid LLM+Semantic Scholar checker flags 94 of 100 confirmed hallucinated NeurIPS 2025 citations, driven mainly by author mismatch rather than title similarity.","lead":"A free web tool checks scientific citations by parsing them with an LLM and matching fields against Semantic Scholar, flagging most known fakes from NeurIPS 2025. Editors and reviewers need scalable ways to catch fabricated references that already slip into published papers.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the paper's own limitations discussion.","rationale":"The paper's strongest claim is a well-supported empirical recall result plus a clear per-field mechanism on an external, expert-verified set of real fabrications. The reader correctly identifies the composite weights and missing-field rule as the weakest design assumptions and notes that they produce three of the six misses. Because the authors themselves diagnose exactly this issue in Section 4.1 and supply the per-citation scores needed to test alternatives, the concern is already internalized rather than an unacknowledged threat to the central claim. No additional load-bearing inconsistency, coverage failure, or unsupported leap appears. The positives-only design, single live index, and stochastic extraction remain real boundaries on what can be claimed about precision or production readiness, which is why the reader's CONDITIONAL verdict is appropriate and needs no adjustment. The concrete re-weighting check would simply confirm the authors' own diagnosis without altering the recall or mechanism results.","tokens_in":11512,"tokens_out":526,"duration_ms":5628,"concrete_test":"Re-score the six verified (false-negative) citations from Table 2 after changing only the deterministic stage: set author weight to 0.50 and title weight to 0.35, and treat a missing author field as a hard partial-status trigger rather than substituting the title score. If all six move out of the verified band while the overall detection rate on the 100 remains ≥90 %, the original weights were the sole cause of the misses and the claim is robust under a corrected scoring rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption correctly flags the title-heavy weights (Eq. 1: title 60 %, authors 25 %, journal 15 %) and missing-field substitution rule as the source of three of the six false negatives (Table 2). However, this is not a hidden soft spot: Section 4.1 already states the misalignment, notes that three misses arise directly from the rule, and proposes re-weighting or an author veto as future work that can be tested on the existing data. The central claim is scoped as a recall result on known positives (94/100 flagged) plus a mechanistic account that author mismatch is the dominant signal (Figure 4). Both are supported by the reported counts, distributions, and the independent Ansari taxonomy alignment. The design choices limit precision and production readiness, but they do not undermine the stated claim under the operating point the authors evaluate.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents and evaluates the AtomGPT reference checker, a web tool that parses free-form citations with an open-weight LLM (gpt-oss-20b, with regex fallback), retrieves candidates from Semantic Scholar, and scores title/author/journal agreement via a fixed weighted composite (Eq. 1: 60/25/15) to assign verified / partial / likely-hallucinated labels. Evaluation is a citation-level recall test on the external GPTZero NeurIPS 2025 set of 100 human-verified hallucinated citations: 66 are labeled likely hallucinated, 28 partial, and 6 verified (94% detection when partial is counted as a flag). A per-field decomposition shows near-zero author similarity as the dominant signal despite high title similarity for fabrications; the six misses are analyzed in Table 2 and attributed mainly to the missing-field substitution rule and title-heavy weights. Limitations (positives-only design, LLM stochasticity, single-index dependence, weight misalignment) are stated explicitly, with public code and data.","tokens_in":11689,"tokens_out":971,"duration_ms":10555,"significance":"Fabricated citations are already entering top-tier proceedings; an open, lightweight, retrieval-grounded checker with an external expert-verified benchmark and a clear mechanistic account (author mismatch as the decisive field) is a useful contribution to research integrity tooling. Strengths include the independent GPTZero ground truth, the alignment with Ansari’s manual taxonomy, the public web tool and evaluation notebook, and the honest scoping as recall-only rather than a precision claim. The work is incremental relative to CheckIfExist but supplies the quantitative external benchmark that system lacked. If the reported detection behavior holds under modest re-weighting and multi-source checks, the tool is a practical drop-in for editorial pipelines.","major_comments":[{"comment":"The evaluation is correctly scoped as recall on known positives, yet the abstract and conclusion still phrase the result as the tool “reliably flags the great majority” without always restating the operating point (partial counted as detection) or the absence of a false-positive estimate. A short, explicit statement in the abstract and §5 that precision remains unmeasured would prevent over-reading of the 94% figure.","section":null},{"comment":"§2 Eq. (1) and the missing-field rule (substitute title score for unextracted author/venue) are load-bearing: Table 2 shows three of the six verified misses arise directly from this rule. Section 4.1 already diagnoses the misalignment and proposes re-weighting or an author veto testable on the existing data. Reporting at least one such re-weighting experiment (or an author-veto ablation) would convert a known failure mode into a quantified improvement and strengthen the central claim without requiring new data.","section":null}],"minor_comments":[{"comment":"Figure 3 and the status counts are clear; adding the composite-score histogram (or box-plots by band) would make the monotonic separation (means 33.3 / 69.4 / 88.5) immediately visible.","section":null},{"comment":"Journal similarity is extractable for only 44/100 citations and does not separate bands; consider demoting or dropping the 15% journal weight in the main reported score, or reporting an ablated composite without it.","section":null},{"comment":"The LLM extraction stage is stochastic; a short multi-run stability table (mean ± range of the 66/28/6 split over, e.g., 5 runs) would quantify the reproducibility caveat already noted in §4.1.","section":null},{"comment":"Typographical: “Harichandaan” vs “Harichandana” in the acknowledgements/author list; “Frankenstein” references are well described but could be cross-referenced to Ansari’s compound-failure modes for consistency.","section":null},{"comment":"Related-work placement of CheckIfExist and CiteAudit is appropriate; a one-sentence explicit statement that the present study supplies the external quantitative benchmark those works lacked would clarify novelty.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is a solid, well-scoped engineering evaluation rather than a methodological breakthrough. Fit for a digital-libraries or research-integrity venue is good; the main risk is readers treating the 94% figure as a production-ready precision claim. The authors already own the limitations, so minor revision should suffice."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a clean, external recall test of their open AtomGPT citation checker on the 100 GPTZero-verified NeurIPS 2025 fabrications. It flags 94 of them (66 likely hallucinated + 28 partial), and the per-field breakdown shows why—author similarity collapses near zero while titles stay high. That is the real result.\n\nWhat is new is not the pipeline. LLM field extraction plus multi-field match against a scholarly index, with authorship as the decisive signal, is essentially the same design as CheckIfExist, which they cite. The contribution is the independent quantitative evaluation on a human-verified real-world set that already survived peer review, plus the mechanistic account (Figure 4, Table 2) and the released notebook/data. They also note the close match to Ansari’s independent manual taxonomy of the same 100 citations, which is a nice external consistency check.\n\nThey do the evaluation carefully. Positives-only design is stated up front; they report sample sizes per field (title n=100, author n=78, journal n=44), composite distributions, and the six misses with their scores. Section 4.1 already flags the soft spots that matter: no precision estimate, single live Semantic Scholar source, stochastic LLM extraction, and the title-heavy weights (60/25/15) plus missing-field substitution rule that let three of the six false negatives through. Those are real limits on production readiness, not hidden flaws that sink the recall claim.\n\nMath and data look solid for what they claim. No circularity; thresholds and weights are fixed a priori. Citations are appropriate. Self-cites are to their own AtomGPT platform and related tools, which is expected.\n\nThis is for people building or evaluating research-integrity tooling and editorial pipelines. It is not a methods breakthrough, but it is a usable open tool with a transparent recall number on a hard external set. I would send it to peer review; the limitations are already owned and the empirical core holds. Worth engaging if you care about citation verification at scale.","headline":"Solid external recall eval of an open citation checker on the NeurIPS 2025 fakes; architecture is not new, but the numbers and author-mismatch mechanism are useful and honestly scoped.","tokens_in":12321,"tokens_out":531,"would_cite":true,"duration_ms":6828,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A hybrid LLM-and-Semantic-Scholar checker flags 94 of 100 confirmed hallucinated citations from NeurIPS 2025, driven mainly by author mismatch.","keywords":["citation verification","reference hallucination","large language models","Semantic Scholar","bibliographic matching","scientific integrity","record linkage","NeurIPS"],"falsifier":"A balanced test set containing both genuine and fabricated citations that produced a high false-positive rate on real references, or repeated runs on the same 100 known fakes that yielded substantially fewer than 94 detections because of LLM extraction variability, would falsify the reliability claim.","tokens_in":12335,"feed_emoji":"🔍","tokens_out":969,"duration_ms":19596,"temperature":0.7,"pith_summary":"Fabricated citations generated by large language models are already reaching published literature, including top conferences, because peer review cannot keep up with their volume and surface plausibility. This paper presents an open web tool that parses free-form references with an LLM, retrieves the closest real papers from Semantic Scholar, and scores agreement on title, authors, and venue to grade each citation as trustworthy, partially supported, or likely fabricated. On an external set of 100 verified hallucinated citations that survived NeurIPS 2025 review, the tool flags 94 of them. Detection works because fake references typically pair a credible-looking title with an author list that matches no real paper; title similarity alone would miss most of them. The result matters because automated, interpretable verification at the speed of modern writing is now required to protect the scientific record.","feed_headline":"Tool flags 94 of 100 fake NeurIPS citations via author mismatch","feed_subtitle":"Open checker catches most machine-made references that already slipped past peer review at a top AI conference.","key_machinery":"The weighted composite score S = 0.60·title similarity + 0.25·author similarity + 0.15·journal similarity (missing fields replaced by the title score), computed against the best Semantic Scholar match after LLM field extraction and then thresholded into verified (≥80), partial (60–79), or likely hallucinated (<60). This score turns free-form citations into graded, field-interpretable judgments grounded in an external scholarly index.","core_discovery":"On the GPTZero NeurIPS 2025 set of 100 confirmed hallucinated citations, the AtomGPT reference checker classifies 66 as likely hallucinated, 28 as partial, and 6 as verified, for a 94 percent detection rate when both non-verified statuses count as flags. A per-field breakdown shows that near-zero author similarity is the dominant signal even when title similarity is high, while venue similarity contributes little; the few misses are fabrications that either recombine real fields or leave authors unparsed so a strong title alone pushes the composite over the verified threshold.","pith_inferences":["Re-weighting the composite toward authorship or adding an explicit author-veto rule would likely cut the six false negatives, since three already arise from the missing-field substitution of title for author.","A balanced real-plus-fake benchmark is the single most important next measurement; without it the high recall could mask an unacceptable false-positive rate on genuine citations.","Cross-checking against multiple scholarly indices would harden the approach against coverage gaps and rate-limit zeros in any single live source.","The same author-driven signal could be run retrospectively over large published corpora to quantify how many already-accepted papers contain undetected fabrications."],"forward_implications":["Editorial and submission systems can insert a lightweight open checker that catches the great majority of fabricated references before publication.","Author-list mismatch, not title plausibility, is the practical diagnostic signal for retrieval-grounded citation verification.","The tool’s automated status split (66 likely / 28 partial / 6 verified) closely tracks an independent manual taxonomy of the same citations, supporting the claim that most are wholesale inventions.","The six misses reveal a concrete failure mode: “Frankenstein” recombinations or unparsed author lists that let a strong title alone exceed the verified threshold."],"fun_headline_variants":["AtomGPT checker flags 94 of 100 NeurIPS hallucinations via author mismatch","Near-zero author similarity detects 94% of fake NeurIPS 2025 citations","Tool marks 94 of 100 confirmed NeurIPS fakes as untrustworthy or partial","Author field mismatch drives 94% detection of hallucinated NeurIPS references","Open checker scores 66 likely-faked plus 28 partial on 100 NeurIPS cases"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The graded judgment is reliable only if a single live scholarly index plus fixed weights that heavily favor title similarity (and substitute the title score for any missing author field) correctly separate fabrications from real papers.","fun_headline_variants_meta":{"raw":{"variants":["AtomGPT checker flags 94 of 100 NeurIPS hallucinations via author mismatch","Near-zero author similarity detects 94% of fake NeurIPS 2025 citations","Tool marks 94 of 100 confirmed NeurIPS fakes as untrustworthy or partial","Author field mismatch drives 94% detection of hallucinated NeurIPS references","Open checker scores 66 likely-faked plus 28 partial on 100 NeurIPS cases"]},"model":"grok-4.5","effort":"low","cost_usd":0.007724,"raw_usage":{"total_tokens":1876,"prompt_tokens":835,"num_sources_used":0,"completion_tokens":111,"cost_in_usd_ticks":77240000,"prompt_tokens_details":{"text_tokens":835,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":930,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":835,"tokens_out":111,"duration_ms":8593,"temperature":1.0,"reasoning_tokens":930,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T15:58:41.244459+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A balanced test set containing both genuine and fabricated citations that produced a high false-positive rate on real references, or repeated runs on the same 100 known fakes that yielded substantially fewer than 94 detections because of LLM extraction variability, would falsify the reliability claim.","supporting_citations":[],"review_version":1}