{"id":"d18dfc97-ea2e-4b88-b74c-c9c3d9cc0661","arxiv_id":"2607.18360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HALLMARK shows that citation verifiers' false-positive rate, not recall, is what determines whether their flags are mostly real catches or mostly noise at realistic hallucination rates.","lead":"Researchers built HALLMARK, a 2,526-entry benchmark that tests how well citation-verification tools catch fake or corrupted BibTeX references, and measured 13 tools on it. The main finding: a verifier's false-positive rate—how often it wrongly flags real citations—matters far more than recall in deciding whether it is usable for real audits.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The low-FPR frontier that drives the deployability claim rests on two unrecoverable FPR measurements; drift re-runs double them, so the headline precision gap may be ~2x smaller than stated.","rationale":"The reader's weakest assumption was synthetic-real transferability, which is real and explicitly acknowledged by the authors. My concern is more proximal: the central claim depends on the measured FPR ordering, and the two models anchoring the low-FPR frontier have irreproducible FPRs — the authors themselves report that a later snapshot doubles them. If the true deploy-time FPR of Opus/Sonnet is near the drifted values, the 7x precision gap becomes roughly 3x, which materially weakens the abstract's unconditional framing even though the directional conclusion may hold. This is not an internal inconsistency: the paper is unusually transparent about the drift and the missing per-entry predictions, and it open-sources code and data. But transparency does not remove the load on those two FPR cells. The concrete test would settle whether the claim survives at its stated magnitude or needs to be rephrased as regime-conditional; until then CONDITIONAL is the right verdict, matching the reader. I therefore recommend no verdict change, with the caveat that the abstract should be softened if the test shows frontier FPRs are ~2x higher.","tokens_in":57886,"tokens_out":3777,"duration_ms":33926,"concrete_test":"Take dev_public valid entries (n=513) plus a newly collected set of ~500 valid citations drawn from published reference lists (not canonical DBLP-scraped records). Run Gemini 2.5 Pro, Claude Opus 4.7, and Claude Sonnet 4.6 with pinned dated model IDs, persist per-entry predictions, and recompute FPR on both valid pools. Then recompute Tab. 21 PPV at 2% prevalence using the new FPRs (re-estimating DR on the hallucinated pool as well). If frontier FPRs exceed ~0.15 or the PPV spread between the frontier and the high-FPR verifiers falls below ~3x, the abstract's unconditional FPR-deployability claim should be softened to a regime- and measurement-conditional claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — FPR, not recall, governs deployability — is a comparative measurement claim. The 7x precision gap at a 2% base rate (Tab. 21) is driven by the low-FPR frontier: Gemini 2.5 Pro (FPR 0.050), Opus 4.7 (0.072), Sonnet 4.6 (0.127). But §7 and §E.2 state that the two Anthropic dev_public rows are summary-only, with no stored per-entry predictions, and that a later OpenRouter snapshot roughly doubles both FPRs (Opus 0.072→0.162, Sonnet 0.127→0.165), diverging by >13 pp from the published aggregates. The paper argues the ranking survives because drift shifts all models in the same direction, but the only direct evidence for rank invariance is under prompt perturbation (Spearman ρ=0.90, §H.1), not under endpoint drift for the full cohort. If deploy-time FPRs for these frontier tools are ~2x the reported values, the 'one true catch per 6–9 flags' becomes one per 12–18, and the precision gap versus aggressive verifiers shrinks from ~7x to ~3x. The qualitative ordering may survive, but the headline quantification — and the abstract's unconditional 'FPR decides deployability' — is calibrated to an operating point that is not reproducible. This is compounded by the paper's own admission (§6, §7) that synthetic-real equivalence is untested and that LLM-generated hallucinations are substantially harder than perturbations (GPT-5.1 DR 0.656 vs 0.846, Tab. 16); measured FPRs on clean DBLP-scraped valid entries may also understate FPR on messy real-world reference lists. The Bayes-rule logic is sound; the load-bearing empirical input is the FPR ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces HALLMARK, a benchmark of 2,526 BibTeX entries spanning 14 hallucination types across three difficulty tiers, with six diagnostic sub-tests per entry and a contamination-resistant split. Thirteen citation verifiers are evaluated under one protocol: a DOI-only baseline, twelve zero-shot LLMs, three agentic harnesses, and the authors' rule-based bibtex-updater. The central claim is that false-positive rate (FPR), not recall, governs deployability: at a venue-realistic ~2% base rate, Bayes' rule makes PPV depend on FPR, and the measured FPR spread (0.05–0.70) produces a reported ~7x precision gap, with low-FPR tools catching one true hallucination per 6–9 flags and aggressive high-recall tools falling below one per 35. Three failure modes are diagnosed: agentic lookups inflate FPR; base-rate precision drops; and most LLMs over-flag post-cutoff papers. The paper is unusually transparent about limitations, including the synthetic–real gap, the co-designed status of bibtex-updater, and endpoint drift on two Anthropic rows.","tokens_in":58221,"tokens_out":6152,"duration_ms":61265,"significance":"The benchmark itself is a valuable shared resource: the code and pinned dataset are released, per-entry verdicts are provided for bibtex-updater, bootstrap CIs accompany most aggregates, and the contamination canary and adaptive-data-analysis hygiene measures are thoughtful. The PPV analysis is a clean, correct application of Bayes' rule once DR and FPR are measured, and the qualitative finding that FPR ordering, not recall, dominates precision at low prevalence is useful for practitioners. However, the headline quantitative claims rest on two load-bearing measurements that are not reproducible (the Anthropic dev_public rows), and the deployment framing assumes that FPRs measured on perturbed entries transfer to real hallucinations—an assumption the paper itself states is untested. The benchmark retains substantial value, but the specific 'one-in-6-to-9' and '~7x gap' numbers should be treated as provisional.","major_comments":[{"comment":"The two Anthropic dev_public rows (Opus 4.7 FPR 0.072, Sonnet 4.6 FPR 0.127) are summary-only, with no stored per-entry predictions, and §7 reports that a later OpenRouter snapshot roughly doubles both (0.162 and 0.165). These two rows anchor the low-FPR frontier that drives the abstract's and §6's quantitative claims: Tab. 21 gives Opus PPV 17.6% and Sonnet 11.2% at a 2% base rate, leading to 'one true hallucination per 6–9 flags' and the '~7x precision gap.' Using the drift-adjusted FPRs, Opus PPV falls to about 8.7% and the gap to DeepSeek-V3.2's 2.6% shrinks to roughly 3x. The paper's decision to report the internally consistent snapshot and to read rankings rather than absolute levels is defensible for the ordinal claim, but the abstract and §6 present these absolute PPV numbers unconditionally. This is load-bearing for the central claim and needs to be fixed, either by basing the h","section":"§7, §E.2, Tab. 3, Tab. 21"},{"comment":"The deployment claim assumes that FPR/DR measured on synthetic perturbed entries transfer to real hallucinations. The paper honestly states in §7 that 'distributional and semantic equivalence remains untested' and that 'whether rankings on synthetic hallucinations predict performance on real errors remains open.' The real-world anchor is only 108 entries, 55% of which are plausible_fabrication, and the KS-based equivalence tests are underpowered. In-paper evidence cuts against transfer: GPT-5.1's DR on LLM-generated entries is 0.656 vs 0.846 on perturbations (Tab. 16), and the cross-domain probe shows bibtex-updater's FPR rising from 0.092 to 0.375. Since FPR is the deployment-decisive metric, even a modest real-world FPR shift changes the 'one-in-N flags' numbers and could reorder the precision ranking. The Walters–Wilder supplement (§C.5) is a useful partial check, but it is articles-o","section":"§6, §7, Tab. 16, §C.5"}],"minor_comments":[{"comment":"'the F1 score (F1) gap' is redundant; should be 'the F1 gap.'","section":"§5.2"},{"comment":"'The three sources play complementary roles ... then stratified by generation method lets us assess' is a sentence fragment; consider splitting into two sentences.","section":"§3.3"},{"comment":"Missing space in 'Opus 4.7dev_public'; also consider defining 'pp' at first use for readers outside the subfield.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong benchmark contribution with unusually candid limitation sections. The main issue is that the two frontier Anthropic rows are non-reproducible and the quantitative headline claims are calibrated to those rows; this is addressable by re-centering the claims on recoverable measurements or by presenting drift ranges. The synthetic–real transferability concern is also load-bearing but is at least openly acknowledged; a revision that tightens the claims and adds any additional real-world validation would make this acceptable. I would not reject, but the current abstract overstates the precision of the numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first real shared benchmark for citation verifiers, and the authors did the field a service by shipping code, data, and a 14-type taxonomy with per-entry diagnostics. The central claim — FPR, not recall, governs deployability at realistic base rates — follows from Bayes once you accept their measured DR/FPR, and they are careful to mark it regime-conditional in the body. But the headline 7x precision gap is calibrated to two Anthropic FPRs that are neither recoverable nor reproducible: the per-entry predictions were not stored, and a later snapshot roughly doubles both FPRs. The paper discloses this honestly, but it means the abstract overstates what is actually measured.\n\nWhat is new: HALLMARK. 2,526 entries, 14 types, 6 sub-tests, temporal splits, a 13-tool cohort, open data. That is real artifact value. The three failure modes are useful framing; the agentic any-no-match diagnosis is well-supported by their deterministic re-aggregation (0.73 to 0.05 FPR), and the cross-split stability of the conservative tool is a genuine finding. The authors are unusually transparent: they flag co-design, endpoint drift, contamination confounds, and the thin real-world anchor themselves, which is why the paper deserves engagement despite the soft spots.\n\nSoft spots, in order. First, the load-bearing low-FPR frontier: the two Anthropic dev rows are summary-only, no per-entry predictions, and a drift re-run doubles their FPRs. The ranking may survive — the paper argues via prompt perturbation rho=0.90 — but the \"one true catch per 6-9 flags\" number is not reproducible at the stated operating point. Second, the real-world set is 108 entries, type-skewed, and the authors admit distributional and semantic equivalence is untested; LLM-generated entries are visibly harder than perturbations, so transfer to deployment is an open question. Third, the kappa=0.24 LLM-rater proxy is not a substitute for human IAA. These are addressable; they don't invalidate the benchmark.\n\nWho it's for: anyone building or deploying citation verifiers, and publishers or preprint servers trying to choose one. It deserves a serious referee; I'd ask for persisted per-entry outputs, human IAA on a sample, more real entries, and an abstract that matches the body's conditionality. Worth citing in the next year.","headline":"A genuinely useful, openly shipped benchmark whose headline FPR-deployability quantification rests on two unrecoverable Anthropic operating points and an untested synthetic-to-real transfer.","tokens_in":58812,"tokens_out":2279,"would_cite":true,"duration_ms":22717,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that for LLM citation verifiers, false-positive rate—not recall—determines whether a tool is deployable, and backs the claim with a 2,526-entry benchmark that exposes three concrete failure modes.","keywords":["citation hallucination","LLM verifier","false positive rate","benchmark","precision at low prevalence","training cutoff","BibTeX","hallucination taxonomy"],"falsifier":"Curate a few thousand hand-verified, real hallucinated citations from diverse venues, run the same verifier cohort under the same protocol, and check at a 2% base rate whether the precision spread (roughly 1-in-6 to 1-in-35 flags) and the FPR-based ranking reproduce on this authentic corpus; if low-FPR tools no longer separate from high-recall tools, the central claim fails.","tokens_in":57678,"feed_emoji":"🔍","tokens_out":3207,"duration_ms":33282,"temperature":0.7,"pith_summary":"The paper sets out to answer which citation-verification tools can be trusted when a venue audits its submissions for hallucinated references. It builds HALLMARK, a controlled benchmark of 2,526 BibTeX entries spanning 14 hallucination types, three difficulty tiers, and six diagnostic sub-tests, then runs thirteen verifiers under one protocol. Its central claim is that the false-positive rate, not recall, decides whether a verifier is usable: at a venue-realistic ~2% hallucination rate, verifiers span an order of magnitude in FPR, producing a ~7x precision gap between the best and most aggressive tools. The paper also diagnoses three failure modes: agentic lookups buy recall but inflate false positives, precision collapses at low base rates, and most LLMs over-flag papers published past their training cutoff. If correct, the paper redirects verifier design and procurement toward FPR and calibration rather than raw detection rate.","feed_headline":"False positives, not recall, make or break citation verifiers","feed_subtitle":"At a 2% hallucination rate the best tools catch one true error per 6–9 flags; aggressive ones fall below 1 in 35.","key_machinery":"The central object is HALLMARK, a benchmark whose load-bearing design combines a 14-type hallucination taxonomy, six diagnostic binary sub-tests per entry, and a contamination-resistant held-out split, so that each verifier is scored on prevalence-independent metrics (detection rate, false-positive rate, MCC). The mechanism that carries the argument is the closed-form relationship PPV = DR·prev / (DR·prev + FPR·(1−prev)), which turns measured FPR differences into precision gaps at deployment prevalence; against that identity the paper evaluates zero-shot LLMs, agentic tool-calling harnesses, a DOI-lookup baseline, and a co-designed rule-based verifier that pairs consensus-absence flagging wi","core_discovery":"Across the benchmark, measured false-positive rates span roughly 0.05 to 0.70 across verifiers, while detection rates all sit in a 48–91% band. At a venue-realistic 2% base rate, Bayes' rule turns that FPR spread into a ~7x precision gap: the lowest-FPR tools catch one true hallucination per 6–9 flags, whereas the most aggressive high-recall models catch fewer than one per 35 flags even at ≥87% detection. The paper argues this is a deployment law, not a curiosity: for a reviewer-bound venue audit, a few points of FPR decide whether a tool's flags are mostly true catches or mostly noise. It also isolates the mechanism behind the failure modes—agentic harnesses flag on any single missing datab","pith_inferences":["If the FPR-ranking generalizes beyond synthetic entries, leaderboards that sort by recall or F1 mislead deployment decisions; the paper's logic implies that any verifier benchmark should report PPV at venue-realistic prevalence and precision-oriented metrics like FPR at 90% coverage.","The post-cutoff over-flagging result suggests a testable product implication: verifier trust expires, so tools could expose their training cutoff, route post-cutoff entries to retrieval, or watermark uncertainty—and models trained with abstention-oriented post-training may hold up better regardless of cutoff recency.","A natural extension would be a prospective live audit of a single conference's full submission pool, comparing each verifier's flags against author confirmations, to test whether the measured 1-in-6 to 1-in-35 precision spread reproduces on real traffic.","The benchmark's perturbation-heavy corpus is easier than LLM-generated hallucinations for at least one verifier (detection 0.846 vs 0.656), so building a much larger hand-validated real-world corpus is the key step to translating the paper's rankings into a deployment guarantee."],"forward_implications":["At low-prevalence venue audits, verifier selection should be driven by false-positive rate and calibration, not recall, because precision is bottlenecked by FPR.","Agentic retrieval harnesses should flag only on consensus absence across databases (plus positive metadata contradictions), not on any single missing record, to avoid a ~15x FPR inflation.","Most LLM verifiers degrade sharply on papers published after their training cutoff, so deployment on recent literature requires cutoff-aware prompting or retrieval augmentation, and the mitigation is model-dependent.","When missed fabrications are far costlier than triaging extra flags, or when downstream human review absorbs false alarms, high-recall tools plus human triage remain the right choice—the FPR rule is regime-conditional, not absolute.","A two-stage cascade (rule-based stage plus LLM diagnoser) can reach detection rates near 0.99 at an FPR near 0.11, providing a concrete high-recall, low-FPR reference design."],"fun_headline_variants":["False-positive rate, not recall, decides if citation verifiers work","Low FPR beats high recall for catching hallucinated citations","Verifier benchmark: one true catch per 6-9 flags at best","Why citation verifiers most often cry wolf: FPR bottleneck","Agentic lookups boost recall but inflate false positives"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's synthetic and perturbed hallucination entries faithfully represent real-world hallucinations, so the measured false-positive rates and rankings transfer to deployment—an assumption the paper itself says remains untested at sample sizes of only 108 real-world entries.","fun_headline_variants_meta":{"raw":{"variants":["False-positive rate, not recall, decides if citation verifiers work","Low FPR beats high recall for catching hallucinated citations","Verifier benchmark: one true catch per 6-9 flags at best","Why citation verifiers most often cry wolf: FPR bottleneck","Agentic lookups boost recall but inflate false positives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2449,"prompt_tokens":858,"completion_tokens":1591,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1503}},"tokens_in":602,"tokens_out":1591,"duration_ms":11744,"temperature":1.0,"reasoning_tokens":1503,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:35:23.662108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Curate a few thousand hand-verified, real hallucinated citations from diverse venues, run the same verifier cohort under the same protocol, and check at a 2% base rate whether the precision spread (roughly 1-in-6 to 1-in-35 flags) and the FPR-based ranking reproduce on this authentic corpus; if low-FPR tools no longer separate from high-recall tools, the central claim fails.","supporting_citations":[],"review_version":1}