Pith. sign in

REVIEW 1 major objections 5 minor 53 references

The unsupported-citation rate for agentic scientific synthesis is not a single reliable number: on identical outputs it ranges from about 3% to about 18% depending only on the verifier's strictness, and verifiers disagree on which citations

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:40 UTC pith:CJ6ZLZEJ

load-bearing objection The verifier-dependence finding is solid and worth publishing, but the conformal guard's recalibration recipe as written samples only already-flagged citations and therefore cannot catch the harder negatives that actually break exchangeability. the 1 major comments →

arxiv 2607.20527 v1 pith:CJ6ZLZEJ submitted 2026-07-10 cs.AI

Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis

Taewan Goo , Junsik Kim , Kyulhee Han , GwonYul Jo , Jong-Soo Kim , Tae-Hyung Kim This is my paper
classification cs.AI
keywords citation faithfulnessagentic AIscientific synthesisverifier reliabilityconformal predictiondistribution-free guaranteeLLM-as-judgere-attribution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish two things. First, the reported rate of unsupported citations produced by agentic scientific-synthesis systems is not a fixed property of the outputs: on identical outputs it ranges from about 3% to about 18% depending only on which verifier judges support, and verifiers agree on what is supported but disagree on what to flag (negative-specific agreement 0.27–0.30). Second, the paper shows this unreliability can be managed: a gold-anchored evaluation protocol validates the verifier against human labels, and a split-conformal layer converts an imperfect verifier into a finite-sample, distribution-free guarantee on how many truly unsupported citations slip past a chosen flagging rule. The guarantee is conditional on calibration negatives resembling deployment negatives, and the paper quantifies how hard that condition is to meet.

Core claim

The central discovery is that automated citation-faithfulness checks are themselves unreliable, and that the unreliability is measurable and bounded. Across five gold-validated verifiers applied to identical agent outputs, the unsupported-citation rate varies threefold to sixfold (about 3% to 18%), and pairwise agreement on which citations to flag is only 0.27–0.30, despite agreement on supported citations of 0.90–0.96. The paper then shows that anchoring the verifier on human gold and wrapping it in a split-conformal calibration step yields a flagging rule with a finite-sample guarantee: for a chosen tolerance alpha, the rule catches at least 1−alpha of truly unsupported citations, provided

What carries the argument

The central object is the split-conformal flagging rule. On a held-out calibration set of gold-labeled unsupported citations, the threshold is set to the quantile s_(k) with k = ceil((n_cal + 1)(1 − alpha)), so that any new unsupported score is at or below the threshold with probability at least 1−alpha under exchangeability. This converts an imperfect verifier's score into a distribution-free guarantee on the catch rate of truly unsupported citations, rather than on conclusion correctness. The verifier itself is treated as a swappable measurement instrument, validated against human gold and chosen on cost; re-attribution is a separate commodity slot.

Load-bearing premise

The conformal guarantee holds only if the calibration negatives are exchangeable with the deployment negatives; the paper shows that when calibration negatives are easy in-paper distractors and deployment negatives are harder on-topic mis-citations, the promised catch rate drops from 90% to 37%.

What would settle it

Run a fixed set of agent outputs through two gold-validated verifiers with matched protocols: if the measured unsupported-citation rates move together within a few points across a range of strictness, the paper's claim of verifier-dependence would be contradicted. Alternatively, calibrate the conformal threshold on easy distractors and apply it to hard negatives: if the achieved catch rate meets the declared target despite the score distribution shift, the calibration-negative-difficulty result would be contradicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Citation-faithfulness numbers are not comparable across papers unless the verifier and protocol are named and gold-validated.
  • A deployment can declare a guaranteed catch rate on unsupported citations at a predictable review budget; on the audited systems, a 90% guarantee costs single-digit flag rates.
  • Recovering a correct supporting passage is a cheap, swappable slot; a deterministic lexical ranker matches or approaches the best generator except on low-overlap entailment cases.
  • The conformal bound holds in finite samples even when gold is scarce, and the finite-sample correction matters at calibration sizes below about 200.
  • The guarantee does not transfer when calibration negatives are easier than deployment negatives; recalibrating on target-domain negatives restores it, at a higher review cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The verifier-dependence finding likely extends to any LLM-as-judge metric in agentic evaluation, not just citation support, so any single-number judge report should be treated as relative to that judge.
  • The conformal guard could be lifted from passage-level support to a claim-level body-of-evidence guarantee, but that would require a model of how multiple passages jointly justify a claim.
  • The recalibration recipe could be automated with active learning, so the roughly 200 labeled flagged citations are selected to minimize calibration-negative difficulty.
  • Reporting verifier score distributions alongside verdicts would make transfer deficits visible and let downstream users re-calibrate without re-labeling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper studies the reliability of citation-faithfulness verification in agentic scientific synthesis. It reports that on identical agent outputs, the measured unsupported-citation rate varies from roughly 3% to 18% across five gold-validated verifiers, and that verifiers agree strongly on which citations are supported (positive-specific agreement 0.90–0.96) but disagree on which to flag (negative-specific agreement 0.27–0.30). The paper then proposes a gold-anchored evaluation protocol and a deployable guard: a split-conformal layer that, under exchangeability between calibration and deployment negatives, gives a distribution-free finite-sample bound on the catch rate of unsupported citations. The method is validated on SciFact, QASA, and PubMedQA with four open 27–35B models across three pipelines, and the paper identifies 'calibration-negative difficulty' as the key condition governing transfer, with a recalibration recipe for deployment.

Significance. If the findings hold, the verifier-dependence result is an important caution for the agentic-citation benchmarking literature: unsupported-citation rates are meaningful only relative to a named, gold-validated verifier and protocol. The conformal guard is a clean, correct application of split conformal prediction to passage-level citation support, and the paper is commendably explicit about the in-distribution nature of its coverage demonstration and about the large transfer gap caused by easy calibration negatives. The reproducibility kit with pinned model snapshots, frozen score files, and confidence intervals on headline numbers is a strength. However, the deployment recalibration recipe, which is load-bearing for the guard claim, contains a statistical flaw: it samples only flagged citations and therefore cannot recover the hard negatives that actually caused the transfer failure.

major comments (1)
  1. [Supplementary S6 (Recalibration recipe)] The recalibration recipe instructs the user to 'label ≳200 flagged citations from the target system' and then run the calibration entry point on these target negatives. But 'flagged' means score ≤ τ_current (e.g., 0.42 for the 0.90 target). The hard negatives responsible for the transfer failure have scores above this threshold (mean 0.50 on SciFact unsupported claims), so they are excluded from the labeled sample. Any new τ estimated from a sample truncated at the old τ is necessarily ≤ old τ, since it is a quantile of scores all ≤ old τ; Table S7 itself shows that restoring 0.90 catch on the hard negatives requires τ = 0.83, which is above the old threshold. Thus the recipe as written cannot restore coverage for the very distribution shift it is meant to address. A valid recalibration requires a score-unbiased sample of target citations (e.g., a random subsample, labeled for support) o
minor comments (5)
  1. [§5.5 and Supplementary S4] The text says 'Calibrated on QASA gold (the full test set, 1375 supported and 1375 unsupported pairs)' while other places refer to 'QASA gold n = 1375'. It should be clarified whether n = 1375 is the number of questions, with 1375 supported and 1375 unsupported pairs, or the total number of claim–passage pairs. The current phrasing is internally confusing and affects the reported calibration size.
  2. [§5.2 and Table 2 / S1] Table 2 reports AttrScore-3B recall 0.90, while S1 states that 'main-text numbers for the adopted verifier use the held-out values' (recall 0.942). The main text says '0.90 on SciFact, Table 2; 0.94 on a disjoint held-out split'. Please make explicit which number is the development-set value and which is the held-out value, and ensure Table 2 is labeled accordingly.
  3. [§5.4] The negative-specific agreement figures (0.27–0.30) are computed at each verifier's native operating threshold. The claim that 'no single flag set can be trusted' is therefore specifically about uncalibrated native thresholds; a threshold-matched or gold-calibrated comparison would strengthen the claim. As written, the reader could infer that disagreement is irreducible, which the paper does not establish.
  4. [Abstract and §5.3] The abstract states 'about 3% to about 18%' but the text and Figure 3 describe roughly 3% to 14–18%. Please report the exact range with confidence intervals for the headline rate and note the pooled nature of the estimate.
  5. [§5.5] The sentence 'The guarantee pays for safety in review effort, the genuine citations it also flags' is grammatically awkward; rephrase to clarify that the flag rate includes genuinely supported citations that are nevertheless flagged for review.

Circularity Check

0 steps flagged

No circularity: central claims rest on external gold, a standard conformal proof, and honest transfer-failure measurements; the flagged-only recalibration concern is a statistical correctness issue, not a circular reduction.

full rationale

No circularity found. The paper's two load-bearing claims are (1) an empirical measurement that unsupported-citation rates and flag sets vary across verifiers on identical agent outputs, and (2) a split-conformal finite-sample catch-rate guarantee. Claim (1) is not derived from its definitions: Definition 3.2 says the rate is verifier-relative, but the observed 3–18% spread and negative-specific agreement 0.27–0.30 are measured on fixed outputs against external gold (SciFact, QASA) and external verifiers, not manufactured by a fitted parameter. Claim (2) rests on Proposition 4.1, which is the standard split-conformal exchangeability argument, and the paper calibrates on human-labeled unsupported-class scores and verifies the achieved catch rate on held-out gold; the theorem does not assume the conclusion it proves. The calibration-negative-difficulty result is an honest empirical failure analysis: the paper reports that a QASA-calibrated threshold catches 90% of easy distractors but only 37% of harder human-labeled negatives, and it explicitly states that the QASA coverage is an in-distribution demonstration rather than a transferable operating point. This is the opposite of relabeling a fit as a prediction. The recalibration recipe's use of flagged-only citations may be statistically problematic as a deployed fix, but that is a correctness/design concern, not a circular reduction: no equation is reused as its own input and no fitted quantity is renamed as an independent result. References such as AttrScore, QASA, OpenScholar/PaperQA2, and the conformal literature are external and involve no load-bearing self-citation by the present authors. Score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central methodological contribution is a gold-anchored protocol with a conformal guarantee; it rests on exchangeability and the validity of human gold labels. There are no ad hoc free parameters or invented entities: the verifier is a preexisting model adopted on cost, and the conformal threshold is calibrated on external gold, not fitted to produce a desired headline result.

axioms (5)
  • domain assumption Calibration and deployment unsupported-class scores are exchangeable
    Required by Proposition 4.1. The paper quantifies its failure in S6 (90% target → 37% achieved on harder human-labeled negatives) and provides a recalibration recipe on target-domain negatives.
  • domain assumption Human gold labels are the ground truth for citation support
    Definition 3.1; all validation and calibration anchor on human labels from SciFact, QASA, and PubMedQA.
  • domain assumption Support is judged over the full cited passage via sliding 600-char windows at stride 300
    Protocol choice in S5; the paper notes that truncating to a prefix inflates unsupported rates severalfold (Limitations), so the definition of 'support' is load-bearing.
  • domain assumption AttrScore-3B's continuous score separates supported from unsupported claims well enough for conformal calibration
    §4.1 and Figure 2; verifier AUC ~0.88 on SciFact n=208. If separation were poor, the conformal bound would be vacuous in practice (very high flag rate).
  • domain assumption The gold benchmarks' negative samples (QASA distractors) can proxy for deployment negatives after recalibration
    S6; the paper argues real deployment negatives lie between easy QASA distractors and hard SciFact refuted pairs, and that recalibrating on ~200 target-domain labels restores the guarantee.

pith-pipeline@v1.3.0-alltime-deepseek · 18278 in / 9670 out tokens · 99045 ms · 2026-08-02T07:40:05.700176+00:00 · methodology

0 comments
read the original abstract

Agentic LLM systems such as OpenScholar and PaperQA2 read the scientific literature and return cited answers, and both they and their benchmarks already check whether those citations hold, with a fixed attribution model or human graders. Neither audits the reliability of that check itself. We show it is not reliable, and that this matters. On identical agent outputs the measured unsupported-citation rate ranges from about 3% to about 18% depending only on the verifier's strictness, and although verifiers agree on which citations are supported, they disagree on which to flag (negative-specific agreement 0.27 to 0.30), so no single flag set is trustworthy and cross-paper comparison is invalid without a named verifier and protocol. We present a gold-anchored evaluation protocol and a deployable guard that make this behavior measurable and bounded. The protocol validates the verifier, measures re-attribution, and calibrates a guarantee against human gold rather than another model's verdict; the verifier is a swappable instrument chosen on cost (recall 0.94 on the supported class, held out), and re-attribution is a commodity step where a deterministic BM25 matches the best open generator. The guard adds a split-conformal layer placing a distribution-free, finite-sample bound on truly unsupported citations that slip past a chosen flagging rule, a guarantee on catch rate rather than conclusion correctness. The bound holds on held-out gold, and we identify and quantify the condition governing its transfer to deployment, calibration-negative difficulty, with a concrete recalibration recipe, left untested by prior conformal-factuality work. Validated across four open 27-35B models and three agentic pipelines on public benchmarks (SciFact, QASA, PubMedQA), with confidence intervals on every headline number, the protocol and guard ship as an open single-GPU kit.

Figures

Figures reproduced from arXiv: 2607.20527 by GwonYul Jo, Jong-Soo Kim, Junsik Kim, Kyulhee Han, Tae-Hyung Kim, Taewan Goo.

Figure 1
Figure 1. Figure 1: Study design. The agentic systems under audit (top) produce cited answers; four evaluation stages then anchor a verifier [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The verifier as a swappable instrument, validated on gold. (A) Matched catch-rate frontier on SciFact: at any common catch [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The unsupported-citation rate has no single value: on the same agent outputs it ranges threefold to sixfold across five [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The conformal guarantee, calibrated on QASA gold ( [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Re-attribution validated against gold (QASA, [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 3 canonical work pages

  1. [1]

    Yasin Abbasi-Yadkori, Ilja Kuzborskij, David Stutz, et al. 2024. Mitigating LLM Hallucinations via Conformal Abstention. arXiv:2405.01563

    Pith/arXiv arXiv 2024

  2. [2]

    Alibaba Qwen Team. 2026. Qwen3.6 Model Family. qwen.ai/blog

    2026

  3. [3]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. 2021. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511 Found. Trends Mach. Learn., 2023

    Pith/arXiv arXiv 2021

  4. [4]

    Akari Asai, Jacqueline He, Rulin Shao, et al . 2026. Synthesizing scientific literature with retrieval-augmented language models.Nature(2026). doi:10.1038/s41586-025-10072-4 OpenScholar; preprint arXiv:2411.14199

    Pith/arXiv arXiv 2026

  5. [5]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. InProceedings of ICLR. arXiv:2310.11511

    Pith/arXiv arXiv 2024

  6. [6]

    Nicola Bena, Marco Anisetti, Ernesto Damiani, Alex Della Bruna, Chan Yeob Yeun, and Claudio A. Ardagna. 2026. A Certification Scheme for Large Language Models-Based Applications.ACM Transactions on Intelligent Systems and Technology(2026). doi:10.1145/3819077

    work page doi:10.1145/3819077 2026

  7. [7]

    Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes

    Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. Autonomous chemical research with large language models.Nature624 (2023). doi:10.1038/s41586-023-06792-0

    doi:10.1038/s41586-023-06792-0 2023

  8. [8]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models.ACM Transactions on Intelligent Systems and Technology15, 3 (2024). doi:10.1145/3641289

    doi:10.1145/3641289 2024

  9. [9]

    Mikaël Chelli, Jules Descamps, Vincent Lavoué, et al . 2024. Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews.Journal of Medical Internet Research26 (2024). doi:10.2196/53164

    doi:10.2196/53164 2024

  10. [10]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475 COLM 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Ran El-Yaniv and Yair Wiener. 2010. On the Foundations of Noise-free Selective Classification.Journal of Machine Learning Research11 (2010)

    2010

  12. [12]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of EACL: System Demonstrations. doi:10.18653/v1/2024.eacl-demo.16

    doi:10.18653/v1/2024.eacl-demo.16 2024

  13. [13]

    Shanghua Gao, Ada Fang, Yepeng Huang, et al. 2024. Empowering biomedical discovery with AI agents.Cell187 (2024). doi:10.1016/j.cell.2024.09.022

    doi:10.1016/j.cell.2024.09.022 2024

  14. [14]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. InProceedings of EMNLP. doi:10.18653/v1/2023.emnlp-main.398

    doi:10.18653/v1/2023.emnlp-main.398 2023

  15. [15]

    Yonatan Geifman and Ran El-Yaniv. 2017. Selective Classification for Deep Neural Networks. InAdvances in Neural Information Processing Systems (NeurIPS). arXiv:1705.08500

    Pith/arXiv arXiv 2017

  16. [16]

    Ali Essam Ghareeb, Benjamin Chang, Ludovico Mitchener, et al. 2026. A multi-agent system for automating scientific discovery.Nature(2026). doi:10.1038/s41586-026-10652-y Robin; preprint arXiv:2505.13400 (2025)

    Pith/arXiv arXiv 2026

  17. [17]

    Google DeepMind. 2026. Gemma 4: Frontier Multimodal Intelligence On-Device. deepmind.google/models/gemma/gemma-4

    2026

  18. [18]

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, et al. 2026. Accelerating scientific discovery with Co-Scientist.Nature(2026). doi:10.1038/s41586- 026-10644-y Preprint arXiv:2502.18864 (2025)

    Pith/arXiv arXiv 2026

  19. [19]

    Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian-Guang Lou, and Haoyi Xiong. 2026. Evaluating LLM-based Agents for Multi-turn Conversations: A Survey.ACM Transactions on Intelligent Systems and Technology17, 4 (2026). doi:10.1145/3793671

    work page doi:10.1145/3793671 2026

  20. [20]

    Zhicheng Guo et al. 2024. StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning.ACL Findings(2024). arXiv:2403.07714

    Pith/arXiv arXiv 2024

  21. [21]

    Rajarshi Haldar and Julia Hockenmaier. 2025. Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks. InFindings of the Association for Computational Linguistics: EMNLP 2025. doi:10.18653/v1/2025.findings-emnlp.1361

    doi:10.18653/v1/2025.findings-emnlp.1361 2025

  22. [22]

    Or Honovich, Roee Aharoni, Jonathan Herzig, et al . 2022. TRUE: Re-evaluating Factual Consistency Evaluation. InProceedings of NAACL. doi:10.18653/v1/2022.naacl-main.287

    doi:10.18653/v1/2022.naacl-main.287 2022

  23. [23]

    Jimenez, John Yang, Alexander Wettig, et al

    Carlos E. Jimenez, John Yang, Alexander Wettig, et al. 2024. SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?. InProceedings of ICLR

    2024

  24. [24]

    Seungone Kim, Juyoung Suk, Shayne Longpre, et al. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. InProceedings of EMNLP. doi:10.18653/v1/2024.emnlp-main.248

    doi:10.18653/v1/2024.emnlp-main.248 2024

  25. [25]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, et al. 2024. RewardBench: Evaluating Reward Models for Language Modeling.arXiv:2403.13787 (2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Junhyeok Lee. 2026. citecheck: An MCP Server for Automated Bibliographic Verification and Repair in Scholarly Writing. arXiv:2603.17339 Manuscript submitted to ACM Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis 15

    arXiv 2026

  27. [27]

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. 2023. QASA: Advanced Question Answering on Scientific Articles. InInternational Conference on Machine Learning (ICML) (PMLR, Vol. 202)

    2023

  28. [28]

    Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. 2024. AttributionBench: How Hard is Automatic Attribution Evaluation?. InFindings of the Association for Computational Linguistics: ACL 2024. doi:10.18653/v1/2024.findings-acl.886

    work page doi:10.18653/v1/2024.findings-acl.886 2024

  29. [29]

    Xiao Liu, Hao Yu, Hanchen Zhang, et al. 2024. AgentBench: Evaluating LLMs as Agents. InProceedings of ICLR. arXiv:2308.03688

    Pith/arXiv arXiv 2024

  30. [30]

    Bran, Sam Cox, Oliver Schilter, et al

    Andres M. Bran, Sam Cox, Oliver Schilter, et al. 2024. Augmenting large language models with chemistry tools.Nature Machine Intelligence6 (2024). doi:10.1038/s42256-024-00832-8

    doi:10.1038/s42256-024-00832-8 2024

  31. [31]

    Chang Ma, Junlei Zhang, et al. 2024. AgentBoard: An Analytical Evaluation Board of Multi-Turn LLM Agents. InAdvances in Neural Information Processing Systems (NeurIPS)

    2024

  32. [32]

    David Manheim and Scott Garrabrant. 2018. Categorizing Variants of Goodhart’s Law. arXiv:1803.04585

    Pith/arXiv arXiv 2018

  33. [34]

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, et al. 2024. GAIA: A Benchmark for General AI Assistants. InProceedings of ICLR. arXiv:2311.12983

    Pith/arXiv arXiv 2024

  34. [35]

    Christopher Mohri and Tatsunori Hashimoto. 2024. Language Models with Conformal Factuality Guarantees. InProceedings of ICML. arXiv:2402.10978

    Pith/arXiv arXiv 2024

  35. [36]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. InAdvances in Neural Information Processing Systems (NeurIPS). arXiv:2404.13076

    Pith/arXiv arXiv 2024

  36. [37]

    Patil et al

    Shishir G. Patil et al. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation. InProceedings of ICML

    2025

  37. [38]

    Victor Quach, Adam Fisch, Tal Schuster, et al. 2024. Conformal Language Modeling. InInternational Conference on Learning Representations (ICLR). arXiv:2306.10193

    Pith/arXiv arXiv 2024

  38. [39]

    Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, et al. 2023. Measuring Attribution in Natural Language Generation Models.Computational Linguistics49, 4 (2023). doi:10.1162/coli_a_00486

    doi:10.1162/coli_a_00486 2023

  39. [40]

    Chawla, and Yanfang Ye

    Kaize Shi, Wei Sun, Zhao Zhang, Li Sun, Nitesh V. Chawla, and Yanfang Ye. 2026. CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era. arXiv:2602.23452

    Pith/arXiv arXiv 2026

  40. [41]

    Noah Shinn, Federico Cassano, Edward Berman, et al. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. InAdvances in Neural Information Processing Systems (NeurIPS)

    2023

  41. [42]

    Skarlinski, Sam Cox, Jon M

    Michael D. Skarlinski, Sam Cox, Jon M. Laurent, et al. 2024. Language agents achieve superhuman synthesis of scientific knowledge. arXiv:2409.13740 PaperQA2

    Pith/arXiv arXiv 2024

  42. [43]

    Bulaong, et al

    Kyle Swanson, Wesley Wu, Nash L. Bulaong, et al . 2025. The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies.Nature(2025). doi:10.1038/s41586-025-09442-9

    doi:10.1038/s41586-025-09442-9 2025

  43. [44]

    Sijun Tan et al. 2025. JudgeBench: A Benchmark for Evaluating LLM-Based Judges. InProceedings of ICLR. arXiv:2410.12784

    Pith/arXiv arXiv 2025

  44. [45]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, et al. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models.arXiv:2404.18796(2024)

    Pith/arXiv arXiv 2024

  45. [46]

    Jonas Wallat, Maria Heuss, Maarten de Rijke, and Avishek Anand. 2024. Correctness is not Faithfulness in RAG Attributions. arXiv:2412.18004

    Pith/arXiv arXiv 2024

  46. [47]

    Hanchen Wang, Tianfan Fu, Yuanqi Du, et al. 2023. Scientific discovery in the age of artificial intelligence.Nature620 (2023). doi:10.1038/s41586- 023-06221-2

    doi:10.1038/s41586- 2023

  47. [48]

    Kevin Wu, Eric Wu, Kevin Wei, et al . 2025. An automated framework for assessing how well LLMs cite relevant medical references.Nature Communications16 (2025). doi:10.1038/s41467-025-58551-6 SourceCheckup

    doi:10.1038/s41467-025-58551-6 2025

  48. [49]

    Yumo Xu, Peng Qi, Jifan Chen, et al. 2025. CiteEval: Principle-Driven Citation Evaluation for Source Attribution. InProceedings of the 63rd Annual Meeting of the ACL. doi:10.18653/v1/2025.acl-long.1574

    doi:10.18653/v1/2025.acl-long.1574 2025

  49. [50]

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective Retrieval Augmented Generation. InarXiv:2401.15884

    Pith/arXiv arXiv 2024

  50. [51]

    2024.𝜏-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.arXiv:2406.12045(2024)

    Shunyu Yao et al. 2024.𝜏-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.arXiv:2406.12045(2024)

    Pith/arXiv arXiv 2024

  51. [52]

    Xiang Yue, Boshi Wang, Ziru Chen, et al. 2023. Automatic Evaluation of Attribution by Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2023. doi:10.18653/v1/2023.findings-emnlp.307 AttrScore

    doi:10.18653/v1/2023.findings-emnlp.307 2023

  52. [53]

    Wei Zhang, Guojun Dai, Ding Luo, Yan Wang, and Chen Ye. 2026. From Hallucination to Certainty: Meta-Knowledge Guided Self-Correcting Large Language Models.ACM Transactions on Intelligent Systems and Technology(2026). doi:10.1145/3797906

    doi:10.1145/3797906 2026

  53. [54]

    OURS”) and a canonical alternative (“CANON

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. InNeurIPS Datasets and Benchmarks Track. arXiv:2306.05685. Manuscript submitted to ACM 16 Goo et al. Supplementary Material This supplement holds the detailed statistics and the reproducibility specification kept out of the main text. Section nu...

    Pith/arXiv arXiv 2023