Pith. sign in

REVIEW 4 major objections 6 minor 46 references

A reference-free NLI-hypergraph audit gives more reliable step-level evaluation of open-ended reasoning than direct LLM judges, especially in clinical settings.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:00 UTC pith:II6GIH7K

load-bearing objection Novel structural audit and a genuinely useful benchmark, but the 'more reliable evaluation' claim outruns the evidence because structural grounding is scored against clinical-correctness labels. the 4 major comments →

arxiv 2607.19678 v1 pith:II6GIH7K submitted 2026-07-22 cs.CL cs.AIcs.LG

Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

Guneet Singh Kohli , Yuxiang Zhou , Michael Sejr Schlichtkrull , Gregory E Dean , Maria Liakata This is my paper
classification cs.CL cs.AIcs.LG
keywords reference-free evaluationLLM-as-judgereasoning auditnatural language inferencehypergraphAND-OR searchopen-ended QAclinical reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that evaluating open-ended LLM answers by final-answer matching or by having an LLM judge the whole response misses how reasoning steps support one another. It proposes decomposing a generated reasoning trace into segments, labelling each premise–target relation with natural language inference (entailment, implied, neutral, contradiction), and organizing these relations into a hypergraph. A deterministic backward AND–OR search then marks each segment as supported, unsupported, or orphaned. Across a deductive math verification benchmark and a new physician-annotated medical benchmark, the audit yields more balanced segment-level verdicts than direct LLM judges, which tend to over-accept fluent clinical reasoning. If right, the framework offers a reference-free, inspectable way to flag where an answer's reasoning is unanchored or built on unsupported claims.

Core claim

The paper's central discovery is that local NLI judgements, composed through a hypergraph with backward AND–OR search, produce a structural audit of a reasoning trace that is more reliable than direct LLM-as-judge verdicts in open-ended settings, and that the advantage is largest precisely where direct judges are most brittle. In the medical benchmark, direct judges show high sensitivity but near-chance specificity, accepting most problematic segments; the audit shifts the balance without needing any gold reference answer. The recovered support structure also reveals a domain-level signature: mathematical reasoning typically fails as broken chains dominated by unsupported segments, while ope

What carries the argument

The central object is an NLI-labelled hypergraph constructed from a reasoning trace. Segments become nodes; a directed hyperedge connects a candidate set of earlier premises to a later target segment, labelled by an NLI model as entailment, implied, neutral, or contradiction. Entailment and implied edges are kept as candidate support, neutral is discarded, and contradiction is held as negative evidence. A deterministic backward AND–OR search then treats each segment in turn as a sink: from the segment node there is an OR choice among candidate supporting premise-set hypernodes, and from each chosen hypernode there is an AND requirement that every premise in the set be itself grounded. If a c

Load-bearing premise

The load-bearing premise is that a segment's trustworthiness in open-ended reasoning is captured by whether it is structurally grounded in the preceding trace through NLI-labelled premise–target relations — that decomposing a reasoning trace into local inference edges and composing them with AND–OR reachability aligns with what physicians mean by a problematic reasoning step; the paper's own scope statement says the labels do not claim external factual correctness, and the qu

What would settle it

A direct test: on a larger, more diverse physician-annotated corpus, compute whether segments labelled SUPPORTED are accepted by physicians at a rate meaningfully above the base rate of acceptable segments, and whether ORPHAN/UNSUPPORTED segments are flagged at a rate meaningfully above chance. If the audit's labels do not separate physician trust labels beyond the base rate, or if a carefully calibrated LLM judge with a structured rubric matches or exceeds the audit's balanced F1 on the same traces, the central advantage claim would be settled against. More narrowly, a single trace in which a

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Step-level evaluation for open-ended QA should be structural: it should ask how inferential relations compose across a trace, not only whether the final answer appears plausible.
  • Strong LLM judges that look accurate on deductive math can be severely over-accepting in abductive, open-ended domains; the paper documents specificity dropping as low as 0.04–0.41 on the medical benchmark.
  • The audit-label distribution gives a diagnostic signature for a domain: orphan-dominated failures suggest unanchored claims, while unsupported-dominated failures suggest broken logic chains.
  • Physician-labelled partial truths and hedged statements should be flagged rather than endorsed; the conservative mapping used to build the benchmark embodies this principle.
  • The framework is reference-free and domain-independent in principle, requiring only a local NLI labeller plus the deterministic graph search, so it can be dropped into other high-stakes QA settings without gold answers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the audit is purely structural and the paper explicitly disclaims external factual correctness, its natural use is as a triage layer: flag segments that need human review rather than as a verdict on clinical truth. A testable extension would combine structural support with retrieved external evidence.
  • The orphan versus unsupported distinction may generalize to other high-stakes domains such as law, finance, or policy, where fluent but unanchored claims are the typical failure mode; that is a prediction the paper does not yet test.
  • The paper leaves open whether the audit labels can serve as training signal for a smaller, specialized NLI model; if that works, the framework would no longer depend on a general-purpose LLM for local labelling, addressing the paper's stated main limitation.
  • One could use the audit as a pre-filter before LLM judging, sending only orphan and unsupported segments to the judge; this might mitigate LLM over-acceptance without relying solely on NLI labelling quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a reference-free framework for auditing LLM-generated reasoning traces. It decomposes a trace into segments, uses an NLI model to label local premise–target relations, organizes them into a hypergraph, and applies a deterministic backward AND–OR search to assign SUPPORTED/UNSUPPORTED/ORPHAN labels. The framework is evaluated on a sampled subset of Hard2Verify and on UROREASON, a new physician-annotated clinical reasoning benchmark (40 cases, 305 segments). The authors report that direct LLM-as-judge baselines over-accept clinical segments (low TNR), whereas the NLI-hypergraph audit yields higher balanced F1 on UROREASON and, for some open-weight models, on Hard2Verify. They conclude that the audit provides a more reliable reference-free evaluation signal and that QA evaluation should exploit compositional inferential structure.

Significance. If the central claim is established, the paper makes a useful contribution: a deterministic, reproducible, reference-free structural audit that exposes where reasoning traces lack grounding, and a new clinical benchmark for step-level evaluation. The strengths are real: the graph algorithm is clearly specified and deterministic; the NLI prompt is held fixed across domains; the failure-mode analysis (orphan-dominant medical reasoning vs. chain-dominant mathematical reasoning) is interesting; and both code and benchmark are promised publicly. The paper also identifies a plausible and practically important failure mode of LLM judges, especially in open-ended medical QA. However, the headline claim of a 'more reliable evaluation signal' is currently supported only through a comparison against physician labels that encode clinical trustworthiness, while the audit explicitly disclaims factual or clinical correctness. That mismatch is load-bearing and needs to be addressed before the central claim can be accepted.

major comments (4)
  1. [§3.4–3.5, §4, Table 4] Construct-validity mismatch: §3.5 states the audit labels 'do not claim external factual correctness', and §5.2 calls them 'structural diagnostics, not factual truth labels'. Yet the evaluation in §4 treats physician labels of clinical trustworthiness (correct/incorrect/uncertain mapped to binary y) as the gold standard and reports balanced F1 as evidence of 'more reliable' evaluation. Table 4 shows the consequence: S3 is physician-labelled uncertain/problematic (gold 0) but is audit-SUPPORTED; S5 and S9 are physician-labelled correct (gold 1) but are audit-ORPHAN. These are direct consequences of the intended construct, not noise. Therefore the reported F1 gains may reflect a proxy correlation rather than a genuine ability to identify clinically trustworthy vs. problematic segments. The authors should either validate the audit against structural annotations (e.g., physician judgements o
  2. [§3.2, Appendix A.1, Table 6] Hyperparameters such as τ=3, premise size ≤2, the support-label ordering, and the pruning rules ('prefer simpler or stronger candidates') materially shape the results. The main text says these are 'the same τ and rules across both domains', but the appendix does not report a quantitative ablation of τ or premise-size bounds. If these values were chosen after inspecting UROREASON results, the comparison to untuned LLM-as-judge baselines is not apples-to-apples and the 'reference-free' framing is weakened. The authors should provide a sensitivity analysis (e.g., τ ∈ {1,2,3,5}, premise size ∈ {1,2,3}) and state explicitly how the default configuration was selected, including whether any UROREASON labels influenced the choice.
  3. [§4, Table 1, §E] Statistical support is missing: UROREASON has only 305 segments clustered in 40 cases, and segments within a case are not independent. The reported differences in balanced F1 (e.g., 0.577 vs. 0.568 for GPT-5.4-mini on the medical set, and 0.549–0.577 across open-weight models) are presented without confidence intervals, case-level bootstrap, or significance tests. Given the small number of cases and the clustered structure, it is plausible that the differences are within sampling noise. The authors should report case-level CIs or a paired analysis; otherwise the 'more reliable' claim is overstated.
  4. [§5.1, Table 1(a)] The headline metric, balanced F1, treats sensitivity and specificity symmetrically. On UROREASON the audit's TPR is consistently lower (≈0.52) than the direct judges' TPR (≈0.82–0.99), while its TNR is higher. The choice of symmetric weighting is not justified for clinical use, where over-accepting a problematic recommendation may be more costly than under-accepting a correct one. The paper should either justify the metric in terms of the intended audit use-case or report a utility/decision-theoretic comparison. This is not a request for a new metric per se, but for a demonstration that the reported F1 gains correspond to a practically preferable operating point.
minor comments (6)
  1. [Abstract / §5] The abstract's 'more reliable reference-free evaluation signal' is stronger than what the evidence supports, given the construct-validity limitations. Suggest qualifying this as 'more stable structural grounding signal'.
  2. [Appendix B / D] The prompt templates are described as being in 'Appendix X' in the main text; no appendix labelled X exists. The appendix letters should be consistent.
  3. [Appendix D] The headings 'D.2 Recovered Support Trees' and 'D.3 Recovered Support Trees' are duplicated; the latter should be renumbered.
  4. [§3.1] The notation m_min and m_max for premise-set size is introduced but not defined numerically in the main text; the values appear later in Table 6. Please define them in §3.1.
  5. [§4] The random sample of 100 Hard2Verify questions is described only as 'randomly sampled'. Please report the sampling procedure and whether the sample is stratified by difficulty or topic.
  6. [Appendix E.1 / E.2] The maximum context length for API models is set to 4096, while open-weight models use 8192. This difference may affect long-trace evaluation and should be discussed as a limitation or controlled for.

Circularity Check

0 steps flagged

No significant circularity: audit labels are computed from NLI relations and deterministic search, independently of the physician gold labels, so the reported comparison is empirical rather than definitional.

full rationale

The claimed derivation chain is not circular. The audit labels are produced from local NLI premise–target labels and a deterministic backward AND–OR search (§3.1–§3.4); the physician gold labels in UROREASON are external and never enter the hypergraph construction or label assignment. No parameter is fitted to the gold labels: the NLI prompt is domain-general and held fixed across both domains (§4), and settings such as τ=3 and premise-size bounds are reported as fixed implementation parameters rather than tuned on UROREASON. The central comparison — balanced F1 against physician annotations — is an empirical benchmark, not a reduction. The paper even reports divergences (Table 4: S3 gold=0 but SUPPORTED, S5 and S9 gold=1 but ORPHAN), which shows the audit output is not forced to match the gold labels. The explicit disclaimers that audit labels are structural and 'do not claim external factual correctness' (§3.4, §5.2, Table 4 caption) further confirm the evaluation is not definitionally aligned with the clinical gold standard. The acknowledged reliance on LLMs for local NLI labels is a dependency, not a circular equivalence, and the paper states this limitation openly. The strongest caveat is a construct-validity question — whether structural grounding is the right proxy for clinical trustworthiness — but that is a correctness/validity risk, not a circularity in the derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The method introduces no new physical or conceptual entities beyond the SUPPORTED/UNSUPPORTED/ORPHAN audit labels and the UroReason benchmark. Its free parameters are implementation thresholds rather than fitted physical constants, but tau and the premise-size cap shape all reported numbers and are not ablated in the paper.

free parameters (2)
  • closest-gap threshold tau = 3
    Locality filter for IMPLIED edges: IMPLIED edges whose closest premise is beyond tau are removed. Stated as an implementation parameter (§3.2, Appendix A.1, Table 6) with an ablation promised but not shown.
  • premise set size bounds = 1–2
    Only singleton and two-premise support sets are considered when constructing hyperedges, so the 'hypergraph' is effectively pairwise. This limits the joint-contribution claim in §3.1.
axioms (3)
  • domain assumption LLM-generated NLI labels reliably capture local premise–target relations in long, domain-specific clinical and mathematical traces.
    The entire local signal rests on this. The Limitations section concedes that existing NLI models are not reliable enough for such relations and that the method is 'not fully independent of LLM behaviour'.
  • ad hoc to paper Structural grounding within the generated trace is a valid proxy for the physician-defined trustworthy/problematic distinction.
    §3.5 says audit labels 'do not claim external factual correctness', yet §4 scores them against physician clinical-correctness labels. Table 4 documents the mismatch (S3, S5, S9).
  • domain assumption The AND–OR composition rule — a target is grounded only if every premise in at least one premise set is recursively grounded — adequately formalizes reasoning support.
    This is a reasonable but unverified modeling choice. Abductive support and plausible clinical reasoning may not satisfy strict all-premises-grounding, which the paper's own IMPLIED label acknowledges.

pith-pipeline@v1.3.0-alltime-deepseek · 18618 in / 12568 out tokens · 125565 ms · 2026-08-01T12:00:06.381376+00:00 · methodology

0 comments
read the original abstract

AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer. We propose a reasoning-based, reference-free framework for auditing LLM-generated outputs. The method decomposes a generated reasoning trace into segments, labels local premise-target relations using Natural Language Inference (NLI), and organizes these relations into a hypergraph. A deterministic backward AND-OR search then assigns segment-level audit labels that indicate how each segment is grounded within the generated response. We evaluate the framework in two settings: deductive mathematical reasoning with Hard2Verify, and open-ended medical reasoning with UroReason, a new physician-annotated benchmark of LLM reasoning traces from real clinical cases. Across these settings, our NLI-hypergraph audit provides a more reliable reference-free evaluation signal than direct LLM-as-judge baselines. In the clinical setting, state-of-the-art LLM judges often fail to identify problematic reasoning segments, over-accepting fluent but weakly grounded responses. Our results show that QA evaluation should account for how inferential relations compose across a reasoning trace, rather than relying only on final answers or LLMs as verifiers. UroReason will be made available through an API, and our code will be released as open source.

Figures

Figures reproduced from arXiv: 2607.19678 by Gregory E Dean, Guneet Singh Kohli, Maria Liakata, Michael Sejr Schlichtkrull, Yuxiang Zhou.

Figure 1
Figure 1. Figure 1: NLI-hypergraph audit framework. The response is segmented into reasoning units, candidate premise– target relations are labelled with NLI, and the resulting relations are organized into a hypergraph. A backward AND–OR search then composes local support relations to determine whether each segment is grounded in the generated trace, yielding audit labels such as SUPPORTED, UNSUPPORTED, and ORPHAN. Orange arr… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of UROREASON. Real clinical cases are used to elicit LLM-generated next-step rec￾ommendations and reasoning traces. These traces are segmented into reasoning units and annotated by physi￾cians, producing a benchmark for evaluating step-level reasoning-audit methods. ently from it, later segments may still be marked as structurally supported. This scope is intentional. In open-ended ques￾tion answe… view at source ↗
Figure 4
Figure 4. Figure 4: Structural signatures of mathematical and [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A break down representation of how the reasoning hypergraph looks like [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 1 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [9]

    Do Natural Language Explanations Represent Valid Logical Arguments? Verifying Entailment in Explainable NLI Gold Standards

    Valentino, Marco and Pratt-Hartmann, Ian and Freitas, Andr \'e. Do Natural Language Explanations Represent Valid Logical Arguments? Verifying Entailment in Explainable NLI Gold Standards. Proceedings of the 14th International Conference on Computational Semantics (IWCS). 2021

    2021

  8. [10]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  9. [14]

    2025 , eprint=

    LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models , author=. 2025 , eprint=

    2025

  10. [15]

    2025 , eprint=

    Automating Expert-Level Medical Reasoning Evaluation of Large Language Models , author=. 2025 , eprint=

    2025

  11. [21]

    2023 , eprint=

    Let's Verify Step by Step , author=. 2023 , eprint=

    2023

  12. [23]

    2025 , eprint=

    Entailed Between the Lines: Incorporating Implication into NLI , author=. 2025 , eprint=

    2025

  13. [24]

    Neither Valid nor Reliable? Investigating the Use of LLMs as Judges , url =

    Chehbouni, Khaoula and Haddou, Mohammed and Cheung, Jackie CK and Farnadi, Golnoosh , booktitle =. Neither Valid nor Reliable? Investigating the Use of LLMs as Judges , url =

  14. [25]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  15. [27]

    2022 , eprint=

    MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering , author=. 2022 , eprint=

    2022

  16. [28]

    2015 , eprint=

    A large annotated corpus for learning natural language inference , author=. 2015 , eprint=

    2015

  17. [29]

    2025 , eprint=

    Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math , author=. 2025 , eprint=

    2025

  18. [30]

    2025 , eprint=

    Self-Preference Bias in LLM-as-a-Judge , author=. 2025 , eprint=

    2025

  19. [32]

    International Conference on Learning Representations , volume=

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting , author=. International Conference on Learning Representations , volume=

  20. [33]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  21. [34]

    Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. arXiv preprint arXiv:1908.05739

    Pith/arXiv arXiv 2019

  22. [35]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://arxiv.org/abs/1508.05326 A large annotated corpus for learning natural language inference . Preprint, arXiv:1508.05326

    Pith/arXiv arXiv 2015

  23. [36]

    Khaoula Chehbouni, Mohammed Haddou, Jackie CK Cheung, and Golnoosh Farnadi. 2025. https://proceedings.neurips.cc/paper_files/paper/2025/file/829e8f32d76a6248815bf5b01633811d-Paper-Position_Paper_Track.pdf Neither valid nor reliable? investigating the use of llms as judges . In Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc

    2025

  24. [37]

    Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androutsopoulos

    Odysseas S. Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androutsopoulos. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.1368 G reek B ar B ench: A challenging benchmark for free-text legal reasoning and citations . In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25099--25119, Suzhou, China. Association fo...

    doi:10.18653/v1/2025.findings-emnlp.1368 2025

  25. [38]

    Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya, and Alex Fabrikant. 2025. https://arxiv.org/abs/2501.07719 Entailed between the lines: Incorporating implication into nli . Preprint, arXiv:2501.07719

    Pith/arXiv arXiv 2025

  26. [39]

    Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, and Mor Geva. 2024. https://doi.org/10.18653/v1/2024.acl-long.254 A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains . In Proceedings of the 62nd Annual Meeting of the Association for Comput...

    doi:10.18653/v1/2024.acl-long.254 2024

  27. [40]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's verify step by step . Preprint, arXiv:2305.20050

    Pith/arXiv arXiv 2023

  28. [41]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. https://arxiv.org/abs/2203.14371 Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering . Preprint, arXiv:2203.14371

    Pith/arXiv arXiv 2022

  29. [42]

    Shrey Pandit, Austin Xu, Xuan-Phi Nguyen, Yifei Ming, Caiming Xiong, and Shafiq Joty. 2025. https://arxiv.org/abs/2510.13744 Hard2verify: A step-level verification benchmark for open-ended frontier math . Preprint, arXiv:2510.13744

    arXiv 2025

  30. [43]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. https://doi.org/10.52202/079017-2197 Llm evaluators recognize and favor their own generations . In Advances in Neural Information Processing Systems, volume 37, pages 68772--68802. Curran Associates, Inc

    doi:10.52202/079017-2197 2024

  31. [44]

    Ellie Pavlick and Tom Kwiatkowski. 2019. https://doi.org/10.1162/tacl_a_00293 Inherent disagreements in human textual inferences . Transactions of the Association for Computational Linguistics, 7:677--694

    doi:10.1162/tacl_a_00293 2019

  32. [45]

    Dennis, and Andre Freitas

    Xin Quan, Marco Valentino, Louise A. Dennis, and Andre Freitas. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.172 Verification and refinement of natural language explanations through LLM -symbolic theorem proving . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2933--2958, Miami, Florida, USA. Associat...

    doi:10.18653/v1/2024.emnlp-main.172 2024

  33. [46]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In International Conference on Learning Representations, volume 2024, pages 25055--25083

    2024

  34. [47]

    Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. 2025. https://arxiv.org/abs/2504.10415 Llm-srbench: A new benchmark for scientific equation discovery with large language models . Preprint, arXiv:2504.10415

    Pith/arXiv arXiv 2025

  35. [48]

    Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. https://doi.org/10.18653/v1/2025.acl-long.1230 PRMB ench: A fine-grained and challenging benchmark for process-level reward models . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25299--25346, Vienna, Austria....

    doi:10.18653/v1/2025.acl-long.1230 2025

  36. [49]

    Qwen Team. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    Pith/arXiv arXiv 2025

  37. [50]

    Marco Valentino, Ian Pratt-Hartmann, and Andr \'e Freitas. 2021. https://aclanthology.org/2021.iwcs-1.8/ Do natural language explanations represent valid logical arguments? verifying entailment in explainable NLI gold standards . In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 76--86, Groningen, The Netherlands...

    2021

  38. [51]

    Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2025. https://arxiv.org/abs/2410.21819 Self-preference bias in llm-as-a-judge . Preprint, arXiv:2410.21819

    Pith/arXiv arXiv 2025

  39. [52]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1...

    doi:10.18653/v1/n18-1101 2018

  40. [53]

    Jaehoon Yun, Jiwoong Sohn, Jungwoo Park, Hyunjae Kim, Xiangru Tang, Daniel Shao, Yong Hoe Koo, Ko Minhyeok, Qingyu Chen, Mark Gerstein, Michael Moor, and Jaewoo Kang. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.837 M ed- PRM : Medical reasoning models with stepwise, guideline-verified process rewards . In Proceedings of the 2025 Conference on Empiri...

    doi:10.18653/v1/2025.emnlp-main.837 2025

  41. [54]

    Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. 2024. https://doi.org/10.52202/079017-3797 Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in ...

    work page doi:10.52202/079017-3797 2024

  42. [55]

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. https://doi.org/10.18653/v1/2025.acl-long.50 P rocess B ench: Identifying process errors in mathematical reasoning . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

    doi:10.18653/v1/2025.acl-long.50 2025

  43. [56]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595--46623

    2023

  44. [57]

    Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay Welton, Xinnie Mai, Yanwei Jin, Zidu Xu, Yuen-Hei Chung, Yiyun Xing, Meng-Han Tsai, Emma Schaffer, Yucheng Shi, Ninghao Liu, Zirui Liu, and Rui Zhang. 2025. https://arxiv.org/abs/2507.07988 Automating expert-level medical reasoning evaluation of large language ...

    Pith/arXiv arXiv 2025

  45. [58]

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.108 P ro SA : Assessing and understanding the prompt sensitivity of LLM s . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950--1976, Miami, Florida, USA. Association for Computational Li...

    doi:10.18653/v1/2024.findings-emnlp.108 2024

  46. [59]

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362

    Pith/arXiv arXiv 2025