Pith. sign in

REVIEW 4 major objections 5 minor 39 references

This paper claims that disagreement between a model's direct answer and its answer under a task-equivalent decomposition is a precise signal of error, and that abstaining on disagreement outperforms standard uncertainty baselines in closed-

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:23 UTC pith:U2MNZ5MY

load-bearing objection Strong empirical study with a simple, useful abstention method; the central claims hold, but the single-LLM judge needs independent validation before the F1 numbers are taken at face value. the 4 major comments →

arxiv 2602.04853 v2 pith:U2MNZ5MY submitted 2026-02-04 cs.CL

Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"

Dhruv Madhwal , Lyuxin David Zhang , Dan Roth , Tomer Wolfson , Vivek Gupta This is my paper
classification cs.CL
keywords question decompositionabstentionhallucination detectionuncertainty estimationmulti-hop QAconsistency probingclosed-book QAreliability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether decomposed prompting can do more than improve accuracy: it can reveal when a model's answer should not be trusted. By comparing direct answers with answers obtained through two task-equivalent decomposition regimes across nine models and six multi-hop QA benchmarks, the authors find that accuracy gains from decomposition disappear for frontier models, but cross-regime agreement becomes a strong predictor of correctness. They introduce Disagreement-Based Abstention (DBA), which outputs 'I don't know' whenever the direct and decomposed answers conflict, and show it improves error-detection F1 and AUROC over prompting-based uncertainty baselines. This matters because it offers a practical, training-free way to make closed-book LLM QA systems recognize their own knowledge limits without retrieval or fine-tuning.

Core claim

The central discovery is a scale-dependent utility shift in question decomposition. For smaller and mid-scale models, decomposition acts as a reasoning scaffold that boosts accuracy; for frontier LLMs these gains plateau or go negative. However, for large models, a disagreement between the Direct answer and a decomposed answer is a highly precise indicator that the Direct answer is wrong. The paper leverages this into Disagreement-Based Abstention (DBA), a training-free abstention policy that rejects answers whenever Direct and decomposed outputs are semantically inequivalent. Across nine LLMs and six datasets, DBA outperforms standard uncertainty baselines such as AYS and IC-IDK, substantia

What carries the argument

The key machinery is a fixed, gold-standard decomposition expressed in a domain-specific language (DSL), executed through three task-equivalent prompting regimes: Direct (single answer), Assistive (all sub-questions answered in one call), and Incremental (sub-questions answered one per call). The DSL guarantees semantic equivalence between the decomposition and the original query, so any disagreement between regimes must stem from model execution or factual knowledge, not from a flawed plan. DBA uses an LLM-as-judge protocol to detect semantic disagreement and abstains on any conflict.

Load-bearing premise

The gold DSL decompositions are semantically equivalent to the original question and free of factual errors; if a decomposition contains outdated or wrong intermediate facts, disagreement will be misattributed to the model and DBA will falsely abstain on correct direct answers.

What would settle it

Construct a benchmark where gold decompositions for a set of questions are deliberately corrupted with outdated or factually wrong intermediate steps, while the direct answers remain correct. If DBA's abstention precision drops sharply on these corrupted questions compared to clean controls, it would confirm that the disagreement signal is driven by decomposition plan errors rather than by genuine model uncertainty.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • DBA provides a training-free, retrieval-free abstention mechanism that improves error detection F1 and AUROC across models and datasets.
  • Cross-regime agreement becomes a progressively stronger accuracy signal with model scale, with reliability multipliers exceeding 50x for some frontier models.
  • Ensembling DBA with self-reported confidence methods like AYS catches both stable-but-uncertain errors and confident-but-fragile hallucinations.
  • DBA flags confident-but-incorrect answers that calibration-based methods often miss, closing a critical gap in closed-book QA reliability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same cross-regime disagreement signal could be extended to other decomposable tasks beyond multi-hop QA, such as math word problems or structured extraction, where gold decompositions can be defined.
  • Combining DBA with sampling-based self-consistency might mitigate self-consistency's low recall while retaining its high precision, since DBA varies the reasoning interface rather than only decoding temperature.
  • The persistent cross-prompt inconsistency in frontier models, even when both regimes use the same gold plan, suggests that these models may not maintain a stable internal world model across semantically equivalent reasoning paths—a caution for deploying them in high-stakes closed-book settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies three task-equivalent prompting regimes (Direct, Assistive, Incremental) on six multi-hop QA datasets across nine LLMs, using manually verified DSL decompositions. It reports that decomposition accuracy gains plateau for frontier models, while disagreement between Direct and decomposed outputs is a useful error signal. The authors introduce Disagreement-Based Abstention (DBA), which abstains when the Direct answer and a decomposed answer are not semantically equivalent, and compare it against AYS, IC-IDK, and self-consistency, reporting F1 and AUROC improvements across models and datasets.

Significance. If the empirical claims hold, the paper makes a useful contribution: a training-free, no-retrieval abstention method motivated by a broad 9-model/6-dataset study, with the public release of code, data, and prompts and an honest discussion of failure modes in Tables 17–18. The manual verification of DSL decompositions and the separate analysis of 'Helped'/'Hurt' cases are strengths. However, the central comparison depends on an unvalidated LLM-as-judge instrument and on point estimates without confidence intervals; until these are addressed, the strength of the headline claim is not fully established.

major comments (4)
  1. [Section 3, 'Consistency Protocol'; Appendix C.1] The headline F1/AUROC comparison for DBA is computed from semantic-equivalence labels produced by Gemini-2.5-Flash, which is also one of the nine evaluated models. DBA's abstention decision is exactly the judge's equivalence label, whereas AYS and IC-IDK do not use this judge; a systematic judge bias (e.g., family-specific leniency or asymmetric application of the tolerance/subset rules) can shift DBA's operating point without affecting the baselines. No human agreement study or independent judge is reported. Please add a validation sample (e.g., 200–300 comparisons stratified by model and dataset) with human equivalence labels, report inter-annotator agreement, and check sensitivity by re-running the evaluation with a second LLM judge. Without this, the reported DBA advantage is not fully distinguishable from judge artifact.
  2. [Tables 3, 7, 8; Section 4.2] No confidence intervals or significance tests are provided, although many pairwise differences are small. For example, on GPT-5.1 Mintaka, DBA-A F1 is 0.43 versus AYS 0.42 (Table 3); and on several Qwen3-8B cells the ensemble AUROC is lower than DBA alone (e.g., MuSiQue: 0.62 vs 0.65). The claim that DBA 'outperforms standard uncertainty baselines' across 18 model–dataset pairs needs bootstrap confidence intervals and pairwise tests (e.g., McNemar for accept/reject decisions, bootstrap or DeLong for AUROC). Point estimates alone are not sufficient for the central empirical claim.
  3. [Table 17; Table 18; Section 2] The abstract and Section 4 describe disagreement as a 'highly precise' error signal, but Table 17 shows that 17.6% (GPT-5.1) and 16.7% (Gemini Pro) of disagreements are 'Hurt' cases where Direct is correct and the decomposed path is wrong. Example 4 in Table 18 is a concrete case where an outdated intermediate fact (Salesforce Tower vs Wilshire Grand Center) causes a false abstention. This weakens the strong wording of Section 2 that the decomposition 'guarantees a semantic equivalence' and means the precision of the disagreement signal is around 0.82 in aggregate, not near-perfect. Please reframe the claim and report the precision of the disagreement signal explicitly.
  4. [Appendix D.2, Table 16] The self-consistency comparison is not cost-controlled: self-consistency uses 7 generations (six at T=0.7 plus one at T=0) while DBA-A uses two calls and DBA-I uses one call per hop plus the Direct call. The much higher recall of DBA could partly reflect additional inference budget. Please report a matched-cost comparison (e.g., self-consistency with 2 samples) or clearly state the compute asymmetry in the main comparison. Also, the self-consistency temperature and majority-vote threshold are additional free parameters; a sensitivity analysis would make the comparison more robust.
minor comments (5)
  1. [Table 6] FanOutQA appears in the dataset-average consistency table, but FanOutQA is not listed among the six evaluated datasets in Section 3. Please remove it or explain the mismatch.
  2. [Appendix B, Table 9] The text says Bamboogle uses the full dataset of 125 questions, but Table 9 reports 123 final instances. Please reconcile.
  3. [Table 11] Typo: 'Return exaclty two lines' should be 'exactly'.
  4. [Section 3, baselines] IC-IDK uses K=15 and D=4 without sensitivity analysis. Since these hyperparameters affect baseline strength, please report at least a small sweep or justify the choice.
  5. [Appendix C.1] The judge uses four few-shot examples per dataset, but the examples are not shown. Including them would improve reproducibility.

Circularity Check

0 steps flagged

No significant circularity: DBA is a fixed training-free abstention rule and the consistency–accuracy correlation is empirical. Minor self-citation (DSL syntax) is not load-bearing; main risk is LLM-judge instrumentation, not circular derivation.

full rationale

The central claim is that cross-regime answer disagreement predicts Direct-answer errors. DBA is defined as abstaining whenever the LLM judge labels Direct and decomposed answers non-equivalent; it has no trainable parameters and is not fitted to correctness labels. The consistency–accuracy link is therefore an empirical correlation, not a tautology: Table 17 shows ~67% of inconsistent cases are 'Both wrong' (knowledge gaps), and Table 18 contains cases where Direct is correct while Assistive is wrong (e.g., Example 4 with an outdated intermediate fact), so disagreement does not equal error by construction. The only self-citation that could be relevant is Wolfson et al. (2025) for the DSL syntax (Section 2); this is not load-bearing because the paper manually verifies every decomposition and argues semantic equivalence from the construction of subquestions, not from a cited uniqueness theorem. No parameter is fitted to the gold labels, and the comparison against AYS/IC-IDK/self-consistency is an external benchmark comparison. The legitimate concern is instrumental: Gemini-2.5-Flash is both the semantic-equivalence judge and one of the nine evaluated models, and no human agreement study or second independent judge is reported (Section 3, Appendix C.1). This could bias the measured consistency/accuracy values and hence the headline F1/AUROC comparison, but it is a measurement-validity risk, not a circular derivation. The paper's Limitations section also acknowledges DBA's dependence on high-quality DSL decompositions. Overall, the derivation chain is self-contained; minor self-citation and judge-instrumentation concerns keep the circularity burden low.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No fitted parameters or invented entities. The central claims rest on the quality of the gold decompositions and the judge model; the listed free parameters are baseline configurations that could affect the comparison but are not core to DBA itself.

free parameters (4)
  • IC-IDK demonstration count K = 15
    Number of in-context demonstrations in the IC-IDK baseline (Section 4.1); chosen by the authors, not tuned by validation.
  • IC-IDK 'I don't know' demonstration count D = 4
    Number of IDK examples among the K demonstrations; ad hoc selection.
  • Self-consistency sample count = 7
    Number of samples for self-consistency baseline (Appendix D.2); fixed by the authors.
  • Self-consistency temperature = 0.7
    Sampling temperature for self-consistency; design choice.
axioms (4)
  • domain assumption The DSL decomposition is semantically equivalent to the original question, so correct sub-answers imply the correct final answer.
    Section 2 states this equivalence; it is central to interpreting cross-regime disagreement as an error signal rather than a planning artifact.
  • domain assumption The LLM-as-judge (Gemini-2.5-Flash) accurately assesses semantic equivalence between answers and correctness against gold.
    All consistency and correctness labels are produced by this judge; systematic judge bias would affect both accuracy measurements and DBA's disagreement labels. The judge is also an evaluated model, creating a potential confound.
  • domain assumption Correct answers are stable across prompting regimes while incorrect answers are more likely to vary across regimes.
    This is the motivation for DBA, supported by Reliability Multiplier values but not proven as a general law. The paper itself shows hallucinations are stable across decoding samples, only varying across prompting regimes.
  • domain assumption Greedy decoding (T=0) makes model outputs deterministic and reproducible.
    Section 3 states all models use greedy decoding; for GPT-5.1 temperature is not exposed, so reasoning effort is set to medium, a weaker reproducibility guarantee.

pith-pipeline@v1.3.0-alltime-deepseek · 27603 in / 11465 out tokens · 102252 ms · 2026-08-03T04:23:23.316184+00:00 · methodology

0 comments
read the original abstract

Large language models often struggle to recognize their knowledge limits in closed-book question answering, leading to confident hallucinations. While decomposed prompting is typically used to improve accuracy, we investigate its impact on reliability. We evaluate three task-equivalent prompting regimes: Direct, Assistive, and Incremental, across different model scales and multi-hop QA benchmarks. We find that although accuracy gains from decomposition diminish in frontier models, disagreements between prompting regimes remain highly indicative of potential errors. Because factual knowledge is typically stable while hallucinations are stochastic, cross-regime agreement provides a precise signal of internal uncertainty. We leverage this signal to implement a training-free abstention policy that requires no retrieval or fine-tuning. Our results show that disagreement-based abstention outperforms standard uncertainty baselines as an error detector, improving both F1 and AUROC across settings. This demonstrates that decomposition-based prompting can serve as a practical diagnostic probe for model reliability in closed-book QA.

Figures

Figures reproduced from arXiv: 2602.04853 by Dan Roth, Dhruv Madhwal, Lyuxin David Zhang, Tomer Wolfson, Vivek Gupta.

Figure 1
Figure 1. Figure 1: DSL decomposition for a multi-hop question. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy vs. consistency rate across 9 models and 6 datasets, grouped by difficulty. Each point represents a (model, dataset) pair. Marker shape encodes LLM size (Frontier, 70B, 32B, 8B) while colors encode different evaluation datasets. Model Bamboogle Mintaka HotpotQA CRAG FRAMES MuSiQue Dir ∆A ∆I Dir ∆A ∆I Dir ∆A ∆I Dir ∆A ∆I Dir ∆A ∆I Dir ∆A ∆I Mistral 7B 15.6 +20.2 +21.0 40.9 +14.8 +9.7 24.1 +3.6 +0.4… view at source ↗
Figure 3
Figure 3. Figure 3: Disagreement-Based Abstention (DBA) Framework. Our method compares a Direct answer against Assistive/Incremental reasoning paths, if the semantic claims disagree, the model abstains (IDK). the harmonic mean of precision and recall; and (d) AUROC, which measures the method’s ability to separate incorrect from correct Direct answers. While precision, recall, and F1 are sensitive to the base error rate, which… view at source ↗
Figure 4
Figure 4. Figure 4: Incremental accuracy vs. Incremental consistency rate (Direct-Incremental agreement) across 9 models [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 3 canonical work pages

  1. [1]

    Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.778 LM vs LM : Detecting factual errors via cross examination . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12621--12640, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.emnlp-main.778 2023

  2. [2]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 others. 2025. https://arxiv.org/abs/2507.06261 Gemini 2.5: Pus...

    Pith/arXiv arXiv 2025

  3. [3]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. https://doi.org/10.18653/v1/2024.findings-acl.212 Chain-of-verification reduces hallucination in large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 3563--3578, Bangkok, Thailand. Association...

    doi:10.18653/v1/2024.findings-acl.212 2024

  4. [4]

    Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. https://doi.org/10.18653/v1/W17-5526 F rames: a corpus for adding memory to goal-oriented dialogue systems . In Proceedings of the 18th Annual SIG dial Meeting on Discourse and Dialogue , pages 207--219, Saarbr \"u cken, German...

    doi:10.18653/v1/w17-5526 2017

  5. [5]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3...

    Pith/arXiv arXiv 2024

  6. [6]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \'e lio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth \'e e Lacroix, and William El Sayed. 2023. https://arxiv.org/ab...

    Pith/arXiv arXiv 2023

  7. [7]

    Yichen Jiang and Mohit Bansal. 2019. https://doi.org/10.18653/v1/P19-1262 Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2726--2736, Florence, Italy. Association for Computational Linguistics

    doi:10.18653/v1/p19-1262 2019

  8. [8]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. https://arxiv.org/abs/2207.05221 Language models (mostly...

    Pith/arXiv arXiv 2022

  9. [9]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. https://openreview.net/forum?id=_nGgzQjzaRy Decomposed prompting: A modular approach for solving complex tasks . In The Eleventh International Conference on Learning Representations

    2023

  10. [10]

    Tim Knappe, Ryan Luo Li, Ayush Chauhan, Kaylee Chhua, Kevin Zhu, and Sean O'Brien. 2024. https://openreview.net/forum?id=2w0CIzWlle Enhancing language model reasoning via weighted reasoning in self-consistency . In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24

    2024

  11. [11]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.557 S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004--9017, Singapore. Association for Computational...

    doi:10.18653/v1/2023.emnlp-main.557 2023

  12. [12]

    Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. https://doi.org/10.18653/v1/P19-1613 Multi-hop reading comprehension through question decomposition and rescoring . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097--6109, Florence, Italy. Association for Computational Linguistics

    doi:10.18653/v1/p19-1613 2019

  13. [13]

    Grigor Nalbandyan, Rima Shahbazyan, and Evelina Bakhturina. 2025. https://doi.org/10.18653/v1/2025.naacl-industry.39 SCORE : Systematic CO nsistency and robustness evaluation for large language models . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies...

    work page doi:10.18653/v1/2025.naacl-industry.39 2025

  14. [14]

    Jekaterina Novikova, Carol Myrick Anderson, Borhane Blili-Hamelin, and Subhabrata Majumdar. 2025. https://openreview.net/forum?id=ejvvhJZJSf Consistency in language models: Current landscape, challenges, and future directions . In ICML 2025 Workshop on Reliable and Responsible Foundation Models

    2025

  15. [15]

    OpenAI . 2025. https://cdn.openai.com/pdf/4173ec8d-1229-47db-96de-06d87147e07e/5_1_system_card.pdf Gpt-5.1 instant and gpt-5.1 thinking system card addendum . Accessed 2026-01-03

    2025

  16. [16]

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.378 Measuring and narrowing the compositionality gap in language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687--5711, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.378 2023

  17. [17]

    Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. 2025. https://doi.org/10.1109/COMST.2025.3527641 Mobile edge intelligence for large language models: A contemporary survey . IEEE Communications Surveys & Tutorials

    arXiv 2025

  18. [18]

    Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandrasekaran, and 5 others. 2023. https://arxiv.org/abs/2307.11768...

    Pith/arXiv arXiv 2023

  19. [19]

    Harsh Raj, Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. 2025. https://openreview.net/forum?id=asiBW1bB9b Improving consistency in large language models through chain of guidance . Transactions on Machine Learning Research

    2025

  20. [20]

    Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. https://aclanthology.org/2022.coling-1.138/ Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering . In Proceedings of the 29th International Conference on Computational Linguistics, pages 1604--1619, Gyeongju, Republic of Korea. International Committee on Computationa...

    2022

  21. [21]

    Adi Simhi, Itay Itzhak, Fazl Barez, Gabriel Stanovsky, and Yonatan Belinkov. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.792 Trust me, I ' m wrong: LLM s hallucinate with certainty despite knowing the answer . In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 14665--14688, Suzhou, China. Association for Computationa...

    doi:10.18653/v1/2025.findings-emnlp.792 2025

  22. [22]

    Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. 2025. https://arxiv.org/abs/2509.09677 The illusion of diminishing returns: Measuring long horizon execution in llms . Preprint, arXiv:2509.09677

    arXiv 2025

  23. [23]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/tacl_a_00475 M u S i Q ue: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    doi:10.1162/tacl_a_00475 2022

  24. [24]

    Ashok Urlana, Charaka Vinayak Kumar, Ajeet Kumar Singh, Bala Mallikarjunarao Garlapati, Srinivasa Rao Chalamala, and Rahul Mishra. 2025. https://arxiv.org/abs/2402.14558 Llms with industrial lens: Deciphering the challenges and prospects -- a survey . Preprint, arXiv:2402.14558

    Pith/arXiv arXiv 2025

  25. [25]

    Rui Wang, Zhiyong Gao, Liuyang Zhang, Shuaibing Yue, and Ziyi Gao. 2025. https://doi.org/10.1016/j.cosrev.2025.100755 Empowering large language models to edge intelligence: A survey of edge efficient llms and techniques . Computer Science Review, 57:100755

    arXiv 2025

  26. [26]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conference on Learning Representations

    2023

  27. [27]

    Tomer Wolfson, Daniel Deutch, and Jonathan Berant. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.193 Weakly supervised text-to- SQL parsing through question decomposition . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2528--2542, Seattle, United States. Association for Computational Linguistics

    doi:10.18653/v1/2022.findings-naacl.193 2022

  28. [28]

    Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. https://doi.org/10.1162/tacl_a_00309 Break it down: A question understanding benchmark . Transactions of the Association for Computational Linguistics, 8:183--198

    doi:10.1162/tacl_a_00309 2020

  29. [29]

    Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, and Reut Tsarfaty. 2025. Monaco: More natural and complex questions for reasoning across dozens of documents. Transactions of the Association for Computational Linguistics

    2025

  30. [30]

    Karlsson, and Manabu Okumura

    Jian Wu, Linyi Yang, Yuliang Ji, Wenhao Huang, Börje F. Karlsson, and Manabu Okumura. 2024. https://arxiv.org/abs/2402.11166 Gendec: A robust generative question-decomposition method for multi-hop reasoning . Preprint, arXiv:2402.11166

    Pith/arXiv arXiv 2024

  31. [31]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025 a . https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    Pith/arXiv arXiv 2025

  32. [32]

    Qwen: An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025 b . https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    Pith/arXiv arXiv 2025

  33. [33]

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu JIANG, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, and 8 others. 2024. https://openreview.net/forum?id=Q7lAqY41HH CRAG - comprehensive RAG benchmark . In The Thirty-eight Con...

    2024

  34. [34]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369--2380, Brussels...

    doi:10.18653/v1/d18-1259 2018

  35. [35]

    Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.1032 SAC ^3 : Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15445--15458, Singapore. ...

    doi:10.18653/v1/2023.findings-emnlp.1032 2023

  36. [36]

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2024. https://doi.org/10.18653/v1/2024.naacl-long.390 Knowing what LLM s DO NOT know: A simple yet effective self-detection method . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa...

    work page doi:10.18653/v1/2024.naacl-long.390 2024

  37. [37]

    Denny Zhou, Nathanael Sch \"a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. https://openreview.net/forum?id=WZH7099tgfM Least-to-most prompting enables complex reasoning in large language models . In The Eleventh International Conference on Learning Representations

    2023

  38. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  39. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...