Pith. sign in

REVIEW 2 major objections 2 cited by

Graph constraints on reasoning paths expand usable multi-hop training data by 4.4 times without improving per-path quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 22:23 UTC pith:HHHHTNK6

load-bearing objection They get 4.4x more multi-hop examples from legal text by offline graph path selection with five geometric constraints, and the ablation shows the gain is volume not quality. the 2 major comments →

arxiv 2605.31238 v1 pith:HHHHTNK6 submitted 2026-05-29 cs.CL cs.LG

Scaling Multi-Hop Training Data via Graph-Constrained Path Selection

Pengyu Chen , Yonggang Zhang , Mingming Chen , Jun Song , Wei Xue , Yike Guo This is my paper
classification cs.CL cs.LG
keywords multi-hop reasoningtraining data generationgraph constraintspath selectioncompositional reasoninglanguage model fine-tuningembedding similaritydata scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that multi-hop reasoning data can be scaled from unstructured text by first building a graph of contextual keyword centroids and enumerating admissible paths using five geometric constraints before asking a teacher model to turn those paths into question-answer pairs. This approach avoids the degradation seen in joint discovery and verbalization methods when dealing with repetitive or densely cross-referenced documents. Experiments demonstrate that the main benefit is a 4.4 times larger usable corpus, leading to improved performance after fine-tuning. At matched training sizes, constrained and unconstrained paths perform similarly, indicating the constraints primarily increase the number of synthesizable examples rather than their individual quality.

Core claim

The central discovery is that enforcing five geometric admissibility constraints on paths through a graph of contextual keyword centroids allows enumeration of 4.4 times more verbalizable multi-hop chains from plain text than unconstrained methods, with the performance gain arising from corpus expansion rather than superior chain quality, as shown by matched-size ablations where both yield indistinguishable results.

What carries the argument

The graph of contextual keyword centroids with five geometric admissibility constraints that enforce local similarity bounds to prevent endpoint drift and escape dense embedding cliques.

Load-bearing premise

The five geometric constraints correctly identify paths that the teacher model can verbalize without introducing systematic bias or excluding valid chains in dense embedding regions formed by boilerplate text.

What would settle it

A matched-scale experiment in which constrained paths produce measurably higher downstream performance than unconstrained paths, or a full-scale run showing no gain from the 4.4 times larger corpus.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Larger volumes of multi-hop data become feasible from specialized corpora without manual curation.
  • Teacher models can be invoked only on pre-validated paths, increasing the fraction of synthesizable examples.
  • Performance on compositional reasoning tasks improves through data volume rather than path quality.
  • The constraints function to raise teacher synthesizability rather than to select higher-quality chains.
  • At equal training scale the two approaches produce equivalent results on downstream tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same decoupling of enumeration and verbalization could apply to other domains with repetitive text structures.
  • Relaxing one or more of the five constraints might alter the expansion factor and offer a tunable parameter for data volume.
  • The result suggests that quantity of examples can outweigh per-example sophistication in multi-hop fine-tuning regimes.
  • Similar graph-based filtering might help scale other forms of synthetic training data where joint generation fails.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The manuscript presents a method for scaling multi-hop QA training data from unannotated specialized text by enumerating paths offline on a graph of contextual keyword centroids subject to five geometric admissibility constraints, with a teacher model used only for verbalization of validated paths. Gram-matrix arguments are provided to justify the constraints by showing that local similarity alone allows up to ~91° endpoint drift and that an upper bound is needed to escape dense cliques from boilerplate. A matched-size ablation demonstrates that constrained and unconstrained chains perform similarly at equal scale, attributing the overall gain to a 4.4× increase in usable data volume. Fine-tuning Qwen3-32B on 80K examples from the CUAD corpus improves closed-book Token F1 from 21.66% to 38.58%. Code is released.

Significance. If the central claim holds—that the constraints primarily increase the number of teacher-verbalizable paths without altering per-path quality or introducing bias—this work could meaningfully advance data construction for compositional reasoning in domain-specific corpora characterized by repetitive structures. The explicit ablation isolating volume from quality is a strength, as is the public release of the code, which supports reproducibility. The approach reframes graph-based constraints as a scalability tool rather than a quality filter.

major comments (2)
  1. [Methods (Gram-matrix arguments)] Methods section, Gram-matrix arguments for the five constraints: the claim that these constraints raise teacher synthesizability without systematic bias or exclusion of valid multi-hop chains in dense boilerplate regions of CUAD is load-bearing for interpreting the 4.4× expansion as the source of gains. The ~91° drift bound is derived, but no direct evidence (e.g., verbalization success rates on selected vs. rejected paths or manual inspection in template-heavy sections) is shown to confirm the constraints avoid bias or omission in repetitive legal embeddings.
  2. [Experiments (matched-size ablation)] Experiments section, matched-size ablation: the report that constrained and unconstrained chains yield indistinguishable downstream performance at equal scale underpins the volume-over-quality interpretation. However, the construction of the unconstrained set (sampling method, path enumeration without graph, and distribution matching to the constrained set) requires explicit detail to rule out selection effects that could confound the ablation result.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and positive assessment of the work's significance. We address each major comment below and will revise the manuscript to incorporate the requested clarifications and evidence.

read point-by-point responses

  1. Referee: [Methods (Gram-matrix arguments)] the claim that these constraints raise teacher synthesizability without systematic bias or exclusion of valid multi-hop chains in dense boilerplate regions of CUAD is load-bearing... no direct evidence (e.g., verbalization success rates on selected vs. rejected paths or manual inspection in template-heavy sections) is shown to confirm the constraints avoid bias or omission in repetitive legal embeddings.

    Authors: We agree that direct empirical evidence would strengthen the load-bearing claim. The Gram-matrix arguments establish theoretical bounds on drift and clique escape, while the matched-size ablation supports that per-path quality is not systematically altered. In the revision we will add verbalization success rates comparing constrained versus rejected paths and qualitative inspection of examples from template-heavy CUAD sections. revision: yes

  2. Referee: [Experiments (matched-size ablation)] the construction of the unconstrained set (sampling method, path enumeration without graph, and distribution matching to the constrained set) requires explicit detail to rule out selection effects that could confound the ablation result.

    Authors: We will expand the Experiments section with explicit details on the unconstrained-set construction, including the path-enumeration procedure without graph constraints, the sampling method, and the exact distribution-matching criteria used to ensure the sets are comparable at equal scale. revision: yes

Circularity Check

0 steps flagged

No circularity; central claims rest on held-out empirical ablations and self-contained Gram-matrix arguments

full rationale

The paper's key result—that gains arise from 4.4× corpus expansion rather than per-chain quality—is isolated by a matched-size ablation showing indistinguishable downstream performance for constrained vs. unconstrained chains on a held-out legal QA task. The five geometric admissibility constraints are justified by Gram-matrix arguments developed within this manuscript (local similarity bounds permitting ~91° drift and necessity of upper bound to exit cliques), without reference to prior self-citations or fitted parameters that would reduce the claim to its inputs by construction. No derivation step equates a reported prediction or uniqueness result to a quantity defined by the method itself. The analysis is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 1 invented entities

The central claim rests on the validity of five geometric constraints whose justification is supplied by Gram-matrix arguments; these constraints and the graph construction itself are introduced by the paper rather than taken from prior literature.

free parameters (1)
  • similarity bounds for graph edges
    Local and upper similarity thresholds that define admissible paths are required to enforce the geometric constraints; their concrete values are not stated in the abstract but must be chosen for any implementation.
axioms (1)
  • domain assumption Gram-matrix arguments establish that local similarity bounds alone admit endpoint drift up to ~91° and that an upper similarity bound is necessary to exit dense embedding cliques formed by boilerplate text.
    Invoked to justify the five geometric admissibility constraints that filter paths before teacher verbalization.
invented entities (1)
  • graph of contextual keyword centroids no independent evidence
    purpose: To enable offline enumeration of reasoning paths that satisfy the geometric constraints before any teacher model is called.
    A new intermediate representation introduced to decouple path discovery from verbalization; no independent evidence outside the paper is provided.

pith-pipeline@v0.9.1-grok · 5836 in / 1612 out tokens · 45617 ms · 2026-06-28T22:23:48.596336+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Scaling Multi-Hop Training Data via Graph-Constrained Path Selection." pith.science (2026). https://pith.science/paper/HHHHTNK6

@misc{pith2026260531238,
  author       = {Pith},
  title        = {Pith review of: Scaling Multi-Hop Training Data via Graph-Constrained Path Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HHHHTNK6}},
  note         = {Machine review of arXiv:2605.31238}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Endowing large language models with compositional reasoning over specialized documents requires multi-hop training data at scale, where such data rarely exists outside of curated benchmarks built on structured sources. To construct it directly from plain, unannotated text, existing methods ask a single teacher model to jointly discover an evidence path through a document and verbalize it as a question-answer pair. However, these methods degrade sharply when documents are structured around repetitive templates and densely cross-referencing clauses, conditions that characterize most real-world specialized corpora. In this work, we decouple the two operations: reasoning paths are enumerated offline over a graph of contextual keyword centroids, and the teacher is invoked only to verbalize pre-validated paths. The graph enforces five geometric admissibility constraints, for which we provide Gram-matrix arguments establishing that local similarity bounds alone admit endpoint drift up to ${\sim}91^{\circ}$, and that an upper similarity bound is necessary to exit dense embedding cliques formed by boilerplate text. A matched-size ablation isolates the mechanism: at equal training scale, constrained and unconstrained chains yield indistinguishable downstream performance, and the gain at full scale comes from a 4.4$\times$ expansion of the usable corpus rather than from higher per-chain quality -- reframing the role of graph constraints, in this setting, as raising teacher synthesizability rather than improving chain content. Fine-tuning Qwen3-32B on 80K examples constructed from the CUAD legal contract corpus improves closed-book Token F1 from 21.66% to 38.58%. We have released our codes at https://github.com/hkgai-official/GCSCS.

Figures

Figures reproduced from arXiv: 2605.31238 by Jun Song, Mingming Chen, Pengyu Chen, Wei Xue, Yike Guo, Yonggang Zhang.

Figure 1
Figure 1. Figure 1: Illustration of the data yield bottleneck when coupling evidence discovery with language generation. We formalize this barrier as the cognitive synthesizability bot￾tleneck [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the four-stage Graph-Constrained Semantic Chain Synthesis pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Data scaling analysis on the GE closed-book test set. Left: Token F1 vs. semantic-chain [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of max angular radius ϵ per DBSCAN-identified boilerplate cluster in CUAD (n = 5,313 clusters). Blue bars: clusters satisfying ϵ < 12.92◦ (N = 4,667, 87.8%); red bars: clusters violating the bound (N = 646, 12.2%). The Theorem 2 bound at 12.92◦ (= 1 2 arccos(0.90)) is shown as a dashed red line. The distribution peaks near 10–11◦ , confirming that the large majority of CUAD boilerplate cliques… view at source ↗
Figure 5
Figure 5. Figure 5: Max angular radius ϵ by DBSCAN cluster size. Each box shows the interquartile range; the orange line is the median; whiskers extend to the 5th/95th percentile. Blue boxes: median ϵ < 12.92◦ ; red boxes: median ϵ ≥ 12.92◦ . Sample counts n are annotated above each box. The absence of a systematic upward trend with cluster size confirms that the Theorem 2 geometric guarantee does not degrade for larger boile… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

    cs.CL 2026-07 conditional novelty 6.0

    A structured data synthesis pipeline using evidence graphs and a 32-type task taxonomy generates 2,000 SFT samples that improve long-context LLM performance over baselines.

  2. LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

    cs.CL 2026-07 conditional novelty 6.0

    LongCrafter generates long-context SFT data via a 32-task taxonomy and evidence-graph-guided instruction synthesis; trained 7B/8B models outperform prior synthetic SFT baselines on LongBench, LongBench v2, and LooGLE.

Reference graph

Works this paper leans on

49 extracted references · 5 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Self-RAG: Learn- ing to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learn- ing to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations, 2024

    2024

  2. [2]

    AlpaGasus: Training a better alpaca with fewer data

    Lichang Chen, Shiyang Li, Jun Y an, Hai Wang, Kalpa Gunaratna, Vikas Y adav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. AlpaGasus: Training a better alpaca with fewer data. In International Conference on Learning Representations , 2024

    2024

  3. [3]

    Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023

    2023

  4. [4]

    DoG-Instruct: Towards premium instruction-tuning data via text-grounded instruction wrapping

    Y ongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi. DoG-Instruct: Towards premium instruction-tuning data via text-grounded instruction wrapping. In Proceed- ings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4125–4135, 2024

    2024

  5. [5]

    QLoRA: Efficient fine- tuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient fine- tuning of quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, 2023

    2023

  6. [6]

    SciNets: Graph-constrained multi-hop reasoning for scientific literature syn- thesis

    Sauhard Dubey. SciNets: Graph-constrained multi-hop reasoning for scientific literature syn- thesis. arXiv preprint arXiv:2601.09727, 2026

    work page arXiv 2026

  7. [7]

    Faith and fate: Limits of transformers on compositionality

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Y uchen Lin, Peter West, Chandra Bhagavatula, et al. Faith and fate: Limits of transformers on compositionality. In Advances in Neural Information Processing Systems , volume 36, 2023

    2023

  8. [8]

    IIRC: A dataset of incomplete information reading comprehension questions

    James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. IIRC: A dataset of incomplete information reading comprehension questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 1137–1147, 2020

    2020

  9. [9]

    SimCSE: Simple contrastive learning of sen- tence embeddings

    Tianyu Gao, Xingcheng Y ao, and Danqi Chen. SimCSE: Simple contrastive learning of sen- tence embeddings. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing, pages 6894–6910, 2021

    2021

  10. [10]

    TrueTeacher: Learning factual consistency evaluation with large language models

    Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. TrueTeacher: Learning factual consistency evaluation with large language models. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing , pages 2053–2070, 2023

    2053

  11. [11]

    Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics , 9:346–361, 2021

    2021

  12. [12]

    LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models

    Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N Rockmore, et al. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems , volume 36, 2023

    2023

  13. [13]

    arXiv preprint arXiv:2505.17063 , year =

    Yiduo Guo, Zhen Guo, Chuanwei Huang, Zi-Ang Wang, Zekai Zhang, Haofei Y u, Huishuai Zhang, and Yikang Shen. Synthetic data RL: Task definition is all you need. arXiv preprint arXiv:2505.17063, 2025

    work page arXiv 2025

  14. [14]

    CUAD: An expert-annotated NLP dataset for legal contract review

    Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. CUAD: An expert-annotated NLP dataset for legal contract review. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , 2021

    2021

  15. [15]

    Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the International Conference on Computational Linguistics , pages 6609–6625, 2020. 10

    2020

  16. [16]

    Unnatural instructions: Tuning language models with (almost) no human labor

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 14409–14428, 2023

    2023

  17. [17]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Y elong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Y uanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In In- ternational Conference on Learning Representations , 2022

    2022

  18. [18]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Y u, Yiming Y ang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing , pages 7969–7992, 2023

    2023

  19. [19]

    Billion-scale similarity search with GPUs

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535–547, 2019

    2019

  20. [20]

    TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 1601–1611, 2017

    2017

  21. [21]

    Learning from synthetic data improves multi-hop reasoning

    Anmol Kabra, Yilun Yin, Albert Gong, Kamile Stankeviciute, Dongyoung Go, Johann Lee, Katie Z Luo, Carla P Gomes, and Kilian Q Weinberger. Learning from synthetic data improves multi-hop reasoning. In International Conference on Learning Representations , 2026

    2026

  22. [22]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas O ˇguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 6769–6781, 2020

    2020

  23. [23]

    Decomposed prompting: A modular approach for solving complex tasks

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Y ao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations , 2023

    2023

  24. [24]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel V eness, Guillaume Desjardins, An- drei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017

    2017

  25. [25]

    Automatic inter- document multi-hop scientific QA generation

    Seungmin Lee, Dongha Kim, Y uni Jeon, Junyoung Koh, and Min Song. Automatic inter- document multi-hop scientific QA generation. In Proceedings of the International Con- ference on Language Resources and Evaluation , Palma, Spain, 2026. Also available as arXiv:2603.14257

    work page arXiv 2026

  26. [26]

    FactCG: Enhancing fact checkers with graph-based multi-hop data

    Deren Lei, Y axi Li, Siyao Li, Mengya Hu, Rui Xu, Ken Archer, Mingyu Wang, Emily Ching, and Alex Deng. FactCG: Enhancing fact checkers with graph-based multi-hop data. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computa- tional Linguistics, pages 5002–5020, 2025

    2025

  27. [27]

    Retrieval- augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020

    2020

  28. [28]

    DESIGNER: Design-logic-guided multidisciplinary data synthesis for LLM reasoning

    Weize Liu, Y ongchi Zhao, Yijia Luo, Mingyu Xu, Jiaheng Liu, Y anan Li, Xiguo Hu, Zhiqi Bai, Y uchi Xu, Wenbo Su, and Bo Zheng. DESIGNER: Design-logic-guided multidisciplinary data synthesis for LLM reasoning. In International Conference on Learning Representations, 2026

    2026

  29. [29]

    SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 9004–9017, 2023. 11

    2023

  30. [30]

    FActScore: Fine-grained atomic evalua- tion of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evalua- tion of factual precision in long form text generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 12076–12100, 2023

    2023

  31. [31]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744, 2022

    2022

  32. [32]

    Mea- suring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Mea- suring and narrowing the compositionality gap in language models. In Findings of the Associ- ation for Computational Linguistics: EMNLP , pages 5687–5711, 2023

    2023

  33. [33]

    Sentence-BERT: Sentence embeddings using Siamese BERT-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing, pages 3982–3992, 2019

    2019

  34. [34]

    The web as a knowledge-base for answering complex questions

    Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 641–651, 2018

    2018

  35. [35]

    MuSiQue: Multihop questions via single-hop question generation

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question generation. Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  36. [36]

    MAUD: An expert- annotated legal NLP dataset for merger agreement understanding

    Steven H Wang, Antoine Scardigli, Leonard Tang, Wei Chen, Dimitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. MAUD: An expert- annotated legal NLP dataset for merger agreement understanding. In Proceedings of the Con- ference on Empirical Methods in Natural Language Processing , pages 16369–16382, 2023

    2023

  37. [37]

    Self-consistency improves chain of thought reasoning in lan- guage models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in lan- guage models. In International Conference on Learning Representations , 2023

    2023

  38. [38]

    Self-Instruct: Aligning language models with self-generated instruc- tions

    Yizhong Wang, Y eganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning language models with self-generated instruc- tions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 13484–13508, 2023

    2023

  39. [39]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , volume 35, pages 24824– 24837, 2022

    2022

  40. [40]

    WizardLM: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large language models to follow complex instructions. In International Conference on Learning Representations , 2024

    2024

  41. [41]

    HotpotQA: A dataset for diverse, explainable multi-hop question answering

    Zhilin Y ang, Peng Qi, Saizheng Zhang, Y oshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing, pages 2369–2380, 2018

    2018

  42. [42]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Y ao, Dian Y u, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Y uan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems , volume 36, 2023

    2023

  43. [43]

    QA- GNN: Reasoning with language models and knowledge graphs for question answering

    Michihiro Y asunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA- GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the Conference of the North American Chapter of the Association for Compu- tational Linguistics, pages 535–546, 2021. 12

    2021

  44. [44]

    AgenticRAGTracer: A hop-aware benchmark for diagnosing multi-step retrieval reasoning in agentic RAG

    Qijie Y ou, Wenkai Y u, and Wentao Zhang. AgenticRAGTracer: A hop-aware benchmark for diagnosing multi-step retrieval reasoning in agentic RAG. arXiv preprint arXiv:2602.19127 , 2026

    work page internal anchor Pith review arXiv 2026

  45. [45]

    GreaseLM: Graph reasoning enhanced language models

    Xikun Zhang, Antoine Bosselut, Michihiro Y asunaga, Hongyu Ren, Percy Liang, Christo- pher D Manning, and Jure Leskovec. GreaseLM: Graph reasoning enhanced language models. In International Conference on Learning Representations , 2022

    2022

  46. [46]

    LIMA: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Y uning Mao, Xuezhe Ma, Avia Efrat, Ping Y u, Lili Y u, et al. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems, volume 36, 2023

    2023

  47. [47]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Repre- sentations, 2023

    2023

  48. [48]

    find candidate evidence paths that the teacher comfortably verbalizes

    Xinyu Zhu, Yihao Feng, Y anchao Sun, Xianzhi Du, Pingzhi Li, Olli Saarikivi, Y un Zhu, and Y u Meng. CHIMERA: Compact synthetic data for generalizable LLM reasoning. arXiv preprint arXiv:2603.00889, 2026. A Proof of Theorem 1 We provide a constructive proof for each L ≥ 3. Base case (L = 3). Consider the 3 × 3 Gram matrix of (v1, v2, v3): G =   1 7 10 ...

    work page arXiv 2026

  49. [49]

    no transfer

    validation questions (no supporting documents; paired t-tests on per-sample Token F1). Ta- ble 11 summarizes the outcome: Token F1 improvements are at most ∼0.6%—orders of magni- tude smaller than the +16%–+18% gains on our CUAD-style test—so the headline improvement is dominated by domain-specific answer style and citation-template adaptation rather than ...