Pith. sign in

REVIEW 3 major objections 6 minor 42 references

Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Even with chain-of-thought prompting, GPT-3.5 still fails to reliably select and combine external knowledge, handle non-sequential reasoning structures, or generalise across numbers of hops.

desk verdict A small, honest empirical study confirming GPT-3.5's fragility in multi-hop reasoning; the non-sequential reasoning claim needs better support than the depth-length proxy. read the letter →

arxiv 2412.08317 v1 pith:36KCCMXI submitted 2024-12-11 cs.CL

Haotong Zhang This is my paper · ORCID
classification cs.CL
keywords multi-hopreasoningChain-of-Thoughtpromptingexternalknowledgecounterfactualnon-sequentialleast-to-mostGPT-3.5generalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that even the best-known prompting technique, chain-of-thought, does not give large language models reliable multi-hop reasoning when the knowledge must come from outside the model. Using GPT-3.5 on four reasoning benchmarks, it shows that the model often cannot pick out the right knowledge pieces, gets confused when that knowledge contradicts what it learned during pretraining, fails to build correct reasoning chains when the reasoning structure is non-sequential, and cannot generalise from seeing few-hop examples to many-hop questions. If true, the result matters because it locates the bottleneck not in the language model's factual memory but in the composition step: combining separate pieces of evidence into a sound argument. The paper concludes that current LLMs still show a large gap with human reasoning on these tasks.

What carries the argument

The central mechanism under test is chain-of-thought (CoT) prompting, the few-shot instruction that asks the model to write out intermediate steps before the final answer, together with its variant least-to-most prompting, which inserts an explicit decomposition stage. The paper probes this mechanism with four benchmarks that isolate different sub-skills: HotpotQA for selecting two disjoint evidence pieces, EntailmentBank for composing many steps including non-sequential entailment trees, QASC for two-hop answer composition with distractors, and bAbI15 for unanswerable (non-deductive) cases. The key diagnostic object is the 'entailment tree' from EntailmentBank, whose depth versus path length is used to label a reasoning problem as non-sequential, and the paper also uses counterfactual knowledge swaps in QASC and bAbI15 to test how the model separates external from internal knowledge.

What would settle it

Take the EntailmentBank samples the paper labels non-sequential (depth differs from length), have independent human annotators judge whether the reasoning is genuinely non-sequential, and compare GPT-3.5's accuracy on those against a matched set of sequential samples of equal proof length; if most labelled samples are judged sequential, or if a newer model with explicit non-sequential prompting shows no accuracy gap, the central claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs 'still suffer from severe drawbacks' in multi-hop reasoning with external knowledge even when prompted with chain-of-thought (CoT) and its variations. Concretely, four sets of experiments on HotpotQA, EntailmentBank, QASC and bAbI15 with text-davinci-002 show: internal knowledge alone gives low accuracy; adding external knowledge helps but the model rapidly loses the ability to select all gold knowledge as distractors increase; counterfactual knowledge causes the model to retreat to its pretraining knowledge or to the prompt's format, producing answers like 'not mentioned' far too rarely; on samples the paper classifies as non-sequential (where entailment-tree depth and path length differ), accuracy drops to 56% versus 78% on sequential samples and none of the generated reasoning paths is correct; and least-to-most prompting does not transfer across hop counts, with overall accuracy falling as the exemplars' hop count rises.

Load-bearing premise

The paper's classification of a reasoning problem as 'non-sequential' rests entirely on the criterion that the depth and the length of its entailment tree differ; if that proxy does not actually capture non-sequential structure, the claim that chain-of-thought fails on non-sequential reasoning is unsupported.

Editorial extensions

If this is right

  • With only internal knowledge, GPT-3.5 scores 40% on HotpotQA and 35% on EntailmentBank, so multi-hop benchmarks cannot be treated as tests of parametric knowledge alone.
  • Adding distractor-free external knowledge raises accuracy substantially, but the selection step degrades quickly: exact-match selection of both gold paragraphs in HotpotQA falls from 100% to 66% when only 20% of the context is gold.
  • Counterfactual context does not make the model reason counterfactually: it tends to hallucinate the factual knowledge it pretrained on, or to copy the prompt's format, and it fails most unanswerable (non-deductive) bAbI15 items.
  • On non-sequential entailment samples (depth differs from path length), accuracy is 56% and no generated proof is correct, whereas sequential samples reach 78%.
  • Least-to-most prompting does not generalise across hop counts: overall accuracy drops as the exemplars' hop count rises, and the model over-decomposes or mis-decomposes questions even when the final answer is right.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same pattern holds on later and larger models, the bottleneck would look like a property of CoT-style prompting itself rather than of one checkpoint; an easy test is to rerun the non-sequential and counterfactual settings on newer models and on open-weight models.
  • The depth-versus-length criterion could be validated against human judgments of non-sequential structure; if the proxy over- or under-selects examples, the 56% versus 78% gap may be partly a labelling artefact rather than a genuine reasoning deficit.
  • The counterfactual results suggest a practical recipe for making models more robust: teach them when to trust the context over parametric memory by mixing counterfactual exemplars or by adding explicit instructions to follow the context even when it contradicts pretraining, an extension the paper does not test.
  • The shortcut analysis (accuracy above exact-match in the 20% setting) implies that benchmark scores on HotpotQA-style tasks can overstate true multi-hop ability, so future benchmarks should report per-sample evidence that both hops were actually used.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper reports experiments with text-davinci-002 on four multi-hop reasoning benchmarks (HotpotQA, EntailmentBank, QASC, bAbI15) under Chain-of-Thought prompting and its variants. The experiments manipulate the availability of external knowledge, the proportion of distractor sentences, the factual consistency of the provided knowledge, the structural type of reasoning (sequential/non-sequential), and the number of hops in the prompt exemplars. The central empirical claims are that external knowledge helps but distractors and counterfactual knowledge confuse the model, that accuracy drops on selected non-sequential entailment samples, and that least-to-most prompting does not improve hop-count generalization.

Significance. If the results hold, this is a useful negative result: it documents that knowledge selection, counterfactual robustness, and non-sequential reasoning remain weak points for CoT-prompted LLMs, consistent with but more comprehensive than existing findings. The study's strengths are its four-benchmark coverage, the transparent prompt templates in the appendix, and the manual error analysis that identifies specific failure modes such as word bias and hallucination. However, the paper's quantitative support is currently limited by small single-model samples and by an unvalidated structural proxy, so the contribution is exploratory and would need robustness work to serve as a definitive claim.

major comments (3)
  1. [Section 5, 'Reasoning in Non-Sequential Cases'] The paper identifies 'non-sequential' samples by the criterion that 'depth and length of the entailment tree are not the same.' This criterion is not validated, and the two groups may differ in proof length, number of premises, or answer complexity rather than in structural non-sequentiality alone. The reported accuracy gap (56% vs. 78%) is therefore not unambiguously attributable to non-sequential reasoning. Please provide a validation of the proxy (e.g., human-annotated structural labels, matched control sets, or an analysis showing that proof length is not a confound); the examples in Appendix A.2 illustrate the intended phenomenon but do not establish that the criterion separates the classes reliably across the dataset.
  2. [Section 3 and Tables 2-6] Every experiment uses 100 test samples and a single model (text-davinci-002), with no confidence intervals, bootstrap estimates, or significance tests, and the sampling procedure for the 100 samples is not described. Differences such as 83% vs. 74% in Table 2 and 64% vs. 46% in Table 6 could be within sampling noise, especially because decoding is stochastic and correctness is checked manually. The abstract's plural 'LLMs' claim also goes beyond the single evaluated model. Please add uncertainty quantification and either replicate on at least one other model or restrict the claims to GPT-3.5 as evaluated here.
  3. [Section 5 and Footnote 1] The manual evaluation is not sufficiently specified to be reproducible. The paper reports that 'we check each sample manually' and that 'none of the generated reasoning paths is correct' in the non-sequential experiment, but no annotation protocol, error taxonomy, second annotator, inter-annotator agreement, or released annotations are provided. This matters because the qualitative conclusions about error types and about the absence of coherent reasoning paths carry as much weight as the accuracy numbers. Please make the rubric and the per-sample annotations available.
minor comments (6)
  1. [Section 3] Clarify whether text-davinci-002 is the GPT-3.5 model referenced in the abstract, since the naming is potentially confusing.
  2. [Section 5] Define 'depth' and 'length' of an entailment tree explicitly; these terms are used without formal definitions.
  3. [Table 5] The second value in each cell is a raw count (out of 50) rather than a percentage, while the first value is a percentage; this mixed format makes the table hard to read.
  4. [Appendix A.2] There are typos in the prompts ('posive', 'lind', 'invisble') that should be corrected.
  5. [Section 6 and Figure 2] The figure caption does not state the number of generated reasoning paths per setting, which makes the length distributions hard to interpret.
  6. [General] The paper would benefit from a brief statement on data and code availability; the appendices provide prompts but not the test subsets or evaluation code.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark study with externally measured results, no fitted parameters, and no load-bearing self-citation chain.

full rationale

This paper does not present a derivation chain in which conclusions are equivalent to inputs by construction. All headline claims are empirical measurements of GPT-3.5 performance on external benchmarks (HotpotQA, EntailmentBank, QASC, bAbI15) under specific prompting conditions. There are no fitted parameters, no internal equations that define the target result, and no self-citations invoked to forbid alternatives or justify a choice. The authors' operational decisions, such as defining non-sequential reasoning samples in Section 5 as those where 'depth and length of the entailment tree are not the same,' are choices of experimental design and measurement. The resulting accuracy comparison (56% vs 78%) is an external empirical observation on that defined subset, not a statement that is true by definition. Whether that proxy validly captures non-sequential reasoning is a validity or correctness concern, not a circularity concern, and the paper itself does not claim the proxy is derived from first principles. Similarly, the counterfactual-knowledge and hop-generalization experiments compare model outputs against dataset ground truths and prior published results; the conclusions are not forced by the prompts or by any fitted quantity. The absence of any identified circular step warrants a score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted, and no new entities are introduced. The paper's conclusions rest on four domain assumptions: benchmark validity, CoT as the proper probe, the operationalization of non-sequential reasoning (ad hoc), and the reliability of manual scoring.

assumptions (4)
  • domain assumption The four benchmarks (HotpotQA, EntailmentBank, QASC, bAbI15) are valid and informative measures of multi-hop reasoning ability.
    The paper uses these benchmarks as ground truth for reasoning capability without questioning whether dataset biases (e.g., shortcuts in HotpotQA, which the paper itself notes) distort the measurement.
  • domain assumption Chain-of-Thought prompting and its variations are the appropriate way to elicit and evaluate reasoning in LLMs.
    The paper assumes that failures under CoT indicate reasoning limitations rather than prompt sensitivity or format issues.
  • ad hoc to paper The criterion in Section 5 that non-sequential samples have depth and length of the entailment tree that differ is a valid proxy for non-sequential reasoning.
    The paper selects non-sequential samples using this operationalization but provides no validation that it matches the intuitive notion of non-sequential reasoning.
  • domain assumption Manual evaluation by the author is a reliable measure of answer correctness.
    The paper states 'We check each sample manually and consider the answer as correct if it has the same meaning as the ground truth,' without inter-annotator agreement or a scoring rubric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge." pith.science (2026). https://pith.science/paper/36KCCMXI

@misc{pith2026241208317,
  author       = {Pith},
  title        = {Pith review of: Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36KCCMXI}},
  note         = {Machine review of arXiv:2412.08317}
}
read the original abstract

We carry out a series of experiments to test large language models' multi-hop reasoning ability from three aspects: selecting and combining external knowledge, dealing with non-sequential reasoning tasks and generalising to data samples with larger numbers of hops. We test the GPT-3.5 model on four reasoning benchmarks with Chain-of-Thought prompting (and its variations). Our results reveal that despite the amazing performance achieved by large language models on various reasoning tasks, models still suffer from severe drawbacks which shows a large gap with humans.

Figures

Figures reproduced from arXiv: 2412.08317 by the authors.

Figure 1
Figure 1. Two examples on HotpotQA and EntailmentBank respectively where providing external knowledge helps the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of length of proof of generated reasoning paths in different settings. X-axis indicates the lengths [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 27 canonical work pages

  1. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019

    2019

  2. [2]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  3. [3]

    Xlnet: Generalized autoregressive pretraining for language understanding

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, 2019

    work page 2019

  4. [4]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020

    2020

  5. [5]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv, arXiv:2111.09543, 2023

    arXiv 2023

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott...

    work page 1901

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    work page 2023

  8. [8]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv, arXiv:2303.08774, 2024

    arXiv 2024

Show all 42 references
  1. [9]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, pages 24824–24837, 2022

    2022

  2. [10]

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learni...

    2023

  3. [11]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, pages 11809–11822, 2023

    2023

  4. [12]

    Multi-hop question answering

    Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. Multi-hop question answering. arXiv, arXiv:2204.09140, 2022

    2022 arXiv

  5. [13]

    Relational graph convolutional neural networks for multihop reasoning: A comparative study

    Ieva Stali¯unait˙e, Philip John Gorinski, and Ignacio Iacobacci. Relational graph convolutional neural networks for multihop reasoning: A comparative study. arXiv, arXiv:2210.06418, 2022

    2022 arXiv

  6. [14]

    Edge-aware graph neural network for multi-hop path reasoning over knowledge base

    Yanan Zhang, Li Jin, Xiaoyu Li, and Honqi Wang. Edge-aware graph neural network for multi-hop path reasoning over knowledge base. Computational Intelligence and Neuroscience, 2022(1):4734179, 2022

    2022

  7. [15]

    Knowledge-enhanced iterative instruction generation and reasoning for knowledge base question answering

    Haowei Du, Quzhe Huang, Chen Zhang, and Dongyan Zhao. Knowledge-enhanced iterative instruction generation and reasoning for knowledge base question answering. In Natural Language Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, Se...

    2022

  8. [16]

    Stepwise relation prediction with dynamic reasoning network for multi-hop knowledge graph question answering

    Hai Cui, Tao Peng, Tie Bao, Ridong Han, Jiayu Han, and Lu Liu. Stepwise relation prediction with dynamic reasoning network for multi-hop knowledge graph question answering. Applied Intelligence, 53(10):12340–12354, 2023

    2023

  9. [17]

    Multi-hop question answering using sparse graphs

    Nima Hemmati and Gholamreza Ghassem-Sani. Multi-hop question answering using sparse graphs. Engineering Applications of Artificial Intelligence, 126(D):107128, 2023

    2023

  10. [18]

    Answering complex open-domain questions with multi-hop dense retrieval

    Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. Answering complex open-domain questions with multi-hop dense retrieval. In International Conference on Learning Represent...

    2021

  11. [19]

    Multi-step reasoning over unstructured text with beam dense retrieval

    Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber, and Hal Daumé III. Multi-step reasoning over unstructured text with beam dense retrieval. arXiv, arXiv:2104.05883, 2021

    2021 arXiv

  12. [20]

    Triple-fact retriever: An explainable reasoning retrieval model for multi-hop qa problem

    Chengmin Wu, Enrui Hu, Ke Zhan, Lan Luo, Xinyu Zhang, Hao Jiang, Qirui Wang, Zhao Cao, Fan Yu, and Lei Chen. Triple-fact retriever: An explainable reasoning retrieval model for multi-hop qa problem. 2022 IEEE 38th International Conference on Data Engineering (ICDE), 53(10):120...

    2022

  13. [21]

    Human parity on commonsenseqa: Augmenting self-attention with external attention

    Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. Human parity on commonsenseqa: Augmenting self-attention with external attention. arXiv, arXiv:2112.03254, 2021

    2021 arXiv

  14. [22]

    Entailment as few-shot learner

    Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Haog Ma. Entailment as few-shot learner. arXiv, arXiv:2104.14690, 2021

    2021 arXiv

  15. [23]

    Multi-hop reading comprehension through question decomposition and rescoring

    Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading comprehension through question decomposition and rescoring. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097–6109, 2019

    2019

  16. [24]

    Measuring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, 2023

    2023

  17. [25]

    Tree-of-reasoning question decomposition for complex question answering with large language models

    Kun Zhang, Jiali Zeng, Fandong Meng, Yuanzhuo Wang, Shiqi Sun, Long Bai, Huawei Shen, and Jie Zhou. Tree-of-reasoning question decomposition for complex question answering with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19560...

    2024

  18. [26]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  19. [27]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V . Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. ...

    2023 arXiv

  20. [28]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022

    2022

  21. [29]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

    2022

  22. [30]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  23. [31]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, pages 22199–22213, 2022

    2022

  24. [32]

    Selection-inference: Exploiting large language models for interpretable logical reasoning

    Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  25. [33]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  26. [34]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proce...

    2018

  27. [35]

    Explaining answers with entailment trees

    Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. Explaining answers with entailment trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7358–7370, 2021

    2021

  28. [36]

    Qasc: A dataset for question answering via sentence composition

    Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8082–8090, 2020

    2020

  29. [37]

    Towards ai-complete question answering: A set of prerequisite toy tasks.arXiv, arXiv:1502.05698, 2015

    Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks.arXiv, arXiv:1502.05698, 2015

    2015 arXiv

  30. [38]

    Cognitive neuroscience of human counterfactual reasoning

    Nicole Van Hoeck, Patrick D Watson, and Aron K Barbey. Cognitive neuroscience of human counterfactual reasoning. Frontiers in human neuroscience, 9:420, 2015

    2015

  31. [39]

    Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, 2023

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, 2023

    2023

  32. [40]

    A study of automatically acquiring explanatory inference patterns from corpora of explanations: Lessons from elementary science exams

    Peter Jansen. A study of automatically acquiring explanatory inference patterns from corpora of explanations: Lessons from elementary science exams. In AKBC @ NIPS, 2017

    2017

  33. [41]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, page...

    2022

  34. [2010]

    Therefore, the answer is 247,597. 11 Q: What basketball player elected to the National Collegiate Basketball Hall of Fame released music through Rendezvous Music? A: Kirk Whalum, Jonathan Butler, Kyle Eastwood and Wayman Tisdale released music through Rendezvous Music. And Way...

    2011

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.