Pith. sign in

REVIEW 5 major objections 5 minor 38 references

Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DeepSeek-R1 consistently outscores GPT-4o and DeepSeek-V3 on family-tree and graph relational reasoning at moderate sizes, while all models collapse at n=40 as token limits truncate reasoning and outputs.

desk verdict Useful narrow empirical comparison, but the paper's headline claim overstates its own Table I: at n=40 Shortest(x,y), DeepSeek-V3 beats DeepSeek-R1, so 'consistently highest' is not supported. read the letter →

arxiv 2506.23128 v1 pith:CER27PF5 submitted 2025-06-29 cs.AI

Chi Chiu So , Yueyue Sun , Jun-Min Wang , Siu Pang Yung , Anthony Wai Keung Loh , Chun Pong Chau This is my paper
classification cs.AI
keywords largelanguagemodelsrelationalreasoningDeepSeek-R1chain-of-thoughtfamilytreegraphzero-shotpromptingF1-score
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how close large language models are to genuine relational reasoning, and answers with a controlled comparison on two zero-shot benchmark families: family-tree relations and general graph properties. It claims that DeepSeek-R1, whose architecture produces long chain-of-thought traces, consistently beats DeepSeek-V3 and GPT-4o on these tasks at problem sizes n=10 and n=20, with F1 scores above 0.9 on several relations while its competitors fall near zero. It also claims that all three models collapse at n=40, and that the main cause is not logical failure but token-length limits: reasoning gets truncated and output matrices come back incomplete, which the scoring rule counts as zero. A close reading of DeepSeek-R1's traces shows real summarization, abstraction, verification, and mid-stream planning, but also incoherent or after-the-fact steps, so the paper argues high scores should not be read as proof of sound internal logic. This matters because relational reasoning is a deeper faculty than pattern matching on standard benchmarks, and this is one of the first direct measures of whether 'deep reasoning' models actually generalize there.

What carries the argument

The machinery is a pair of generated benchmarks built from Boolean matrices. Family-tree data starts with four basic predicates (IsFather, IsMother, IsSon, IsDaughter) and derives four target relations (HasSister, IsGrandson, IsAunt, IsPaternalGreatAunt) by the formal Boolean formulas in Section III; graph data starts with a directed HasEdge adjacency matrix and targets Connectivity and Shortest. Each instance becomes a shuffled natural-language prompt asking for a JSON matrix, and the paper scores the returned matrix with standard or macro-averaged F1 against the Boolean ground truth, assigning F1 0 to invalid JSON or wrong matrix shapes. This machinery lets the authors scale relational depth by n and compare models in a zero-shot setting without in-context examples.

What would settle it

Independently reimplement the generation and deduction formulas from Section IV-B, regenerate the 100 instances per task at n=10 and n=20, and recompute F1 scores from the model outputs against those matrices; if the ground-truth labels differ from the authors' or DeepSeek-R1 no longer leads the ranking, the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that DeepSeek-R1 is the only tested model that can carry out multi-step relational deduction from raw facts: at n=10 it reaches F1 0.803 on HasSister, 0.778 on IsGrandson, 0.980 on IsAunt, and 0.969 on Shortest, while GPT-4o and DeepSeek-V3 usually stay below 0.6 and often at 0.000 on the same tasks. The advantage persists at n=20 for the easier family relations but thins on graph problems, and at n=40 every model essentially fails, with all three at 0.000 on IsGrandson, IsAunt, and IsPaternalGreatAunt and graph scores at or near zero. The paper attributes the leader's edge to its long chain-of-thought style acquired through reinforcement learning, and attributes the universal collapse to token limits rather than to missing relational concepts, because failed runs show truncated reasoning and malformed JSON matrices. It also reports that the winner's reasoning traces mix structured planning and verification with mid-stream plan formation and incoherent stretches, which it takes as evidence that visible coherence can conceal unsound inference dynamics.

Load-bearing premise

Every F1 comparison depends on the Boolean-matrix generator and the deduction formulas in Section IV-B producing correct ground truth, but the preprint promises code without releasing it, so a bug in those formulas would silently invalidate the reported scores.

Editorial extensions

If this is right

  • Deep-reasoning architecture translates into a measurable relational-reasoning advantage at moderate scale: DeepSeek-R1's long-CoT model beats non-CoT models on almost every family-tree and graph task at n=10 and n=20.
  • Token budget, not relational concept knowledge, is the main barrier to scaling: incomplete and malformed outputs, scored as zero, explain much of the n=40 collapse.
  • The benchmark's six problems and Boolean ground-truth generation can serve as a reusable zero-shot probe for other LLMs claiming deep reasoning ability.
  • High F1 scores and visibly logical traces do not guarantee sound reasoning: the paper documents mid-stream planning and incoherent steps inside the winner's own chain of thought.
  • Problem complexity itself matters: the deepest relation tested, IsPaternalGreatAunt, holds DeepSeek-R1 to F1 near 0.37-0.39 even at n=10 and n=20, where simpler relations reach 0.9.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The scoring rule (any malformed or incomplete output equals F1 0) mixes two failure modes; a follow-up that distinguishes parse failures from completed-but-wrong answers would separate token exhaustion from actual reasoning errors.
  • If token limits are indeed the bottleneck, a testable prediction follows now: giving the same n=40 prompts a much larger output budget, or requiring a more compact output format, should restore nonzero F1 scores on IsGrandson and IsAunt.
  • Because the generated facts are shuffled and the target relations are fixed Boolean formulas, this benchmark tests execution of a rule set rather than discovery of relational semantics; varying the proof depth of each instance could map F1 against required reasoning steps.
  • The observed 'plan appears midway' behavior invites a controlled intervention: explicitly prompting for an upfront plan before the solution should improve accuracy if planning is causal, and should not if the plan is post-hoc narration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper evaluates three LLMs (DeepSeek-R1, DeepSeek-V3, GPT-4o) on two families of relational reasoning benchmarks: family-tree relations (HasSister, IsGrandson, IsAunt, IsPaternalGreatAunt) and general graph reasoning (Connectivity, Shortest). Datasets of 100 instances are generated for sizes n=10, 20, and 40, prompts are given zero-shot in JSON-request format, and performance is measured using F1-scores against ground-truth Boolean matrices. The paper claims DeepSeek-R1 consistently achieves the highest F1-scores, that all models degrade at n=40 largely due to token limits, and that qualitative analysis of R1's chain-of-thought reveals planning and verification behaviors. A public code repository is promised but not yet available.

Significance. If the results hold, the paper would provide a useful empirical comparison of deep-reasoning LLMs on structured relational tasks that are harder than simple arithmetic or symbolic benchmarks. The benchmark design is reasonable: target relations are defined by explicit logical formulas, prompts are zero-shot, and F1 is an appropriate primary metric for class-imbalanced matrices. The qualitative analysis of DeepSeek-R1's long chain-of-thought is a positive feature, illustrating planning and verification patterns that are rarely documented in this detail. However, the contribution is currently weakened by an internal inconsistency in the headline claim, missing statistical support, and the absence of public code/data to verify the ground-truth generation.

major comments (5)
  1. [Table I, Section V-A] The central claim that DeepSeek-R1 'consistently achieves the highest F1-scores across multiple tasks and problem sizes' is directly contradicted by Table I: at n=40 for Shortest(x,y), DeepSeek-V3 scores 0.238 while DeepSeek-R1 scores 0.201. Section V-A states that at n=40 all three models fail completely only on IsGrandson, IsAunt, and IsPaternalGreatAunt, but this omits the Shortest(x,y) exception. The claim should be corrected to specify that R1 is highest at n=10 and n=20 on most tasks, with explicit exceptions at n=40, or the abstract and Section V-A should be revised to match the reported data.
  2. [Section IV-C, Table I] No confidence intervals, standard deviations, or significance tests are reported for any F1-scores, although the paper draws comparative conclusions such as 'DeepSeek-R1 consistently outperforms GPT-4o and DeepSeek-V3.' With 100 runs per condition, differences such as HasSister at n=10 (0.516 vs 0.542) or Connectivity at n=40 (0.070 vs 0.103) may well be within run-to-run variability. The authors should provide error bars or statistical tests (e.g., bootstrap or paired tests) to support the claimed advantage, especially for cells where the margin is small.
  3. [Section IV-B] The ground truth is not verifiable from the manuscript: the claim that 'All generated data are verified to ensure logical consistency, particularly for family relationships' is not accompanied by the verification algorithm, the generated datasets, or working code. The repository at https://github.com/kelvinhkcs/Deep-Relational-Reasoning is promised but not available in the preprint. Because every reported F1-score depends on the correctness of the Boolean-matrix generator and the deduction formulas, the authors should release the data-generation and verification code, or at least include a precise pseudocode description, before the empirical claims can be fully assessed.
  4. [Section V-C] The paper attributes performance degradation at n=40 to 'intrinsic token limits' and to truncated reasoning, but no evidence is provided: there are no measurements of token usage, no statement of the max_tokens setting used for each model, and no systematic detection of truncation in the outputs. The explanation should be supported by comparing output token counts against model limits, or by showing that failed runs are indeed truncated rather than merely incorrect. Without this, the token-limit conclusion is an unsupported causal claim.
  5. [Section IV-D] Assigning an F1-score of 0 to any invalid JSON or incorrectly shaped matrix conflates a formatting failure with a reasoning failure. This choice can substantially affect the reported scores, particularly for weaker models that may return malformed outputs. The paper should report format-validity rates separately from reasoning accuracy, or justify why a format failure should be treated identically to a wrong answer in a relational-reasoning evaluation.
minor comments (5)
  1. [Section IV-A] DeepSeek-V3 is described as a 'smaller and more lightweight version of DeepSeek-R1'; in fact, DeepSeek-V3 and DeepSeek-R1 share a 671B-parameter MoE architecture, so this description is inaccurate and should be corrected.
  2. [Section III-A] The definitions of IsAunt and IsPaternalGreatAunt include conditions such as ¬IsMother(x,y) and rely on an implicit gender assumption for 'aunt.' The paper should explicitly state that all relations are evaluated with the given gender assignments and clarify whether spouse relationships are represented or assumed.
  3. [Section IV-B] The illustrative Boolean matrices and the natural-language prompt example in Section IV-B are not shown to be mutually consistent; for instance, the prompt lists facts such as 'P0 is P4's father' and 'P1 is P9's mother,' but the reader cannot verify that the displayed father/mother matrices encode exactly these facts. A short consistency check in the text would help.
  4. [References] Reference [33] and Reference [38] appear to be the same paper ('Neural Logic Machines' by Dong et al.), but they are listed separately; please consolidate the duplicate citation.
  5. [Throughout] There are several typographical inconsistencies, including 'DeepSeek–R1' with an en dash in Section V-D and 'textttConnectivity' in Section V-B; these should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the benchmark evaluation is external, with ground-truth matrices generated independently of model outputs.

full rationale

The paper's central claim is an empirical comparison: DeepSeek-R1 versus DeepSeek-V3 and GPT-4o on family tree and graph reasoning tasks. The ground-truth labels are produced by explicit Boolean-matrix generation and deduction formulas described in Section IV-B, and model outputs are scored against these precomputed matrices using F1-score in Section IV-D. No fitted parameter is used to define the result, and no conclusion is derived from the authors' prior claims or from a self-citation chain. The only self-references are the promise of a public repository and the use of benchmark ideas adapted from Neural Logic Machines [38], which is an external source and does not load-bear the comparison. The paper's internal inconsistency in Table I at n=40 for Shortest(x,y), where DeepSeek-V3 scores 0.238 against DeepSeek-R1's 0.201 despite the abstract claiming DeepSeek-R1 'consistently achieves the highest F1-scores,' is a correctness or reporting concern, not a circularity concern, because the reported numbers are still compared against an independent ground truth rather than being the paper's own output folded back into itself. The benchmark construction, model prompting, and evaluation protocol are all external to the conclusion, so the derivation chain is self-contained and no circular reduction exists.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central comparison rests on the correctness of the generated ground truth, the fairness of the zero-shot JSON evaluation protocol, and the comparability of API runs; none of these are independently verifiable from the preprint because the code and data are not released. There are no fitted numerical parameters in the paper's own analysis.

assumptions (4)
  • domain assumption The Boolean matrix generator produces logically consistent family trees and directed graphs, and the deduction formulas in Section III-A and III-B correctly compute ground truth.
    Section IV-B says data are verified to ensure logical consistency, but no verification algorithm or code is provided; the repository is not yet public.
  • ad hoc to paper Assigning F1=0 to invalid JSON or wrong matrix shape is a fair measure of reasoning failure.
    Section IV-D assigns zero to any malformed response, which conflates output-format compliance and token truncation with reasoning failure; this affects long Chain-of-Thought outputs more.
  • domain assumption API outputs were sampled under conditions that make 100 runs comparable across models.
    Section IV-C does not report temperature, max_tokens, API version, or sampling seed; run-to-run variance is not measured.
  • standard math Standard F1 and macro-averaged F1 are appropriate metrics for the binary and multiclass tasks.
    The metric definitions in Section IV-D follow standard practice for classification under class imbalance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons." pith.science (2026). https://pith.science/paper/CER27PF5

@misc{pith2026250623128,
  author       = {Pith},
  title        = {Pith review of: Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CER27PF5}},
  note         = {Machine review of arXiv:2506.23128}
}
read the original abstract

How far are Large Language Models (LLMs) in performing deep relational reasoning? In this paper, we evaluate and compare the reasoning capabilities of three cutting-edge LLMs, namely, DeepSeek-R1, DeepSeek-V3 and GPT-4o, through a suite of carefully designed benchmark tasks in family tree and general graph reasoning. Our experiments reveal that DeepSeek-R1 consistently achieves the highest F1-scores across multiple tasks and problem sizes, demonstrating strong aptitude in logical deduction and relational inference. However, all evaluated models, including DeepSeek-R1, struggle significantly as problem complexity increases, largely due to token length limitations and incomplete output structures. A detailed analysis of DeepSeek-R1's long Chain-of-Thought responses uncovers its unique planning and verification strategies, but also highlights instances of incoherent or incomplete reasoning, calling attention to the need for deeper scrutiny into LLMs' internal inference dynamics. We further discuss key directions for future work, including the role of multimodal reasoning and the systematic examination of reasoning failures. Our findings provide both empirical insights and theoretical implications for advancing LLMs' reasoning abilities, particularly in tasks that demand structured, multi-step logical inference. Our code repository will be publicly available at https://github.com/kelvinhkcs/Deep-Relational-Reasoning.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 18 canonical work pages

  1. [1]

    Can llms understand time series anomalies?

    Z. Zhou and R. Yu, “Can llms understand time series anomalies?” arXiv preprint arXiv:2410.05440, 2024

    arXiv 2024

  2. [2]

    Time-llm: Time series forecasting by re- programming large language models,

    M. Jin, S. Wang, L. Ma, Z. Chu, J. Y . Zhang, X. Shi, P.-Y . Chen, Y . Liang, Y .-F. Li, S. Panet al., “Time-llm: Time series forecasting by re- programming large language models,” arXiv preprint arXiv:2310.01728, 2023

    arXiv 2023

  3. [3]

    Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning,

    Q. Wen, J. Liang, C. Sierra, R. Luckin, R. Tong, Z. Liu, P. Cui, and J. Tang, “Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 6743–6744

    work page 2024

  4. [4]

    Software engineering education must adapt and evolve for an llm environment,

    V . D. Kirova, C. S. Ku, J. R. Laracy, and T. J. Marlowe, “Software engineering education must adapt and evolve for an llm environment,” in Proceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, 2024, pp. 666–672

    work page 2024

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    arXiv 2025

  6. [6]

    How far are we from agi: Are llms all we need?

    T. Feng, C. Jin, J. Liu, K. Zhu, H. Tu, Z. Cheng, G. Lin, and J. You, “How far are we from agi: Are llms all we need?” arXiv preprint arXiv:2405.10313, 2024

    arXiv 2024

  7. [7]

    Llm/gpt generative ai and artificial general intelligence (agi): The next frontier,

    A. F. Mohammad, B. Clark, R. Agarwal, and S. Summers, “Llm/gpt generative ai and artificial general intelligence (agi): The next frontier,” in 2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE) . IEEE, 2023, pp. 413–417

    work page 2023

  8. [8]

    A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges,

    Y . Yan, J. Su, J. He, F. Fu, X. Zheng, Y . Lyu, K. Wang, S. Wang, Q. Wen, and X. Hu, “A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges,” arXiv preprint arXiv:2412.11936 , 2024

    arXiv 2024

Show all 38 references
  1. [9]

    Mathodyssey: Bench- marking mathematical problem-solving skills in large language models using odyssey math data,

    M. Fang, X. Wan, F. Lu, F. Xing, and K. Zou, “Mathodyssey: Bench- marking mathematical problem-solving skills in large language models using odyssey math data,” arXiv preprint arXiv:2406.18321 , 2024

    2024 arXiv

  2. [10]

    Logicgame: Benchmarking rule-based reasoning abilities of large language models,

    J. Gui, Y . Liu, J. Cheng, X. Gu, X. Liu, H. Wang, Y . Dong, J. Tang, and M. Huang, “Logicgame: Benchmarking rule-based reasoning abilities of large language models,” arXiv preprint arXiv:2408.15778 , 2024

    2024 arXiv

  3. [11]

    Logicvista: Multimodal llm logical reasoning benchmark in visual contexts,

    Y . Xiao, E. Sun, T. Liu, and W. Wang, “Logicvista: Multimodal llm logical reasoning benchmark in visual contexts,” arXiv preprint arXiv:2407.04973, 2024

    2024 arXiv

  4. [12]

    Scaling relationship on learning mathematical reasoning with large language models,

    Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou, “Scaling relationship on learning mathematical reasoning with large language models,” arXiv preprint arXiv:2308.01825 , 2023

    2023 arXiv

  5. [13]

    Solving math word problems con- cerning systems of equations with gpt models,

    M. Zong and B. Krishnamachari, “Solving math word problems con- cerning systems of equations with gpt models,” Machine Learning with Applications, vol. 14, p. 100506, 2023

    2023

  6. [14]

    The mathematics of deepseek-r1: Theoretical foundations and comparative analysis,

    N. I. Alonso et al. , “The mathematics of deepseek-r1: Theoretical foundations and comparative analysis,” The Mathematics of DeepSeek- R1: Theoretical Foundations and Comparative Analysis (January 22, 2025), 2025

    2025

  7. [15]

    Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change,

    K. Valmeekam, M. Marquez, A. Olmo, S. Sreedharan, and S. Kambham- pati, “Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change,” Advances in Neural Information Processing Systems , vol. 36, pp. 38 975–38 987, 2023

    2023

  8. [16]

    Neurocognitive development of rela- tional reasoning,

    E. A. Crone, C. Wendelken, L. Van Leijenhorst, R. D. Honomichl, K. Christoff, and S. A. Bunge, “Neurocognitive development of rela- tional reasoning,” Developmental science , vol. 12, no. 1, pp. 55–66, 2009

    2009

  9. [17]

    Processing capacity defined by relational complexity: Implications for comparative, devel- opmental, and cognitive psychology,

    G. S. Halford, W. H. Wilson, and S. Phillips, “Processing capacity defined by relational complexity: Implications for comparative, devel- opmental, and cognitive psychology,” Behavioral and brain sciences , vol. 21, no. 6, pp. 803–831, 1998

    1998

  10. [18]

    Llms for relational reasoning: How far are we?

    Z. Li, Y . Cao, X. Xu, J. Jiang, X. Liu, Y . S. Teo, S.-W. Lin, and Y . Liu, “Llms for relational reasoning: How far are we?” in Proceedings of the 1st International Workshop on Large Language Models for Code , 2024, pp. 119–126

    2024

  11. [19]

    How good are gpt models at machine translation? a comprehensive evaluation,

    A. Hendy, M. Abdelrehim, A. Sharaf, V . Raunak, M. Gabr, H. Mat- sushita, Y . J. Kim, M. Afify, and H. H. Awadalla, “How good are gpt models at machine translation? a comprehensive evaluation,” arXiv preprint arXiv:2302.09210, 2023

    2023 arXiv

  12. [20]

    Chatmusician: Understanding and generating music intrinsically with llm,

    R. Yuan, H. Lin, Y . Wang, Z. Tian, S. Wu, T. Shen, G. Zhang, Y . Wu, C. Liu, Z. Zhou et al. , “Chatmusician: Understanding and generating music intrinsically with llm,” arXiv preprint arXiv:2402.16153 , 2024

    2024 arXiv

  13. [21]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  14. [22]

    Multimodal prompt- ing with missing modalities for visual recognition,

    Y .-L. Lee, Y .-H. Tsai, W.-C. Chiu, and C.-Y . Lee, “Multimodal prompt- ing with missing modalities for visual recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14 943–14 952

    2023

  15. [23]

    Next-gpt: Any-to-any multimodal llm,

    S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, “Next-gpt: Any-to-any multimodal llm,” in Forty-first International Conference on Machine Learning, 2024

    2024

  16. [24]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    2021

  17. [25]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  18. [26]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  19. [27]

    Chain-of-verification reduces hallucination in large language models,

    S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston, “Chain-of-verification reduces hallucination in large language models,” arXiv preprint arXiv:2309.11495 , 2023

    2023 arXiv

  20. [28]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in neural information processing systems , vol. 36, pp. 11 809–11 822, 2023

    2023

  21. [29]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  22. [30]

    Prompt engineering for zero- shot and few-shot defect detection and classification using a visual- language pretrained model,

    G. Yong, K. Jeon, D. Gil, and G. Lee, “Prompt engineering for zero- shot and few-shot defect detection and classification using a visual- language pretrained model,” Computer-Aided Civil and Infrastructure Engineering, vol. 38, no. 11, pp. 1536–1554, 2023

    2023

  23. [31]

    Vita-clip: Video and text adaptive clip via multimodal prompting,

    S. T. Wasim, M. Naseer, S. Khan, F. S. Khan, and M. Shah, “Vita-clip: Video and text adaptive clip via multimodal prompting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2023, pp. 23 034–23 044

    2023

  24. [32]

    Worldgpt: Empowering llm as multimodal world model,

    Z. Ge, H. Huang, M. Zhou, J. Li, G. Wang, S. Tang, and Y . Zhuang, “Worldgpt: Empowering llm as multimodal world model,” in Proceed- ings of the 32nd ACM International Conference on Multimedia , 2024, pp. 7346–7355

    2024

  25. [33]

    Neural logic machines,

    H. Dong, J. Mao, T. Lin, C. Wang, L. Li, and D. Zhou, “Neural logic machines,” in International Conference on Learning Representations

  26. [34]

    Differentiable logic machines,

    M. Zimmer, X. Feng, C. Glanois, Z. JIANG, J. Zhang, P. Weng, D. Li, J. HAO, and W. Liu, “Differentiable logic machines,” Transactions on Machine Learning Research, 2023. [Online]. Available: https://openreview.net/forum?id=mXfkKtu5JA

    2023

  27. [35]

    Learning explanatory rules from noisy data,

    R. Evans and E. Grefenstette, “Learning explanatory rules from noisy data,” Journal of Artificial Intelligence Research, vol. 61, pp. 1–64, 2018

    2018

  28. [36]

    Neural probabilistic logic programming in deepproblog,

    R. Manhaeve, S. Duman ˇci´c, A. Kimmig, T. Demeester, and L. De Raedt, “Neural probabilistic logic programming in deepproblog,” Artificial Intelligence, vol. 298, p. 103504, 2021

    2021

  29. [37]

    Logical reasoning in large language models: A survey,

    H. Liu, Z. Fu, M. Ding, R. Ning, C. Zhang, X. Liu, and Y . Zhang, “Logical reasoning in large language models: A survey,” arXiv preprint arXiv:2502.09100, 2025

    2025 arXiv

  30. [38]

    Neural logic machines,

    H. Dong, J. Mao, T. Lin, C. Wang, L. Li, and D. Zhou, “Neural logic machines,” arXiv preprint arXiv:1904.11694 , 2019

    1904 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.