Pith. sign in

REVIEW 4 major objections 7 minor 38 references

LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LogiPlan is a new benchmark that dynamically varies the size and depth of relational graphs, and its evaluation of current LLMs shows that they handle simple relational planning but fail as logical depth grows.

desk verdict A solid benchmark artifact whose headline scaling claim is confounded by prompt length; worth refereeing, but requires code release and a control condition. read the letter →

arxiv 2506.10527 v1 pith:VNV2EEVA submitted 2025-06-12 cs.AI cs.PF

Yanan Cai , Ahmed Salem , Besmira Nushi , Mark Russinovich This is my paper
classification cs.AIcs.PF
keywords LogiPlanlogicalplanningrelationalreasoningLLMbenchmarkcycledetectiondirectedgraphsself-correctionmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LogiPlan is a benchmark that tests whether large language models can generate and reason about directed graphs of comparative relations, with difficulty set by the number of objects, the number of relations, and the minimum depth of the relational chains. The paper's central claim is that this controlled variation yields a fine-grained picture of logical planning ability, and that current models solve simple relational instances but degrade sharply on larger, deeper configurations. Across the three tasks—plan generation, consistency detection, and comparison questions—consistency detection is the hardest, with even the strongest models dropping to modest F1 scores as graph size grows. The paper also reports that prompting models with 'Are you sure?' produces unreliable self-correction: some improve, while instruction-tuned models often flip correct answers to 'Unknown'.

What carries the argument

The central object is a directed graph of comparative statements over labeled objects, written with $>$ and $<$ relations and generated under explicit control of three complexity parameters: the number of objects, the number of relations, and the minimum depth of relational chains (for Consistency Detection, the minimum cycle length; for Comparison Question, the number of inference steps needed to derive the queried relation). The benchmark converts each graph into a randomized textual prompt—arbitrary object labels, mixed forward and backward relation phrasing, and shuffled statement order—so that a model must reconstruct the underlying relation network instead of matching surface patterns. Three tasks exercise the pipeline: Plan Generation asks the model to produce a consistent relation list meeting size constraints; Consistency Detection asks it to identify all cycles or contradictions; Comparison Question asks it to classify a queried relation as True, False, or Unknown. A self-correction protocol then re-prompts the model with 'Are you sure?' and measures whether the revision improves or degrades the original answer.

What would settle it

Generate two sets of Comparison Question instances with identical relation counts and prompt lengths, one set where the queried relation requires a long chain of inferences and one set where the same relation appears directly; if accuracy is equal across the two, LogiPlan's difficulty gradient is a prompt-length effect rather than evidence about the depth of logical planning required.

Watch

Extended reading notes

Core claim

The discovery is that logical relational reasoning in LLMs should be evaluated as a complete pipeline—constructing a relational graph, checking it for cycles and contradictions, and answering queries about it—and that each stage has a different difficulty profile. Across nine state-of-the-art instruction-tuned and reasoning-enhanced models, the paper reports that the reasoning models can discover a simple total-ordering strategy ($A > B > C > \cdots$) and use it to reach near-perfect Plan Generation accuracy, yet the same models struggle on Consistency Detection as the numbers of objects, relations, and the minimum cycle length increase. The strongest reasoning models lead on that task but still fall to low F1 scores on the largest configurations. For Comparison Question, the gaps narrow, and the 'Unknown' answer category becomes a distinctive failure mode, especially for instruction-based models. The paper's conclusion is that current LLMs have partial, uneven relational planning ability, and that a benchmark with dynamically varied complexity exposes exactly where that ability breaks down.

Load-bearing premise

The load-bearing premise is that the three task parameters (objects, relations, chain depth) isolate logical difficulty, but LogiPlan does not control for the longer prompts that accompany those parameters, and the paper's own cited evidence shows longer contexts hurt LLM reasoning.

Editorial extensions

If this is right

  • Benchmark reporting will need per-configuration scores rather than a single average, because model rankings and failure thresholds shift across object counts, relation counts, and chain depths.
  • The near-perfect Plan Generation scores of reasoning models should be treated cautiously, since the paper shows they often rest on a discovered total-order shortcut rather than on general graph construction.
  • Applications that use LLMs to audit relational structures, such as network topologies or knowledge bases, should not assume reliable cycle detection at scale.
  • Self-correction prompts cannot be assumed to help: the same 'Are you sure?' query improves some models and degrades others, with instruction-based models especially prone to converting correct answers to 'Unknown'.
  • Because LogiPlan generates fresh instances from the same parameterized generator, benchmarks of this type can be re-sampled to separate memorized performance from genuine reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: if a length-matched control shows the same performance collapse, LogiPlan's difficulty gradient would be better described as context-length sensitivity than as logical planning depth.
  • Adding constraints that forbid the total-order solution (for example, requiring a specified topology such as a DAG with a missing edge) would test whether reasoning models can plan rather than rediscover a shortcut.
  • The 'Unknown' response could be repurposed as a calibration probe: comparing how often each model says 'Unknown' when it should versus when it should not would quantify over- and under-confidence as a function of graph size.
  • A natural testable extension is to sample two sets of LogiPlan instances—one matching publicly released graphs and one freshly generated—and compare model accuracy, which would directly measure benchmark memorization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces LogiPlan, a synthetic benchmark for evaluating LLMs on three relational planning and reasoning tasks—Plan Generation, Consistency Detection, and Comparison Question—along with a self-correction protocol. The benchmark varies the number of objects, relations, and the minimum cycle length (or inference depth) to create difficulty levels, and the authors evaluate nine state-of-the-art models. They report that reasoning models outperform instruction-tuned models, especially on Plan Generation, but that all models degrade as graph size grows and as cycles/depths increase. The appendix provides data-group lists, prompt templates, and per-group score tables.

Significance. If its difficulty manipulations are validated, LogiPlan would be a useful and easily extensible benchmark: the data groups are clearly enumerated, prompts are provided, and the evaluation covers multiple runs for most models, with a self-correction probe that goes beyond single-shot accuracy. The paper also gives credit for reporting per-group performance rather than only aggregate scores. However, the central scaling claim—that performance drops because tasks require deeper logical planning—is currently confounded with prompt length, and several evaluation-pipeline details are missing. Because the benchmark's value as a diagnostic instrument depends on isolating logical complexity from input-length effects, these issues are load-bearing and need to be fixed before the results can be interpreted as claimed.

major comments (4)
  1. [§2.1.2, §2.1.3, Figures 2b and 4b] The claim that performance degradation reflects deeper logical planning is confounded with prompt length. In Consistency Detection and Comparison Question, the prompt is the rendered relation list, so increasing the number of objects and relations increases the input length roughly linearly. The paper's own related work (§4) cites Levy et al. (2024) and Balachandran et al. (2024) showing that LLM reasoning accuracy drops with context length, yet no control condition separates input length from logical complexity. The sub-conditions that hold V and E fixed while varying minimum cycle length or inference depth are cleaner, but the headline scaling conclusions (e.g., the abstract's claim that models 'struggle with more complex configurations requiring deeper logical planning') rely on the confounded comparisons. Please add a matched-length control (e.g., filler relations that do not affect the answer), or restrict the headline claims to the fixed-V, fixed-E sub-conditions and re-analyze the object/relation scaling accordingly.
  2. [§3.1, Figure 1] The overall Plan Generation Accuracy is not defined as a function of the four submetrics (Consistency, Duplicates, Object Numbers, Relation Numbers). It is unclear whether Accuracy is the fraction of outputs that simultaneously satisfy all four checks, a product of sub-scores, an average, or a different aggregation; consequently, the values in Figure 1a cannot be reproduced or compared across models. The paper should state the exact decision rule and whether partial credit is assigned when only some sub-checks pass.
  3. [§3.3.2, Appendix A.2] The paper does not describe how free-form model outputs for Consistency Detection are parsed into the cycle list that is compared with ground truth. The prompt asks models to emit a 'Yes/No' decision and an arbitrary list of cycles in unconstrained text; the matching criterion (exact cycle match, cyclic-shift invariance, handling of repeated nodes, whether sub-cycles receive partial credit, and how extra reported cycles are treated) can materially change the F1 scores in Figure 2 and Figure 8. Please provide the parsing algorithm and the matching rule, or state explicitly that outputs were manually evaluated.
  4. [§2.1.2 Data Generation] The graph generation procedure is only described as using NetworkX with 'complete connectivity' and enforced minimum cycle lengths. The actual algorithm for sampling a DAG with exactly V nodes and E edges, or a cyclic graph with a specified minimum cycle length but a randomized number of cycles, is not given. This matters because the distribution over graphs determines both the ground-truth enumeration (e.g., the average of 4.37 cycles) and the difficulty ordering that the benchmark claims to control. Please provide pseudocode or a formal specification, including how acyclic graphs with dense edge counts are generated and how minimum cycle lengths are enforced.
minor comments (7)
  1. [Appendix A.2 heading] The heading uses 'minium' instead of 'minimum.'
  2. [Abstract vs. §2.1.2/§2.1.3] The abstract refers to 'the minimum depth of relational chains,' while the tasks use 'minimum cycle length' and 'depth of comparison'; the terminology should be unified to avoid confusion.
  3. [§2.1.2 Data Generation] The phrase 'ensuring complete connectivity' is ambiguous: it should clarify whether the underlying undirected graph is connected, whether every pair of nodes has a directed path, or neither.
  4. [§3.2 and §3.3.1] Section 3.2 states that five runs are used 'unless explicitly mentioned otherwise,' but Section 3.3.1 reports three runs for Plan Generation; this should be stated at the first mention of Plan Generation.
  5. [Figures 3 and 5] The self-correction results are based on a single run for all models, as disclosed, but given the high variance of the underlying metrics, the main text should add a caveat that these results are preliminary and should be interpreted with caution.
  6. [Reproducibility Statement] The reproducibility statement refers to 'an open-source evaluation framework' without naming it; please provide a name, URL, or version so readers can identify the exact pipeline.
  7. [Figures 1b, 2b, 4b, and Appendix B] The x-axis labels and per-group heat-map entries are dense and difficult to read at print size; consider rotating labels, using larger fonts, or moving the detailed tables to an appendix with clearer formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

LogiPlan is an empirical benchmark whose instances are generated against ground truth; no derivation step reduces to fitted inputs or load-bearing self-citations, so there is no significant circularity.

full rationale

The paper's central claims are empirical measurements of LLM performance on generated relational-graph tasks. The benchmark instances are produced by a Python generator (Section 2.1) with explicit parameters (object count, relation count, minimum cycle length, and comparison depth), and model outputs are scored against ground-truth answers computed from the generated graphs. No equation in the paper defines a predicted quantity in terms of the fitted quantity; no parameter is fitted from a subset of data and then renamed as a prediction; and the accuracy/F1 metrics are evaluated directly against independently generated labels. The one self-citation in the paper (Balachandran et al., 2024, in the Related Work section) is used only to support the contextual observation that multi-hop reasoning accuracy drops with context length; it is not invoked to define LOGIPLAN's tasks, to justify the benchmark's validity, or to derive any result, so it is not load-bearing. The reader's note about a possible prompt-length confound (larger object/relation counts also lengthen the prompt) is a validity threat to the interpretation that performance drops are due to logical complexity rather than context length, but this is a confound or experimental-design concern, not a circularity: the benchmark's ground truth and scoring remain external to the models' outputs. The paper itself acknowledges the context-length effect in Section 4, but does not use that observation as an input to the benchmark construction or evaluation. For these reasons, the derivation chain is self-contained rather than circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims depend on assumptions about construct validity, pattern-randomization effectiveness, prompt-length confounds, and the correctness of the un-specified data generation and output parsing. No fitted free parameters or invented physical entities are introduced.

assumptions (5)
  • domain assumption LLM performance on LOGIPLAN tasks is a valid proxy for logical planning and relational reasoning ability.
    The paper assumes that success on generated directed-graph tasks reflects the reasoning skills needed in real applications like supply chains and network configuration. This is a construct-validity assumption that is not empirically validated.
  • domain assumption Randomization of object labels, relation notation, and statement order prevents models from exploiting superficial patterns.
    The paper states the randomization is deliberate so that models must reconstruct the relation network logically, but this is not empirically validated. The claim that models cannot use surface patterns is an assumption.
  • domain assumption Prompt length does not confound the difficulty scaling.
    Larger object and relation counts produce longer prompts; prior work cited by the paper shows context length alone degrades reasoning accuracy. The paper does not control for this, so the difficulty scaling is confounded.
  • domain assumption The graph generation algorithm using NetworkX correctly enforces the specified numbers of objects, relations, and minimum cycle lengths.
    The paper does not specify the exact algorithm or validate that generated graphs match the stated parameters, such as complete connectivity and cycle length constraints.
  • standard math Standard graph algorithm complexities (Johnson for cycle detection, BFS/DFS for reachability) correctly characterize task difficulty.
    These are textbook results used to motivate task difficulty. They are correct but their use as a proxy for LLM difficulty is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs." pith.science (2026). https://pith.science/paper/VNV2EEVA

@misc{pith2026250610527,
  author       = {Pith},
  title        = {Pith review of: LogiPlan: A Structured Benchmark for Logical Planning and Relational Reasoning in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNV2EEVA}},
  note         = {Machine review of arXiv:2506.10527}
}
read the original abstract

We introduce LogiPlan, a novel benchmark designed to evaluate the capabilities of large language models (LLMs) in logical planning and reasoning over complex relational structures. Logical relational reasoning is important for applications that may rely on LLMs to generate and query structured graphs of relations such as network infrastructure, knowledge bases, or business process schema. Our framework allows for dynamic variation of task complexity by controlling the number of objects, relations, and the minimum depth of relational chains, providing a fine-grained assessment of model performance across difficulty levels. LogiPlan encompasses three complementary tasks: (1) Plan Generation, where models must construct valid directed relational graphs meeting specified structural constraints; (2) Consistency Detection, testing models' ability to identify inconsistencies in relational structures; and (3) Comparison Question, evaluating models' capacity to determine the validity of queried relationships within a given graph. Additionally, we assess models' self-correction capabilities by prompting them to verify and refine their initial solutions. We evaluate state-of-the-art models including DeepSeek R1, Gemini 2.0 Pro, Gemini 2 Flash Thinking, GPT-4.5, GPT-4o, Llama 3.1 405B, O3-mini, O1, and Claude 3.7 Sonnet across these tasks, revealing significant performance gaps that correlate with model scale and architecture. Our analysis demonstrates that while recent reasoning-enhanced models show promising results on simpler instances, they struggle with more complex configurations requiring deeper logical planning.

Figures

Figures reproduced from arXiv: 2506.10527 by the authors.

Figure 1
Figure 1. Overview Accuracy of different models for the Plan Generation task. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Overview F1 Score of different models for the Consistency Detection task. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Self correction results of the different models for the Consistency Detection task. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Overview Accuracy of different models for the Comparison Question task. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Self correction results of the different models for the Comparison Question task. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: An example graph for the Consistency Detection task. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Plan Generation Relation Duplications in Model Output [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Detailed Average F1 Score of Consistency Detection Task - Continued [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Detailed Average Accuracy of Comparison Question Task [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Detailed average accuracy of Plan Generation task [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Detailed average accuracy of Plan Generation task - Continued [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Detailed average accuracy of Plan Generation task - Continued [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Detailed Average Accuracy of Plan Generation Task - Continued [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 14 canonical work pages

  1. [1]

    How far can transformers reason? the globality barrier and inductive scratchpad

    Emmanuel Abbe, Samy Bengio, Aryo Lotfi, Colin Sandon, and Omid Saremi. How far can transformers reason? the globality barrier and inductive scratchpad. Advances in Neural Information Processing Systems, 37: 0 27850--27895, 2024

    work page 2024

  2. [2]

    Graph of logic: Enhancing llm reasoning with graphs and symbolic logic

    Fatimah Alotaibi, Adithya Kulkarni, and Dawei Zhou. Graph of logic: Enhancing llm reasoning with graphs and symbolic logic. In 2024 IEEE International Conference on Big Data (BigData), pp.\ 5926--5935. IEEE, 2024

    work page 2024

  3. [3]

    Claude 3.7 sonnet

    Anthropic. Claude 3.7 sonnet. https://www.anthropic.com/news/claude-3-7-sonnet, 2025. Accessed: 2025-03-17

    work page 2025

  4. [4]

    Eureka: Evaluating and understanding large foundation models

    Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, and Safoora Yousefi. Eureka: Evaluating and understanding large foundation models. arXiv preprint arXiv:2409.10566, 2024

    arXiv 2024

  5. [5]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  6. [6]

    Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes

    Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes. arXiv preprint arXiv:2312.14890, 2023

    arXiv 2023

  7. [7]

    Magentic-one: A generalist multi-agent system for solving complex tasks

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.04468, 2024

    arXiv 2024

  8. [8]

    Gemini 2.0 pro experimental

    Google. Gemini 2.0 pro experimental. https://deepmind.google/technologies/gemini/pro/, 2025 a . Accessed: 2025-03-17

    work page 2025

Show all 38 references
  1. [9]

    Gemini 2.0 flash thinking

    Google. Gemini 2.0 flash thinking. https://deepmind.google/technologies/gemini/flash-thinking/, 2025 b . Accessed: 2025-03-17

    2025

  2. [10]

    Reinforced self-training (rest) for language modeling

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023

    2023 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  5. [13]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  6. [14]

    Finding all the elementary circuits of a directed graph

    Donald B Johnson. Finding all the elementary circuits of a directed graph. SIAM Journal on Computing, 4 0 (1): 0 77--84, 1975

    1975

  7. [15]

    Same task, more tokens: the impact of input length on the reasoning performance of large language models

    Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. ACL, 2024

    2024

  8. [16]

    Large language models for supply chain optimization

    Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language models for supply chain optimization. arXiv preprint arXiv:2307.03875, 2023

    2023 arXiv

  9. [17]

    Llms for relational reasoning: How far are we? In Proceedings of the 1st International Workshop on Large Language Models for Code, pp.\ 119--126, 2024

    Zhiming Li, Yushi Cao, Xiufeng Xu, Junzhe Jiang, Xu Liu, Yon Shin Teo, Shang-Wei Lin, and Yang Liu. Llms for relational reasoning: How far are we? In Proceedings of the 1st International Workshop on Large Language Models for Code, pp.\ 119--126, 2024

    2024

  10. [18]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  11. [19]

    Zebralogic: On the scaling limits of llms for logical reasoning

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.01100, 2025

    2025 arXiv

  12. [20]

    Evaluating cognitive maps and planning in large language models with cogeval

    Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36: 0 69736...

    2023

  13. [21]

    Foster, and Michael W

    Iulian Neamtiu, Jeffrey S. Foster, and Michael W. Hicks. Understanding source code evolution using abstract syntax tree matching. ACM SIGSOFT Software Engineering Notes, 30: 0 1 -- 5, 2005

    2005

  14. [22]

    Openai o3-mini system card

    OpenAI. Openai o3-mini system card. https://openai.com/index/o3-mini-system-card/, 2025. Accessed: 2025-03-17

    2025

  15. [23]

    Logicbench: Towards systematic evaluation of logical reasoning ability of large language models

    Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. ACL, 2024

    2024

  16. [24]

    Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models

    Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. arXiv preprint arXiv:2406.17169, 2024

    2024 arXiv

  17. [25]

    Divide and translate: Compositional first-order logic translation and verification for complex logical reasoning

    Hyun Ryu, Gyeongman Kim, Hyemin S Lee, and Eunho Yang. Divide and translate: Compositional first-order logic translation and verification for complex logical reasoning. ICLR, 2025

    2025

  18. [26]

    Morehopqa: More than multi-hop reasoning

    Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. Morehopqa: More than multi-hop reasoning. arXiv preprint arXiv:2406.13397, 2024

    2024 arXiv

  19. [27]

    Causal language modeling can elicit search and reasoning capabilities on logic puzzles

    Kulin Shah, Nishanth Dikkala, Xin Wang, and Rina Panigrahy. Causal language modeling can elicit search and reasoning capabilities on logic puzzles. arXiv preprint arXiv:2409.10502, 2024

    2024 arXiv

  20. [28]

    Musique: Multihop questions via single-hop question composition

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10: 0 539--554, 2022

    2022

  21. [29]

    Holy grail 2.0: From natural language to constraint models

    Dimos Tsouros, H \'e l \`e ne Verhaeghe, Serdar Kad o g lu, and Tias Guns. Holy grail 2.0: From natural language to constraint models. arXiv preprint arXiv:2308.01589, 2023

    2023 arXiv

  22. [30]

    On the planning abilities of large language models-a critical investigation

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36: 0 75993--76005, 2023

    2023

  23. [31]

    Llms still can't plan; can lrms? a preliminary evaluation of openai's o1 on planbench

    Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. Llms still can't plan; can lrms? a preliminary evaluation of openai's o1 on planbench. arXiv preprint arXiv:2409.13373, 2024

    2024 arXiv

  24. [32]

    Is a picture worth a thousand words? delving into spatial reasoning for vision language models, 2024 a

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models, 2024 a . URL https://arxiv.org/abs/2406.14852

    2024 arXiv

  25. [33]

    On the planning abilities of openai's o1 models: Feasibility, optimality, and generalizability

    Kevin Wang, Junbo Li, Neel P Bhatt, Yihan Xi, Qiang Liu, Ufuk Topcu, and Zhangyang Wang. On the planning abilities of openai's o1 models: Feasibility, optimality, and generalizability. arXiv preprint arXiv:2409.19924, 2024 b

    2024 arXiv

  26. [34]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. ACL, 2024 c

    2024

  27. [35]

    Autogen: Enabling next-gen llm applications via multi-agent conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  28. [36]

    Faithful logical reasoning via symbolic chain-of-thought

    Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic chain-of-thought. arXiv preprint arXiv:2405.18357, 2024

    2024 arXiv

  29. [37]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  30. [38]

    Natural plan: Benchmarking llms on natural language planning

    Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04520, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.