Pith. sign in

REVIEW 2 major objections 4 minor 36 references

A new temporal-reasoning test shows mid-sized models guess correctly about 28% of the time with broken logic, while accuracy falls almost perfectly with a controllable hardness score.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 13:33 UTC pith:PGUNMLJK

load-bearing objection Solid systems paper: usable graded temporal benchmark + process oracle that actually measures spurious guessing; the hand-tuned difficulty scalar is the softest point but does not sink the contribution. the 2 major comments →

arxiv 2607.04784 v1 pith:PGUNMLJK submitted 2026-07-06 cs.SE

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

Shide Zhou , Kailong Wang , Ling Shi , Haoyu Wang This is my paper
classification cs.SE
keywords Large Reasoning Modelstemporal reasoningAllen Interval Algebradifficulty controltrace verificationspurious guessingbenchmark generationconstraint satisfaction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large Reasoning Models are often scored only on final answers, which hides whether they actually deduced or merely guessed. TRACE turns temporal reasoning into constraint networks drawn from Allen's Interval Algebra so that logical hardness can be dialed up in a controlled way and every step of a model's reasoning trace can be checked against the true algebraic closure. The authors build TRACEBench with 1,200 graded instances and evaluate eight LRMs. Performance drops almost linearly with the proposed difficulty score (average Pearson r ≈ -0.96). Mid-sized models produce correct answers on invalid reasoning roughly 28% of the time; small models more often finish with the wrong label after sound steps; advanced models stay faithful but can generate chains so long they exhaust the context window. The practical claim is that outcome-only scores systematically overstate reliability and that a process-checking, difficulty-controlled generator is required to measure genuine temporal deduction.

Core claim

When temporal reasoning tasks are generated as path-consistent Allen Interval Algebra networks whose scalar difficulty is controlled by event count and relation-weight average, True Reasoning Accuracy of eight LRMs falls with that score at Pearson r ≈ -0.96, while a Trace-Based Verification Oracle shows mid-sized models reach the right final answer via invalid steps about 28% of the time and exposes scale-dependent structural failures (Degenerative Loops versus Reasoning Explosion).

What carries the argument

TRACE: a three-module pipeline that (1) samples path-consistent constraint graphs under a scalar difficulty model D(G) = |V|^α · (mean Allen-relation weight), (2) converts explicit edges into natural-language premises and implicit deterministic relations into yes/no questions, and (3) verifies each claimed step of a model's JSON reasoning trace against the algebraic closure of the ground-truth network, classifying outcomes into True Reasoning, Spurious Guessing, Answer Misalignment, or Complete Failure.

Load-bearing premise

The hand-chosen numerical weights for the thirteen Allen relations and the single calibration that sets three-event transitive inference to difficulty 10 are assumed to measure real cognitive load for language models; if those weights mis-rank hardness, the claimed controllability and the correlation become circular.

What would settle it

Re-run the eight models on a new TRACEBench generated with deliberately altered relation weights or a different reference calibration; if the difficulty-accuracy correlation collapses or the spurious-guessing rates reverse order, the difficulty model is not a faithful measure of load.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces TRACE, a framework that models temporal reasoning as Allen Interval Algebra CSPs so that constraint graphs can be generated with a tunable scalar difficulty score D(G)=|V|^α · (average relation weight). Graphs are turned into natural-language premise/query pairs and scored by a Trace-Based Verification Oracle that checks every claimed step against the algebraic closure of the premises, yielding a four-way taxonomy (True Reasoning / Spurious Guessing / Answer Misalignment / Complete Failure). TRACEBench (1 200 instances, six difficulty tiers) is used to evaluate eight LRMs; True Reasoning accuracy falls with D at Pearson r≈−0.96, mid-sized models show ~28 % spurious guessing, and structural failures split into Degenerative Loops (small/mid models) versus Reasoning Explosion (advanced models).

Significance. If the results hold, TRACE supplies a contamination-resistant, process-level diagnostic that current temporal benchmarks lack. The dual-verification oracle and the concrete failure taxonomy (Tables 4–6) are immediately useful for LRM evaluation and training, independent of the precise numerical scale of D. The public generation pipeline and graded benchmark further lower the barrier to reproducible process-centric testing.

major comments (2)
  1. §3.2.1 and Eq. (1): the relation weights (0.8/1.0/1.1/1.5/2.0) and the exponent α≈1.75 are hand-assigned so that the three-event atomic task equals D_ref=10. Table 3 and Fig. 2 then show that achieved D tracks target D and that accuracy falls with D (r≈−0.96). Because the metric is both the generation target and the independent variable, the correlation is partly by construction. A short ablation that re-ranks the weights or replaces α with an alternative scaling (e.g., |E| or tree-width) and re-computes the correlation would establish that the gradient is robust rather than an artifact of the chosen scalars.
  2. §3.4, Eq. (14): a reasoning step is accepted only when the model’s claimed relation set exactly equals the solver-computed closure. This strict equality rejects any sound but incomplete intermediate claim (e.g., a correct disjunction that is later refined). The paper should either (a) report how many steps fail solely because of this exact-match rule, or (b) supply a relaxed “subset-of-closure” variant so that readers can judge whether the reported spurious-guessing rates are inflated by the verification criterion itself.
minor comments (4)
  1. Table 1 column headers are terse; expanding “Difficulty Control” and “Verification Method” into full phrases would improve readability.
  2. §4.1.1 lists “GPT-5-mini” and “Claude-Sonnet-4.6”; if these are placeholder or internal codenames, the public model identifiers should be stated for reproducibility.
  3. Fig. 2 legend reports per-model Pearson r values; adding 95 % confidence intervals or a note on the number of difficulty points (n=6) would clarify the strength of the correlations.
  4. The anonymous repository link (§7) should be replaced by a permanent DOI or public URL before camera-ready.

Circularity Check

0 steps flagged

No significant circularity: TRACE is a self-contained constructive benchmark whose difficulty metric is defined a priori, used for generation, and then empirically checked for correlation; the process-level oracle results are independent of that metric.

full rationale

The paper proposes a scalar difficulty model D(G)=|V|^α · (mean relation weight) with hand-assigned tier weights and α calibrated solely so that the three-event atomic transitive task equals 10 (§3.2.1). Candidate configurations are then generated to match a target D_tar (Alg. 1) and instantiated into path-consistent graphs (Alg. 2); Table 3 simply confirms that achieved D tracks the targets. Model True-Reasoning accuracy is subsequently shown to fall with those targets (Fig. 2, r≈−0.96). This is ordinary engineering validation of a controllable generator, not a first-principles derivation that reduces to its own inputs by construction. The weights and α are never fitted to the LRM accuracy numbers themselves, nor is any uniqueness theorem or prior self-result invoked to force the form of D. The load-bearing empirical claims—spurious-guessing rates (Table 4), Answer Misalignment, Degenerative Loops vs. Reasoning Explosion (Tables 5–6)—rest entirely on the independent Trace-Based Verification Oracle (§3.4) that checks each reasoning step against the algebraic closure of the ground-truth constraint network. No self-citation chain, fitted-parameter-as-prediction, or definitional loop appears. Score 0 is therefore the correct, proportionate finding.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central empirical claims rest on Allen's classical algebra (standard), a hand-crafted difficulty formula whose free parameters are calibrated to a single reference task, and the modeling decision that pure internal deduction without external solvers is the right evaluation target. No new physical entities are postulated; the free parameters are the relation weights and the scale exponent.

free parameters (3)
  • relation complexity weights w(r) = 0.8 / 1.0 / 1.1 / 1.5 / 2.0
    Four-tier hand assignment (0.8 for equals, 1.0 for before/after, 1.1 for meets, 1.5 for starts/finishes, 2.0 for overlaps/during) chosen by cognitive-load intuition rather than data fit or theorem (§3.2.1).
  • scale exponent α = ≈1.75
    Set so that the three-event atomic transitive task has D_ref=10 under mean weight ≈1.46, yielding α≈1.75 (§3.2.1).
  • edge density ρ and tolerance τ = ρ=0.5, τ=0.1
    Fixed at 0.5 and 0.1 respectively for all TRACEBench instances (§4.1.2); control generation but are free design choices.
axioms (4)
  • standard math Allen's Interval Algebra with its 13 basic relations and composition table is a complete and sound model of qualitative temporal relations.
    Taken as given from Allen 1983; used throughout §2.3 and generation.
  • domain assumption Path consistency on the constraint network is necessary and sufficient to guarantee a contradiction-free temporal interpretation for the generated instances.
    Enforced at generation time (§2.3, Algorithm 2); standard for IA but not always sufficient for full consistency in larger networks.
  • ad hoc to paper Evaluating pure internal deduction (no external solvers or tools) is the appropriate way to measure an LRM's temporal reasoning capability.
    Explicit design choice stated in abstract and §5; excludes tool-augmented regimes common in practice.
  • ad hoc to paper A reasoning step is valid if and only if the claimed relation set exactly equals the solver-computed algebraic closure of the premises.
    Strict equality criterion of the Trace-Based Verifier (§3.4); alternative weaker notions (subset, non-contradiction) are not explored.
invented entities (2)
  • TRACE difficulty score D(G) no independent evidence
    purpose: Scalar that maps a constraint graph to a continuous hardness level used both for generation and for claiming controllability.
    Defined by Eq. (1) with free parameters above; no independent psychophysical or external validation beyond the correlation it was designed to produce.
  • Four-way faithfulness taxonomy (True Reasoning / Spurious Guessing / Answer Misalignment / Complete Failure) no independent evidence
    purpose: Classify every model output by joint correctness of answer and full reasoning trace.
    Operational definition in §3.4; useful but paper-specific labeling scheme.

pith-pipeline@v1.1.0-grok45 · 26423 in / 3392 out tokens · 24478 ms · 2026-07-11T13:33:07.777347+00:00 · methodology

0 comments
read the original abstract

Defining the reasoning boundaries and ensuring the reliability of Large Reasoning Models (LRMs) remains a critical challenge. Current benchmarks primarily rely on static datasets susceptible to data contamination or synthetic tasks lacking fine-grained difficulty control. Furthermore, standard outcome-based evaluations often conceal reasoning flaws by neglecting the reasoning process. To address these limitations, we introduce TRACE, a testing framework that models temporal reasoning as constraint satisfaction problems via Allen's Interval Algebra. This approach enables precise regulation of logical complexity and incorporates a Trace-Based Verification Oracle to validate reasoning faithfulness. Using this framework, we construct TRACEBench, an extensive benchmark comprising 1,200 synthesized test instances across graded difficulty levels. We employ TRACE to evaluate eight widely used LRMs on TRACEBench. The results confirm a strong negative correlation between model performance and our difficulty metric (Pearson's r approximately -0.96), validating the effectiveness of our difficulty control mechanism. Moreover, our trace-based analysis exposes significant discrepancies between reasoning validity and final answers, revealing a high spurious guessing rate of approximately 28% in mid-sized models. In addition, we diagnose scale-dependent failure modes, ranging from Degenerative Loops in small models to Reasoning Explosion in advanced architectures. TRACE thus provides a robust, automated platform for benchmarking the true temporal reasoning capabilities of LRMs.

Figures

Figures reproduced from arXiv: 2607.04784 by Haoyu Wang, Kailong Wang, Ling Shi, Shide Zhou.

Figure 1
Figure 1. Figure 1: The Workflow of TRACE. 3.2 Difficulty-Aware Constraint Graph Generation 3.2.1 Difficulty Modeling. To systematically quantify the difficulty of a temporal reasoning task, we propose a model based on two primary dimensions: event-set scale and constraint complexity. Problem Formalization. We model a temporal reasoning task as a constraint graph G = (𝑉 , 𝐸), where 𝑉 = {𝑣1, 𝑣2, . . . , 𝑣𝑛} denotes the set of … view at source ↗
Figure 2
Figure 2. Figure 2: True Reasoning Accuracy of Evaluated LRMs Across Six Target Difficulty Levels. The bars represent individual model performance, while the dashed black line indicates the average accuracy trajectory. Pearson correlation coefficients (𝑟) are reported in the legend to quantify the negative correlation between difficulty and True Reasoning Accuracy. To verify whether TRACE can effectively generate temporal rea… view at source ↗
Figure 3
Figure 3. Figure 3: Fine-Grained Performance Breakdown of Evaluated LRMs Across Six Difficulty Levels. The 100% stacked bar charts visualize the composition of model outputs, categorized into True Reasoning, Spurious Guessing, Answer Misalignment, Complete Failure, and Parse Fail. Answer to RQ2-1: Reasoning performance generally improves with model scale. However, the base architecture is a critical factor: a smaller model (e… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 2 canonical work pages

  1. [1]

    James F. Allen. 1983. Maintaining knowledge about temporal intervals.Commun. ACM26, 11 (Nov. 1983), 832–843. doi:10.1145/182.358434

    doi:10.1145/182.358434 1983

  2. [2]

    Christian Bessière. 1996. A Simple Way to Improve Path Consistency Processing in Interval Algebra Networks. In Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence Conference, AAAI 96, IAAI 96, Portland, Oregon, USA, August 4-8, 1996, Volume 1, William J. Clancey and Dan...

    1996

  3. [3]

    Bradley C. A. Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini

  4. [4]

    arXiv:2407.21787 doi:10.48550/ARXIV.2407.21787

    Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.CoRRabs/2407.21787 (2024). arXiv:2407.21787 doi:10.48550/ARXIV.2407.21787

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21787 2024

  5. [5]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2024. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs.CoRRabs/2412.21187 (2024). arXiv:2412.21187 doi:10.48550/ARXIV.2412.21187

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.21187 2024

  6. [6]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 202...

    work page doi:10.18653/v1/2024.acl-long.66 2024

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems.CoRRabs/2110.14168 (2021). arXiv:2110.14168 https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  8. [8]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report.CoRRabs/2412.19437 (2024). arXiv:2412.19437 doi:10.48550/ARXIV. 2412.19437

    Pith/arXiv arXiv doi:10.48550/arxiv 2024

  9. [9]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.CoRR abs/2501.12948 (2025). arXiv:2501.12948 doi:10.48550/ARXIV.2501.12948

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2025

  10. [10]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating Large Language Models in Class-Level Code Generation. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 81:1–81:13. doi:10.1...

    doi:10.1145/3597503.3639219 2024

  11. [11]

    Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2025. Test of Time: A Benchmark for Evaluating LLMs on Temporal Reason- ing. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https:/...

    2025

  12. [12]

    Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-Based Prompting for Multi-step Reasoning. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

    2023

  13. [13]

    https://openreview.net/forum?id=yf1icZHC-l9

    OpenReview.net. https://openreview.net/forum?id=yf1icZHC-l9

  14. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  15. [15]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt

  16. [16]

    Measuring Mathematical Problem Solving With the MATH Dataset. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, Joaquin Vanschoren and Sai-Kit Yeung (Eds.). https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ be83ab3ecd0db773eb2dc1b0a...

    2021

  17. [17]

    Amar Isli and Hachemi Bennaceur. 1996. Networks of Qualitative Interval Relations: Combining Circuit Consistency and Path Consistency in the Search for a Solution. InProceedings of the Third International Workshop on Temporal Representation and Reasoning, TIME-96, Key West, Florida, USA, May 19-20, 1996, Luca Chittaro, Scott D. Goodwin, Howard J. Hamilton...

    work page doi:10.1109/time.1996.555682 1996

  18. [18]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Aga...

    2022

  19. [19]

    Lake and Marco Baroni

    Brenden M. Lake and Marco Baroni. 2018. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. InProceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 (Proceedings of Machine Learning Research, Vol. 80), Jennifer G. Dy an...

    2018

  20. [20]

    Jeongsoo Lee, Daeyong Kwon, and Kyohoon Jin. 2025. GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation.CoRRabs/2508.16994 (2025). arXiv:2508.16994 doi:10.48550/ARXIV.2508.16994

    doi:10.48550/arxiv.2508.16994 2025

  21. [21]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s Verify Step by Step. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id= v8L0pN6EOi

    2024

  22. [22]

    OpenAI. 2025. Introducing OpenAI o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/. Accessed: 2026-01-26

    2025

  23. [23]

    Chatterji, Faisal Ladhak, and Tatsunori Hashimoto

    Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2024. Proving Test Set Contamination in Black-Box Language Models. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=KS8mIvetg2

    2024

  24. [24]

    Keqin Peng, Liang Ding, Yuanxin Ouyang, Meng Fang, and Dacheng Tao. 2025. Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubt.CoRRabs/2505.23480 (2025). arXiv:2505.23480 doi:10.48550/ ARXIV.2505.23480

    Pith/arXiv arXiv 2025

  25. [25]

    Abulhair Saparov and He He. 2023. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/forum?id=qFVVBzXxR2V

    2023

  26. [26]

    Hamilton

    Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Nov...

    doi:10.18653/v1/d19-1458 2019

  27. [27]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.CoRRabs/2408.03314 (2024). arXiv:2408.03314 doi:10.48550/ARXIV.2408.03314

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.03314 2024

  28. [28]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. InFindings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Roge...

    doi:10.18653/v1/2023.findings-acl.824 2023

  29. [29]

    TRACE. 2026. A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation. https://sites.google.com/view/tracebench Accessed: 2026-01-28

    2026

  30. [30]

    Dingmin Wang, Bocheng Zou, Zhen Han, and zhiqiang xu. 2025. t-BEN: A Temporal Logic Guided Approach for Temporal Reasoning Benchmark Generation. https://openreview.net/forum?id=XkzGgKJAA2

    2025

  31. [31]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/forum?id=1PL1NIMMrw

    2023

  32. [32]

    Yuqing Wang and Yun Zhao. 2024. TRAM: Benchmarking Temporal Reasoning for Large Language Models. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 6389–6415. doi:10.18653/V1/2024.F...

    doi:10.18653/v1/2024.findings-acl.382 2024

  33. [33]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem...

    2022

  34. [34]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023...

    2023

  35. [35]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. STaR: Bootstrapping Reasoning With Rea- soning. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgra...

    2022

  36. [36]

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024. DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=gjfOL9z5Xr , Vol. 1, No. 1, Article...

    2024