Pith. sign in

REVIEW 4 major objections 6 minor 79 references

Computational Reasoning of Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Turing Machine Bench scores LLMs by how faithfully they simulate a queue-based m-tag system, and reports that those scores strongly correlate with mainstream reasoning benchmarks.

desk verdict A genuinely useful synthetic benchmark for rule-execution, but the cross-domain proxy claim rests on a correlation analysis that doesn't control for scale or family. read the letter →

arxiv 2504.20771 v2 pith:VXHQF6OJ submitted 2025-04-29 cs.CL

Haitao Wu , Zongbo Han , Joey Tianyi Zhou , Huaxi Huang , Changqing Zhang This is my paper · ORCID
classification cs.CL
keywords LLMevaluationcomputationalreasoningTuringmachinetagsystemmulti-steprulefollowingbenchmarkcorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TMBench asks an LLM to simulate an m-tag system step by step: read the head symbol of a growing queue, append the production rule's output to the tail, then delete m symbols. The paper argues that this isolates a basic, knowledge-free ability it calls computational reasoning—strictly following rules and accurately managing internal state across many steps. Its central empirical claim is that the pass rate on TMBench strongly correlates with performance on established reasoning benchmarks across twelve leading LLMs, with the strongest correlations on GPQA and AIME2024. If that claim holds, a small simulation task can serve as a cross-domain proxy for LLM reasoning without depending on domain knowledge that may leak into training data.

What carries the argument

The m-tag system is a queue-based computational model: at each step the machine reads the queue head, appends the production word for that symbol to the tail, and deletes the first m symbols; for $m > 1$ it is Turing-complete, so any computable function can in principle be encoded. This machinery carries the argument because the LLM must perform exactly the operations that define computational reasoning—reading a rule, updating a queue, deleting symbols, and tracking a growing state—with every intermediate queue recorded for scoring. The benchmark defines three metrics on the resulting trace: per-step accuracy, step-weighted accuracy under uniform and linear weights, and pass rate, the probability of completing the simulation within the step limit without error.

What would settle it

Regress established-benchmark scores on TMBench pass rate within model families matched for parameter count: if the correlation largely disappears, the claim that TMBench measures a distinct computational-reasoning factor is falsified. A single counterexample model with a high TMBench pass rate but poor AIME or GPQA performance, or the reverse, would also break the reported relationship.

Watch

Extended reading notes

Core claim

The paper's discovery is a measurable, controllable proxy for multi-step rule execution. On TMBench, accuracy declines smoothly with simulation step number for every model, yet overall pass rate tracks real-task reasoning performance; the top proprietary models keep accuracy above 90% at step 30, while models below about 4 billion parameters struggle even at the first step. Across twelve models, TMBench pass rate correlates with AIME2024, MATH500, GPQA Diamond, and MMLU Pro, with reported $R^2$ values of 0.72, 0.64, 0.82, and 0.63, and the average reasoning score across AIME, MATH, and GPQA gives a Pearson correlation of 0.882. The authors read the correlation pattern—stronger on GPQA and AIME than on MMLU—as evidence that tasks demanding deeper computation depend more heavily on computational reasoning. The benchmark also produces clean step-wise accuracy curves, so it can reveal exactly where multi-step execution breaks down.

Load-bearing premise

TMBench's value as a proxy depends on the assumption that the observed correlations across twelve models reflect one shared computational-reasoning ability, and not simply model size, API access, or general instruction-following strength.

Editorial extensions

If this is right

  • A strong TMBench pass rate gives a knowledge-agnostic signal that the same model will likely do well on reasoning-heavy benchmarks such as GPQA and AIME2024.
  • The step-wise accuracy curves localize error propagation, so users can see at which step a model starts to lose the state rather than only whether the final answer is correct.
  • Because the deletion number m, rule lengths, and initial string lengths can be varied, TMBench can generate a continuous difficulty gradient for evaluating reasoning at chosen depths.
  • Because the tasks are self-contained and synthetic, benchmark scores are largely resistant to contamination by memorized test questions from pre-training data.
  • Since every autoregressive model eventually fails as the number of steps grows, TMBench can also serve as a stress test for long-horizon rule following.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a TMBench-style simulator run could serve as a cheap pre-screen before expensive multi-benchmark evaluations, provided the reported correlation generalizes past the twelve models.
  • Beyond the paper: a natural next test is whether pass rates also predict execution-style tasks such as code tracing or formal proof step-checking, which share the same rule-following load.
  • Beyond the paper: because the reported correlations do not control for model scale or API availability, the shared-factor interpretation is not yet separated from overall capability; controlling for parameter count within a family would make the proxy claim much stronger.
  • Beyond the paper: the smooth difficulty curve in the deletion-number ablation suggests a scaling-law-style experiment measuring how model size or inference compute must grow with step depth to keep pass rate constant.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Turing Machine Bench (TMBench), a benchmark that evaluates LLMs on the step-by-step execution of m-tag systems, a Turing-complete queue-based rewriting model. Each problem provides a small alphabet, production rules, an initial queue, and a step limit; the model must output the queue state after each transition. The authors define three metrics—step accuracy, step-weighted accuracy, and pass rate—and evaluate roughly three dozen open-source and proprietary models, producing per-step accuracy curves (Figure 1) and a model ranking table (Table 2). The central validation claim, in Section 4.3 and Figure 2, is that TMBench pass rate correlates strongly with AIME2024, MATH500, GPQA Diamond, and MMLU Pro across 12 models (reported R² values 0.63–0.82 and a composite Pearson r = 0.882), supporting TMBench as a proxy for general computational reasoning. The paper also reports ablations on unbounded-step execution, temperature, alphabet type, and deletion number (Section 4.4).

Significance. If the proxy claim holds, TMBench would be a valuable addition to LLM evaluation: it is knowledge-agnostic, has exact ground truth at every step, offers controllable difficulty, and yields interpretable step-wise performance curves. The core measurement is straightforward and internally consistent, and the paper covers a broad and current model set. The main contribution is empirical rather than theoretical—the m-tag system is a known Turing-complete model—but the benchmark design and metrics are clean and reproducible in principle. The significance is therefore contingent on whether the cross-benchmark correlations in Section 4.3 survive appropriate statistical controls; in its current form, the evidence for the headline proxy claim is suggestive but not conclusive.

major comments (4)
  1. [Section 4.3, Figure 2, Table 2] The central proxy-validity claim rests on correlations across only 12 models with no control for model scale, model family, or training recency. TMBench pass rate and the four reasoning benchmarks all improve substantially with model size and recent post-training, so the reported correlations could be driven by a common scaling/recency confound rather than by a shared 'computational reasoning' factor. The claim that TMBench 'serves as an effective proxy' therefore needs support from partial correlations controlling for log parameter count, within-family comparisons (e.g., Qwen models alone), or residual analysis after removing a scale trend. Please add these analyses and report model parameter counts for the 12 models.
  2. [Section 4.3, Figure 2, Figure 5] The correlation analysis uses min-max normalized scores on a small, nonrandom sample of 12 models, and the reported p-values are not corrected for multiple comparisons. A single outlier can drive an R² value when n=12, and min-max normalization can exaggerate correlation magnitudes. Please report raw-score correlations with confidence intervals, add leave-one-out or bootstrap analyses, and report Spearman correlations as a robustness check. The composite reasoning score (r = 0.882, Figure 5) is also presented without a confidence interval; the claim that the correlation order GPQA > AIME2024 > MATH500 > MMLU reflects reasoning depth is not supported without additional error bars and a test of the differences between these correlations.
  3. [Section 3.3, Figure 1, Table 2] The step-accuracy curves and SWA values are presented without error bars or variance estimates, making it difficult to know whether differences between models (e.g., Qwen3-32B at 10.5 vs. R1-Distill-Qwen-32B at 33.5 in Table 2) are meaningful. Since each configuration appears to use a single greedy decoding pass per instance, please report standard errors, bootstrap confidence intervals, or repeated sampling results for the per-step accuracies and the SWA metrics.
  4. [Section 4.4, Figure 3a] The unbounded-step ablation is based on a single model (Gemini-2.5-Pro) with 10 samples and budget forcing. The statement that LLMs 'inevitably fail with increasing steps due to their autoregressive nature' is too strong for this evidence. Either restrict the conclusion to Gemini-2.5-Pro or include additional models and more samples before making a general claim about autoregressive LLMs.
minor comments (6)
  1. [Table 1] The step-by-step queue states in Table 1 are inconsistent: some rows appear to show the queue after appending but before deletion, while others appear to show the state after deletion. Please clarify the convention used in the table so that it matches the formal step definition in Eq. (1) and the prompt template in Appendix B.2.
  2. [Abstract and Section 1] The phrase 'strictly follow rules and accurately manage internal states for multi-step' is missing a noun (e.g., 'multi-step computation'), and the phrase 'referred to as computational reasoning' is grammatically incomplete. Please revise for clarity.
  3. [Section 4.4, Figure 3c] Figure 3c does not identify which models are included in the alphabet ablation, and the text says 'as illustrated in Figure Figure 3c.' Please label the models and fix the typo.
  4. [Figure 4 caption] The caption contains the typo 'Token Distrubition' and should be 'Token Distribution.'
  5. [Figure 5 and Section 4.3] The composite 'Reasnoing Score' in Figure 5 is labeled inconsistently with the text (the text says Reasoning Score) and it averages only AIME2024, MATH500, and GPQA, while MMLU Pro is shown separately in Figure 2. Please define the composite score clearly and consistently.
  6. [References] Reference [70] is cited for GPT-4.1 but is titled 'OpenAI o1 system card'; please verify the citation and provide the correct source for GPT-4.1.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional circularity in the construct label; the Sec. 4.3 correlation evidence is independent and not fitted, so no substantive circularity.

  1. self definitional [Section 3.1 (construct definition and operationalization); cf. Section 2.3]
    "Computational reasoning is the ability to systematically select and accurately apply rules ... Therefore, we propose an evaluation framework to evaluate computational reasoning by Universal Turing Machine Simulation."

    The target construct 'computational reasoning' is stipulated as exactly the rule-execution behavior that TMBench's m-tag simulation tasks measure. Thus the assertion that TMBench is a proxy for computational reasoning is true by construction rather than by empirical derivation. This is an explicit operationalization rather than a hidden fitted result, and the later predictive-validity claim is supported by independent correlations with AIME2024, MATH500, GPQA, and MMLU Pro, which do not reduce to the definition.

full rationale

The only potentially circular element is the stipulative identification of computational reasoning with tag-system rule-following in Sections 2.3 and 3.1, making 'TMBench measures computational reasoning' tautological. The paper's substantive empirical claim in Section 4.3, however, is a post-hoc correlation between TMBench pass rate and external reasoning benchmarks across 12 models; no parameter is fitted to those benchmarks and no TMBench score is derived from them. The absence of controls for model scale or family is a validity and confounding concern, not a circularity under the stated criteria, and the paper's Turing-completeness argument rests on external literature rather than self-citation. The definitional step is minor and does not force the correlation results, so the circularity score is low.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on fitted constants; the benchmark's difficulty parameters are hand-set, and the validation is correlational. The main load-bearing assumptions are the Turing-completeness framing (a standard theorem), the representativeness of the default 100-instance configuration, and the construct validity of using correlation with existing reasoning benchmarks as evidence. No new physical or formal entities are introduced; 'computational reasoning' is a definition, not an entity.

free parameters (1)
  • TMBench dataset configuration (m=2, alphabet size 5, rule lengths 1-5, initial lengths 2-9, max steps 30) = m=2; alphabet=5; rules=1-5; init=2-9; max_step=30
    These hand-set parameters define task difficulty and are not derived from any principle. The paper shows difficulty varies with m but does not justify the default configuration as optimal or representative.
assumptions (4)
  • standard math Tag systems with deletion number m>1 are Turing-complete (Cocke-Minsky theorem)
    The paper's theoretical framing relies on this theorem to claim computational generality; invoked in Section 3.2 and Appendix A.
  • domain assumption Correlation with AIME2024, MATH500, GPQA Diamond, and MMLU Pro is a valid indicator that TMBench measures real reasoning, not just an artifact of model scale
    Section 4.3 treats R^2 values on 12 models as validation that TMBench is a cross-domain proxy; no control for scale or instruction-following is provided.
  • domain assumption The 100 sampled tag systems are representative enough to produce stable step-wise accuracy curves
    Section 4.1 specifies one default configuration; no sampling variance or confidence intervals are reported.
  • domain assumption Prompting the model to emit the queue state after each step measures the model's internal execution rather than memorized patterns
    Section 3.3 and Appendix B.2 rely on the one-shot prompt and exact-match evaluation; no control for contamination or pattern matching is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Computational Reasoning of Large Language Models." pith.science (2026). https://pith.science/paper/VXHQF6OJ

@misc{pith2026250420771,
  author       = {Pith},
  title        = {Pith review of: Computational Reasoning of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VXHQF6OJ}},
  note         = {Machine review of arXiv:2504.20771}
}
read the original abstract

With the rapid development and widespread application of Large Language Models (LLMs), multidimensional evaluation has become increasingly critical. However, current evaluations are often domain-specific and overly complex, limiting their effectiveness as cross-domain proxies for core capabilities. To address these limitations and enable a unified and simple evaluation framework, an ideal proxy task should target a basic capability that generalizes across tasks and is independent of domain-specific knowledge. Turing machine provides a powerful theoretical lens by reducing complex processes to basic, domain-agnostic computational operations. This perspective offers a principled framework for evaluating basic computational abilities essential to a wide range of tasks. Motivated by this abstraction, we introduce \textbf{Turing Machine Bench}, a benchmark designed to assess the ability of LLMs to \textbf{strictly follow rules} and \textbf{accurately manage internal states} for multi-step, referred to as \textbf{computational reasoning}. TMBench incorporates four key features: self-contained and knowledge-agnostic reasoning, a minimalistic multi-step structure, controllable difficulty, and a solid theoretical foundation based on Turing machine. Empirical results demonstrate that TMBench serves as an effective proxy for evaluating computational reasoning on representative LLMs. It produces clear step-wise accuracy curves, revealing LLMs' ability to execute multi-step reasoning processes. By analyzing performance trends across TMBench and established reasoning benchmarks, we find strong correlations with real-world tasks, bridging real-task evaluation with basic ability assessment. These findings suggest that TMBench holds potential as a cross-domain dimension for evaluating reasoning in LLMs. Code and data are available at \href{https://github.com/HaitaoWuTJU/Turing-Machine-Bench}{Repo}.

Figures

Figures reproduced from arXiv: 2504.20771 by the authors.

Figure 1
Figure 1. Illustration of the multi-step performance curve on TMBench across a diverse range of both [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Correlation between TMBench Pass Rate (computational reasoning) and established [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Illustration of ablation results. (a) Maximum correct step achieved by Gemini under [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of token distrubition. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Correlation between TMBench Pass Rate (computational reasoning ability) and Reasoning [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 25 canonical work pages

  1. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  2. [2]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  3. [3]

    Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. arXiv preprint arXiv:2406.12753, 2024

    arXiv 2024

  4. [4]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    arXiv 2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  6. [6]

    Code simulation challenges for large language models

    Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Samuele Marro, Anthony Cohn, Nigel Shadbolt, and Michael Wooldridge. Code simulation challenges for large language models. arXiv preprint arXiv:2401.09074, 2024

    arXiv 2024

  7. [7]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023

    arXiv 2023

  8. [8]

    Autonomous chemical research with large language models

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023

    2023

Show all 79 references
  1. [9]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020

    2020

  2. [10]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    2017 arXiv

  3. [11]

    Gpt-4 can’t reason

    Konstantine Arkoudas. Gpt-4 can’t reason. arXiv preprint arXiv:2308.03762, 2023

    2023 arXiv

  4. [12]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023

    2023

  5. [13]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2024, February 2024

    2024

  6. [14]

    Minif2f: a cross-system benchmark for formal olympiad-level mathematics

    Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics. arXiv preprint arXiv:2109.00110, 2021

    2021 arXiv

  7. [15]

    Proofnet: Autoformalizing and formally proving undergraduate-level mathematics

    Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level mathematics. arXiv preprint arXiv:2302.12433, 2023

    2023 arXiv

  8. [16]

    Executing arith- metic: Fine-tuning large language models as turing machines

    Junyu Lai, Jiahe Xu, Yao Yang, Yunpeng Huang, Chun Cao, and Jingwei Xu. Executing arith- metic: Fine-tuning large language models as turing machines. arXiv preprint arXiv:2410.07896, 2024

    2024 arXiv

  9. [17]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. 11

    2019

  10. [19]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  11. [20]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  12. [21]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  13. [22]

    ToxicChat: Unveiling hidden challenges of toxicity detection in real-world user-AI conversation

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. ToxicChat: Unveiling hidden challenges of toxicity detection in real-world user-AI conversation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Compu...

    2023

  14. [23]

    Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2023

  15. [24]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Infor...

    2023

  16. [25]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023

    2023

  17. [26]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022

    2022

  18. [27]

    Competition-level problems are effective llm evaluators

    Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, et al. Competition-level problems are effective llm evaluators. arXiv preprint arXiv:2312.02143, 2023

    2023 arXiv

  19. [28]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  20. [29]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  21. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  22. [31]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35...

    2022

  23. [32]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 12

    2022

  24. [33]

    Show your work: Scratchpads for intermediate computation with language models

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021

    2021 arXiv

  25. [34]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  26. [35]

    Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark

    Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13480–13488, 2021

    2021

  27. [36]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021

    2021

  28. [37]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  29. [38]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  30. [39]

    Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models

    Man Luo, Shrinidhi Kumbhar, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, Chitta Baral, et al. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models. arXiv preprint arXiv:2310.00836, 2023

    arXiv

  31. [40]

    Folio: Natural language reasoning with first-order logic

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, et al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022

    2022 arXiv

  32. [41]

    Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models

    Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. arXiv preprint arXiv:2406.17169, 2024

    2024 arXiv

  33. [42]

    Towards systematic evaluation of logical reasoning ability of large language models

    Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Towards systematic evaluation of logical reasoning ability of large language models. arXiv preprint arXiv:2404.15522, 2024

    2024 arXiv

  34. [43]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020

    2007 arXiv

  35. [44]

    Proofwriter: Generating implications, proofs, and abductive statements over natural language

    Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048, 2020

    2012 arXiv

  36. [45]

    From recognition to cognition: Visual commonsense reasoning

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019

    2019

  37. [46]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021

    2021

  38. [47]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  39. [48]

    Actionreason- ingbench: Reasoning about actions with and without ramification constraints

    Divij Handa, Pavel Dolin, Shrinidhi Kumbhar, Tran Cao Son, and Chitta Baral. Actionreason- ingbench: Reasoning about actions with and without ramification constraints. arXiv preprint arXiv:2406.04046, 2024

    2024 arXiv

  40. [49]

    Axiomatic set theory

    Patrick Suppes. Axiomatic set theory. Courier Corporation, 2012

    2012

  41. [50]

    Principles of mathematics

    Bertrand Russell. Principles of mathematics. Routledge, 2020

    2020

  42. [51]

    The feynman lectures on physics

    Richard Phillips Feynman. The feynman lectures on physics. (No Title), 1:46, 1963

    1963

  43. [52]

    The game of life

    John Conway et al. The game of life. Scientific American, 223(4):4, 1970. 13

    1970

  44. [53]

    The lean 4 theorem prover and programming language

    Leonardo de Moura and Sebastian Ullrich. The lean 4 theorem prover and programming language. In Automated Deduction–CADE 28: 28th International Conference on Automated Deduction, Virtual Event, July 12–15, 2021, Proceedings 28, pages 625–635. Springer, 2021

    2021

  45. [54]

    On computable numbers, with an application to the entschei- dungsproblem

    Alan Mathison Turing et al. On computable numbers, with an application to the entschei- dungsproblem. J. of Math, 58(345-363):5, 1936

    1936

  46. [55]

    Hao Wang, John Cocke, Marvin Minsky, and Stephen A. Cook. Tag systems and lag systems. Journal of Symbolic Logic, 36(2):344–344, 1971

    1971

  47. [56]

    Universality of tag systems with p = 2

    John Cocke and Marvin Minsky. Universality of tag systems with p = 2. J. ACM, 11(1):15–20, January 1964

    1964

  48. [57]

    Formal reductions of the general combinatorial decision problem

    Emil L Post. Formal reductions of the general combinatorial decision problem. American journal of mathematics, 65(2):197–215, 1943

    1943

  49. [58]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  50. [59]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  51. [60]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  52. [61]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  53. [62]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  54. [63]

    Qvq: To see the world with wisdom, December 2024

    Qwen Team. Qvq: To see the world with wisdom, December 2024

    2024

  55. [64]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  56. [65]

    Sky-t1: Train your own o1 preview model within $450

    NovaSky Team. Sky-t1: Train your own o1 preview model within $450. 2025

    2025

  57. [66]

    Gemini 2.5 pro, 2025

    Google DeepMind. Gemini 2.5 pro, 2025. Accessed: 2025-03-28

    2025

  58. [67]

    Grok-3, 2025

    xAI. Grok-3, 2025. Accessed: 2025-02-19

    2025

  59. [68]

    Claude 3.7 sonnet thinking, 2025

    Anthropic. Claude 3.7 sonnet thinking, 2025. Accessed: 2025-02-25

    2025

  60. [69]

    Doubao 1.5 pro, 2025

    Doubao. Doubao 1.5 pro, 2025. Accessed: 2025-01-22

    2025

  61. [70]

    Openai o1 system card, Sep

    OpenAI. Openai o1 system card, Sep. 2024. (Accessed on 2024/09/23)

    2024

  62. [71]

    Transformers: State- of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in n...

    2020

  63. [72]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  64. [73]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  65. [74]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025. 14 A Theory. In this section, we demonstra...

    2025 arXiv

  66. [75]

    Write the symbol Si onto the current tape cell

  67. [76]

    Move the tape head one cell in direction Di, whereDi∈{L,R}

  68. [77]

    Read the symbol S′ from the cell currently

  69. [78]

    • If S′ = 1, the next state isQi1

    Transition to the next state based on the symbol read S′: • If S′ = 0, the next state isQi0. • If S′ = 1, the next state isQi1. In this specific transition rule, the symbol writtenSi and the tape head directionDi are determined solely by the current stateQi. The subsequent sta...

    2024

  70. [79]

    In each transition, the machine performs the following steps: - If the queue length is less than m, halt - Read the head symbol of queue - Append symbols to the tail based on the head symbol and the corresponding transition rule - Delete m symbols from the head of the queue

  71. [80]

    The machine halt if: - The queue’s length is less than m. ## Example: m: 2 Alphabet: A, B, C Init: [B C A] Transition rules: A : C A C B : A C : B Simulation steps: ### step 0: - Action: Init - Queue State: [B C A] ### step 1: - Head Symbol: B - Action: Append A to the end of ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.