Pith. sign in

REVIEW 4 major objections 5 minor 40 references

This paper claims that inserting model-generated 'insights' between reasoning steps lets a 7B LLM match self-consistency accuracy at roughly half the wall-time, and add over 5–7.5 absolute points when combined with self-consistency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

TBYS inserts generated situation-and-goal insights between reasoning steps and reports math accuracy gains, but the insight library is filtered using test-set answers.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The proactive insight idea is worth a look, but the evaluation leaks test-set labels into insight filtering and library-size selection, so the headline gains are not established. the 4 major comments →

arxiv 2508.18648 v2 pith:QVHSIWFP submitted 2025-08-26 cs.CL

Thinking Before You Speak: A Proactive Test-time Scaling Approach

Cong Liu , Wenchang Chai , Hejun Wu , Yan Pan , Pengxu Wei , Liang Lin This is my paper
classification cs.CL
keywords LLM reasoningproactive promptinginsight generationtest-time scalingin-context learningmathematical reasoningself-consistencychain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Complex reasoning fails, the paper argues, because human-written training traces usually omit the quiet inner thoughts that connect one step to the next. TBYS closes that gap by having the model proactively write an 'insight'—a short situation-and-goal statement—before each reasoning step, using exemplars retrieved from an automatically built and filtered insight library. On MATH-500 and AIME, a 7B instruction-tuned model with TBYS matches self-consistency accuracy at five or seven samples, and TBYS combined with self-consistency gains over 5% and 7.5% absolute accuracy while roughly halving wall-time and cutting completion tokens by a third. If the result holds, any LLM can scale its test-time reasoning without fine-tuning or fixed prompt workflows, and the resulting traces are more human-readable.

Core claim

The paper's central claim is that the missing ingredient in LLM mathematical reasoning is not more steps or more samples but the explicit 'insight' that bridges steps. A TBYS run is built from alternating insight and solution-step generations: first a preliminary insight summarizes the situation and states the goal, then eight exemplar insights are retrieved from a library and used to refine it, then the next solution step is generated conditioned on history plus insight. The library is constructed by splitting existing CoT solutions into 1–3 steps, synthesizing an insight for each step, and scoring every insight by how often it appears in runs that reach the correct final answer (with a log

What carries the argument

The insight: a two-part meta-prompt inserted before each reasoning step, whose 'situation' summarizes the current reasoning state and whose 'goal' states the intent of the coming step. The insight is generated in two passes—a first unconditional draft, then a refined version conditioned on eight retrieved exemplars from an insight library. The library is the second load-bearing piece: it is built automatically by splitting chain-of-thought solutions, synthesizing an insight per step, and filtering by the score ri/(ri+wi)·log(ri+wi), which rewards insights that appear in correct runs while still requiring enough usage. These two mechanisms—proactive, situation-aware prompting and data-driven

Load-bearing premise

The gains are measured after the insight library has been selected and scored on the same test questions whose answers the method then reports, so the method is being tuned to the test set rather than tested for generalization.

What would settle it

Build the insight library using only a disjoint training split (or no ground-truth answers) and evaluate on a held-out set; if the accuracy premium over 8-shot CoT shrinks or vanishes, the reported gains come from test-set filtering rather than the insight mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If accurate, TBYS offers a drop-in test-time scaling method that needs no fine-tuning, no hand-written prompt workflows, and only an embedding model for retrieval.
  • The method converts sparse human rationales into reusable, scored reasoning fragments, so the insight library itself becomes a transferable asset for future problems.
  • Combining proactive insights with self-consistency appears to stack gains, implying the two mechanisms correct different error modes: step-connection failures versus sample-level variance.
  • Because every step is announced by an explicit situation and goal, TBYS reasoning traces are more inspectable and potentially more useful as educational explanations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's filtering procedure scores insights using the final answer on the same test set it evaluates on; a fairer test would build and filter the library on a disjoint set, and the reported margins may shrink—an experiment the authors did not run.
  • The method's benefits likely depend on the base model's own ability to write good goals; on weaker models the self-generated insights may degrade, so the library may be more valuable than the generation step.
  • The matching of situation strings could let an insight library curated by one model transfer to another, making the library a model-agnostic resource; this is directly testable.
  • The proactive-prompting idea extends beyond situation/goal pairs to any meta-cognitive prompt (checklists, subgoal templates, verification cues), which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TBYS (Thinking Before You Speak), a test-time prompting framework that inserts 'insights' (situation/goal pairs) between reasoning steps. Insights are generated by an LLM guided by retrieved examples from an automatically constructed 'insight library'. The library is initialized from a chain-of-thought dataset and then filtered using a scoring rule based on whether the insight leads to correct answers on a dataset DG. The experiments on MATH-500 and AIME claim that TBYS matches Self-Consistency with 5 samples on MATH-500 and 7 samples on AIME, and that combining TBYS with Self-Consistency gives large accuracy gains with reduced wall-time and completion tokens. A central element of the protocol is that the test sets themselves are used as DG for filtering and for selecting library size.

Significance. The idea of proactively generating 'insights' that describe the situation and goal of each reasoning step is an interesting departure from static prompting and from purely reactive self-correction. The unfiltered-library ablation (Table 2) shows that the basic mechanism already outperforms 8-shot prompting on both benchmarks, which suggests some genuine potential. However, the headline claims of parity with Self-Consistency and of large accuracy gains depend on a filtering step and a library-size selection that both use the test-set labels. As reported, the numbers in Section 5.2 do not estimate generalization to unseen problems. If the evaluation were redone with a held-out validation set, the method might be of interest, but the current manuscript does not support its central claims.

major comments (4)
  1. [Section 3, 'Filtering'] The filtering rule explicitly uses ground-truth correctness: 'If the reasoning yields a correct answer, increment ri for each ii used; otherwise, increment wi.' The same section states that 'the test set, e.g., MATH-500 or AIME, serves as DG'. Therefore the insights used during inference are scored and selected on the very questions used to report accuracy in Section 5.2. This is label leakage, not test-time adaptation, which normally uses unlabeled test inputs. The ablation '- Library Construction' (Table 2) does not rescue the headline: without filtering, accuracy drops from 61.17% to 58.90% on MATH-500 and from 21.90% to 19.51% on AIME, so the claimed parity with SC@5/SC@7 depends on the leaked filtering step.
  2. [Section 5.5, Figure 5] The library size kL is chosen as the value that maximizes accuracy on the test set: peak at 50 for MATH-500 and 500 for AIME. This is a second selection on the evaluation labels. Even if filtering were moved to a separate validation set, choosing kL by test accuracy would inflate the reported results. A valid protocol must use a validation split for both insight filtering and library-size selection, and report accuracy on a held-out test set.
  3. [Section 5.3, Table 1] The overhead comparison is built on the same invalid accuracy match. TBYS accuracies (0.61 on MATH-500 and 0.22 on AIME) are obtained after test-set filtering and library-size selection; SC@5 and SC@7 are compared against these fitted numbers. If the filtering does not transfer to unseen problems, the wall-time and token savings at 'comparable accuracy' are not meaningful. The table also reports no variance across the 8 runs, so the claimed accuracy equivalence cannot be assessed statistically.
  4. [Section 6, Table 3] The comparison to Skill-based CoT and k-wait lacks essential setup information: the dataset is not explicitly identified, the number of runs and seeds are not given, and no error bars or statistical tests are provided. The sentence 'Results in Figure 3 shown that TBYS is slight better than Skill-based CoT' is also unverified by the table's numbers. This is secondary to the main evaluation problem but makes the additional comparison difficult to interpret.
minor comments (5)
  1. [Abstract and typography] The abstract says 'we proposes'; Figure 2 caption spells 'TYBS Reasoning'; Section 5.1 reads 'use utilize the LLM'; Section 7 repeats 'with the, with the the'. These should be corrected.
  2. [References] The AIME dataset is cited as (Zhang et al., 2023a), but the reference list entry for Zhang et al. (2023a) is about 'Action inference by maximising evidence', not the AIME benchmark. The dataset source should be cited correctly.
  3. [Section 5.4, Table 2] The ablation labels are terse ('- Library Construction', '- Coding'). Clarify what exactly is removed for each row, and whether coding removal also affects the baseline 8-shot results.
  4. [Section 5.5, Figure 5] The figure shows only single curves with no error bars or per-run variability, despite the text saying results are averaged over 8 runs. Reporting standard deviations would help assess the stability of the kL peaks.
  5. [Section 3] The hyperparameters kF=25, kE=8, and the seed set S are fixed without sensitivity analysis. Since the method's success depends on insight retrieval and filtering, some robustness evidence would be useful.

Circularity Check

2 steps flagged

Test-set labels leak into insight filtering and k_L selection; headline accuracy and overhead comparisons are therefore partially circular.

specific steps
  1. fitted input called prediction [Section 3 (Filtering) and Section 5.2 (Comparison)]
    "In our experiments, the MATH-500 dataset (Lightman et al., 2023) serves as DS and the test set, e.g., MATH-500 or AIME (Zhang et al., 2023a), serves as DG in a test-time adaptation (Jang et al., 2023) manner, with kL as a variable parameter. ... If the reasoning yields a correct answer, increment ri for each ii used; otherwise, increment wi."

    The library L1 used for inference is built by scoring each insight with ri/(ri+wi)*log(ri+wi), where ri/wi are counts of whether the final answer was correct on DG. DG is explicitly the same MATH-500 or AIME set whose accuracy is later reported in Figures 3-4 and Table 1. Thus the insights are selected using the test-set ground-truth labels, and the reported accuracy on that same set is not an independent prediction. The ablation in Table 2 confirms the leakage is load-bearing: removing library filtering drops MATH-500 from 61.17% to 58.90% and AIME from 21.90% to 19.51%, so the claimed parity with SC@5/SC@7 depends on this test-label-selected library.

  2. fitted input called prediction [Section 5.5 (Impact of Library Size)]
    "Figure 5 shows that on MATH-500, TBYS achieves peak accuracy with an insight library size of 50. On AIME, the optimal size is 500."

    kL, the number of insights kept after filtering, is chosen by peaking accuracy on the same test sets used in the main comparisons. Since kL is a hyperparameter selected from evaluation labels, the reported accuracies after this selection are post-hoc best-case numbers, not out-of-sample predictions. This compounds the test-label leakage from Section 3: both the contents of the library and its size are fitted to MATH-500/AIME answers, and the headline accuracy/overhead claims are then measured on those same answers.

full rationale

TBYS contains no mathematical derivation chain, so the classic 'Eq. X = Eq. Y by construction' circularity does not apply. The circularity is empirical: the paper's central evidence loop is partially closed. Section 3 states that the test set serves as DG and filters insights by whether TBYS produces the correct final answer on DG; Section 5.5 selects kL by peaking accuracy on the same test sets. Consequently, the reported accuracies in Figures 3-4, Table 1, and Table 2 are produced by a pipeline whose exemplar library and library size were both chosen using the test-set labels that define those accuracies. This is a fitted-input-called-prediction pattern, and the unfiltered-library ablation shows the leaked filtering contributes the difference (61.17 vs 58.90 on MATH-500; 21.90 vs 19.51 on AIME). However, there is independent content: the framework's insight-generation mechanism is not defined by the test set, the raw-library ablation still beats 8-shot, and the paper does not rely on self-citations for its core premise. The appropriate score is therefore partial circularity (6), not full circularity. If the authors had used a held-out validation set for filtering and kL selection, or only transductive unlabeled inputs, the remaining evaluation would be cleaner; as written, the headline comparisons are not estimates of generalization on those benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central method depends on several hand-chosen numbers (kE, kF, kL, seed examples, sampling settings) and on the disputed assumption that filtering on labeled test answers is valid. The only genuinely invented construct is the insight and its library, both lacking independent validation outside the paper.

free parameters (5)
  • kL (insight library size) = 50 for MATH-500, 500 for AIME
    Selected in Section 5.5 as the size that gives peak accuracy on the test sets, a clear test-set fit.
  • kE (number of retrieved insight exemplars) = 8
    Chosen by hand in Section 2 with no sensitivity study; affects insight quality and thus final accuracy.
  • kF (number of candidate insights in filtering) = 25
    Chosen by hand in Section 3; affects filtering noise with no analysis of robustness.
  • Seed examples S = 3 manually curated examples
    Manually defined in Section 2; the library and all insights inherit their style and quality, and the examples are not provided in the text.
  • Sampling parameters = temperature=0.2, top_k=40, top_p=0.7
    Fixed inference settings from Section 5.1 that affect output diversity and are part of the reported results.
axioms (5)
  • domain assumption Ground-truth answers in MATH-500 and AIME are correct and the datasets measure mathematical reasoning ability.
    Section 5 uses these datasets as both filter set and evaluation set; if labels are noisy, the filtering and the accuracy are both affected.
  • domain assumption Averaging over 8 runs with a commercial API provides a stable accuracy estimate.
    Section 5.1 reports averages over 8 runs without variance or significance testing; sampling noise is assumed negligible.
  • domain assumption Retrieval with bge-large-en-v1.5 embeddings returns insight exemplars that are useful for the current situation.
    Section 2 relies on retrieved insights for in-context learning; no analysis of retrieval quality is provided.
  • ad hoc to paper Using the test set as DG with ground-truth labels for test-time adaptation is a valid evaluation protocol.
    This is the paper's own framing in Section 3; it is the main load-bearing and contested premise, because filtering uses test-set answers before reporting accuracy on the same test set.
  • domain assumption LLM-generated step splits and insight drafts are faithful enough to serve as library content.
    The automatic library construction in Section 3 assumes the LLM produces usable steps and insights without manual verification beyond the correctness filter.
invented entities (2)
  • Insight (situation + goal) no independent evidence
    purpose: Inserted before each reasoning step to summarize status and set intention, guiding the model's next step.
    Introduced by the paper as the core new construct; only within-paper qualitative examples and ablations are provided, with no independent outside handle.
  • Insight library L no independent evidence
    purpose: Stores filtered insights for retrieval as in-context examples.
    Internal artifact; its construction depends on test-set labels in the current paper, so it is not independently validated.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking Before You Speak: A Proactive Test-time Scaling Approach." pith.science (2026). https://pith.science/paper/QVHSIWFP

@misc{pith2026250818648,
  author       = {Pith},
  title        = {Pith review of: Thinking Before You Speak: A Proactive Test-time Scaling Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVHSIWFP}},
  note         = {Machine review of arXiv:2508.18648}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) often exhibit deficiencies with complex reasoning tasks, such as maths, which we attribute to the discrepancy between human reasoning patterns and those presented in the LLMs' training data. When dealing with complex problems, humans tend to think carefully before expressing solutions. However, they often do not articulate their inner thoughts, including their intentions and chosen methodologies. Consequently, critical insights essential for bridging reasoning steps may be absent in training data collected from human sources. To bridge this gap, we proposes inserting \emph{insight}s between consecutive reasoning steps, which review the status and initiate the next reasoning steps. Unlike prior prompting strategies that rely on a single or a workflow of static prompts to facilitate reasoning, \emph{insight}s are \emph{proactively} generated to guide reasoning processes. We implement our idea as a reasoning framework, named \emph{Thinking Before You Speak} (TBYS), and design a pipeline for automatically collecting and filtering in-context examples for the generation of \emph{insight}s, which alleviates human labeling efforts and fine-tuning overheads. Experiments on challenging mathematical datasets verify the effectiveness of TBYS. Project website: https://gitee.com/jswrt/TBYS

Figures

Figures reproduced from arXiv: 2508.18648 by Cong Liu, Hejun Wu, Liang Lin, Pengxu Wei, Wenchang Chai, Yan Pan.

Figure 1
Figure 1. Figure 1: A simplified example to compare the reason [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The TBYS reasoning framework (Section 2) and insight library construction (Section 3). 2 The TBYS Reasoning Framework TBYS utilizes a library L of high-quality insights. The automatic construction of this library is de￾tailed in Section 3. During inference, examples are retrieved from L using some off-the-shelf em￾bedding model for in-context learning. We also manually define three seed examples S, each co… view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison on AIME [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Impact of insight library size In Section 3, we sorted the insight library L0 and selected the top-kL insights to form L1 [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example 1 for Qualitative Analysis of Insight Quality [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Example 2 for Qualitative Analysis of Insight Quality [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 16 canonical work pages

  1. [1]

    https://siliconflow.cn/ https://siliconflow.cn/

    work page

  2. [2]

    Anonymous. 2024 a . https://openreview.net/forum?id=z5uVAKwmjf AF low: Automating agentic workflow generation . In The Thirteenth International Conference on Learning Representations (ICLR)

    work page 2024

  3. [3]

    Anonymous. 2024 b . Chain of ideas: Revolutionizing research in idea development with LLM agents. In The Thirteenth International Conference on Learning Representations (ICLR)

    work page 2024

  4. [4]

    Anonymous. 2025. https://openreview.net/forum?id=FSjIrOm1vz Inference scaling for long-context retrieval augmented generation . In The Thirteenth International Conference on Learning Representations (ICLR)

    work page 2025

  5. [5]

    Shulin Cao, Jiajie Zhang, Jiaxin Shi, Xin Lv, Zijun Yao, Qi Tian, Lei Hou, and Juanzi Li. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.835 Probabilistic tree-of-thought reasoning for answering knowledge-intensive complex questions . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12541--12560, Singapore. Associatio...

    doi:10.18653/v1/2023.findings-emnlp.835 2023

  6. [6]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research

    2023

  7. [7]

    Aniket Rajiv Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy P Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Curtis Mozer, and Sanjeev Arora. 2024. https://openreview.net/forum?id=0MsI3bSmmD Metacognitive capabilities of LLM s: An exploration in mathematical problem solving . In AI for Math Workshop @ ICML 2024

    work page 2024

  8. [8]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J \"u rgen Schmidhuber. 2024. https://openreview.net/forum?id=VtmBAGCN7o Meta GPT : Meta programming for a multi-agent collaborative framework . In The Twelfth ...

    2024

  9. [9]

    Minguk Jang, Sae-Young Chung, and Hye Won Chung. 2023. Test-time adaptation via self-training with nearest neighbor information. ICLR 2024

    work page 2023

  10. [10]

    Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. 2024. https://arxiv.org/abs/2411.13504 Disentangling memory and reasoning ability in large language models . Preprint, arXiv:2411.13504

    Pith/arXiv arXiv 2024

  11. [11]

    Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. 2025 a . https://arxiv.org/abs/2503.04625 Start: Self-taught reasoner with tools . Preprint, arXiv:2503.04625

    Pith/arXiv arXiv 2025

  12. [12]

    Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei Han, Le Sun, and Yongbin Li. 2025 b . Structrag: Boosting knowledge intensive reasoning of llms via inference-time hybrid information structurization. In International Conference on Learning Representations (ICLR)

    work page 2025

  13. [13]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. ICLR 2024

    work page 2023

  14. [14]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-...

    doi:10.18653/v1/2022.acl-long.556 2022

  15. [15]

    Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.20 Faithful chain-of-thought reasoning . In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the ...

    doi:10.18653/v1/2023.ijcnlp-main.20 2023

  16. [16]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://arxiv.org/abs/2303.17651 Self-refine: Iterative refinement with self-feedback . In NeurIPS

    Pith/arXiv arXiv 2023

  17. [17]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. https://arxiv.org/abs/2501.19393 s1: Simple test-time scaling . Preprint, arXiv:2501.19393

    Pith/arXiv arXiv 2025

  18. [18]

    OpenAI. 2024. https://openai.com/index/learning-to-reason-with-llms/ Learning to Reason with LLMs

    work page 2024

  19. [19]

    Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. https://arxiv.org/abs/2408.06195 Mutual reasoning makes smaller llms stronger problem-solvers . In Arxiv

    Pith/arXiv arXiv 2024

  20. [20]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters . Preprint, arXiv:2408.03314

    Pith/arXiv arXiv 2024

  21. [21]

    Chemagent: Self-updating library in large language models improves chemical reasoning

    Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, et al. Chemagent: Self-updating library in large language models improves chemical reasoning. In The Twelfth International Conference on Learning Representations

    work page

  22. [22]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://doi.org/10.18653/v1/2023.acl-long.557 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014...

    doi:10.18653/v1/2023.acl-long.557 2023

  23. [23]

    Boshi Wang, Xiang Deng, and Huan Sun. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.174 Iteratively prompt pre-trained language models for chain of thought . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2714--2730, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.174 2022

  24. [24]

    Jianing Wang, Qiushi Sun, Xiang Li, and Ming Gao. 2024. https://doi.org/10.18653/v1/2024.acl-long.271 Boosting language models reasoning with chain-of-knowledge prompting . In The 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4958--4981, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.acl-long.271 2024

  25. [25]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.147 Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    doi:10.18653/v1/2023.acl-long.147 2023

  26. [26]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 b . https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conference on Learning Representations

    work page 2023

  27. [27]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Processing Systems

    2022

  28. [28]

    Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. 2024. https://doi.org/10.18653/v1/2024.acl-long.720 Faithful logical reasoning via symbolic chain-of-thought . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13326--13365, Bangkok, Thailand. Association for C...

    doi:10.18653/v1/2024.acl-long.720 2024

  29. [29]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng ...

    Pith/arXiv arXiv 2024

  30. [30]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024 b . https://openreview.net/forum?id=Bb4VGOWELI Large language models as optimizers . In The Twelfth International Conference on Learning Representations

    work page 2024

  31. [31]

    Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. 2024 c . Buffer of thoughts: Thought-augmented reasoning with large language models. arXiv preprint arXiv:2406.04271

    Pith/arXiv arXiv 2024

  32. [32]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/639a9a172c044fbb64175b5fad42e9a5-Paper-Conference.pdf Star: Bootstrapping reasoning with reasoning . In Advances in Neural Information Processing Systems, volume 35, pages 15476--15488. Curran Associates, Inc

    work page 2022

  33. [33]

    Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Rest-mcts*: Llm self-training via process reward guided tree search. Thirty-eighth Conference on Neural Information Processing Systems (NeurIPS)

    work page 2024

  34. [34]

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. 2025. https://arxiv.org/abs/2503.24235 A survey on test-time scaling in large language models: What, how, where, and how well? Preprint, arXiv:2503.24235

    Pith/arXiv arXiv 2025

  35. [35]

    Xingyuan Zhang, Philip Becker-Ehmck, Patrick van der Smagt, and Maximilian Karl. 2023 a . https://openreview.net/forum?id=WjlCQxpuxU Action inference by maximising evidence: Zero-shot imitation from observation with world models . In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)

    work page 2023

  36. [36]

    Zhebin Zhang, Xinyu Zhang, Yuanhang Ren, Saijiang Shi, Meng Han, Yongkang Wu, Ruofei Lai, and Zhao Cao. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.1 IAG : Induction-augmented generation framework for answering reasoning questions . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1--14, Singapore. ...

    doi:10.18653/v1/2023.emnlp-main.1 2023

  37. [37]

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. https://arxiv.org/abs/2205.10625 Least-to-most prompting enables complex reasoning in large language models . In The Eleventh International Conference on Learning Representations

    Pith/arXiv arXiv 2023

  38. [38]

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.108 P ro SA : Assessing and understanding the prompt sensitivity of LLM s . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950--1976, Miami, Florida, USA. Association for Computational Li...

    doi:10.18653/v1/2024.findings-emnlp.108 2024

  39. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  40. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.