Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Long or short CoT? Investigating Instance-level Switch of Large Reasoning Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A per-question long/short CoT switch halves token cost and keeps accuracy.

desk verdict A practical, clearly-written empirical study on instance-level CoT switching with a real cost-saving method, but the core result is built on the same generator's own labels and needs broader validation. read the letter →

arxiv 2506.04182 v1 pith:TA6SZDH6 submitted 2025-06-04 cs.CL

Ruiqi Zhang , Changyi Xiao , Yixin Cao This is my paper
classification cs.CL
keywords chain-of-thoughtlargereasoningmodelstokenefficiencybudget-awareinstance-levelstrategyselectioncostreductionSwitchCoT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large reasoning models that always think long spend far more tokens than the accuracy gain justifies, especially on easier or memory-based questions. The paper argues that the right unit of control is the individual instance, not the task or model: some questions need extended reasoning, others are best answered quickly. It proposes SwitchCoT, a lightweight selector that chooses short or long chain-of-thought per question, optionally conditioned on an available token budget. The central empirical claim is that this switch reduces inference tokens by up to 50% while matching or exceeding the accuracy of using either strategy alone, and that it does best under tight budgets.

What carries the argument

The central object is SwitchCoT, a two-stage routing mechanism. Stage I is a fine-tuned selector model that maps either the question alone or the question with a budget to a strategy in {short, long}; Stage II is the unchanged generation model prompted with the selected style, with outputs truncated to the budget if one is given. The selector is trained on labels derived from a confusion matrix of the base model's own short- and long-CoT outcomes, with samples where both strategies fail discarded, plus budget-dependent labels that compare strategies under sampled generation budgets. The mechanism works because the selector is lightweight, the generation stage uses only prompt changes and truncation, and the training labels reflect exactly the per-instance conditions under which long CoT helps.

What would settle it

Take GSM8K under a 200-token budget, where short CoT outperforms long CoT by about 20 points; if the budget-aware SwitchCoT selector routes those instances to long CoT, its claimed budget adaptivity is wrong. Repeating the same check with a different base model tests whether the learned routing labels transfer.

Watch

Extended reading notes

Core claim

The paper's central discovery is that long and short CoT are complementary at the instance level, and that a model can learn to route between them. On benchmarks spanning math, knowledge, fact recall, social reasoning, sentiment, and creative tasks, long CoT improves accuracy mainly on hard math and knowledge questions, while short CoT is more cost-effective and often more accurate under strict token limits. SwitchCoT operationalizes this by fine-tuning the base model as a strategy selector that predicts, for each question and optionally a budget, whether to generate with short or long CoT; the same base model then answers with the chosen prompt style. The paper reports that SwitchCoT cuts average token use from roughly 1000 to 500 per instance, maintains or improves accuracy relative to long CoT, and under budget constraints matches or beats both fixed strategies.

Load-bearing premise

The load-bearing premise is that the labels used to train the selector, produced by the same base model answering each question with short and long prompts and discarding samples where both fail, are reliable enough that a model trained on them will route new questions correctly; if those labels are noisy or do not transfer across models or budgets, the reported savings disappear.

Editorial extensions

If this is right

  • Deploying SwitchCoT on math, knowledge, and social benchmarks cuts average token use from about 1000 to 500 per instance while keeping accuracy comparable to long CoT.
  • Under tight generation budgets, the budget-aware selector beats both fixed strategies, because short CoT wins when tokens are scarce and long CoT wins when they are plentiful.
  • On out-of-distribution fact, creative, and sentiment datasets, the selector keeps accuracy near long-CoT levels while spending roughly half the tokens.
  • No generator retraining is needed: the answer stage uses prompt changes and truncation, so SwitchCoT can wrap existing reasoning models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A step the paper leaves implicit: because the selector learns labels from one base model's own short and long outcomes, its routing may not transfer to a different model; cross-model routing is a natural extension the experiments do not test.
  • The paper notes in its conclusion that reasoning depth is a continuum; a natural continuation is a controller that decides how many tokens to spend per question rather than choosing between two prompt styles.
  • The same budget-aware routing idea could be applied to other cost-varying choices at inference time, such as sampling count, self-consistency, or retrieval, wherever an instance-level switch is cheaper than always using the expensive option.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates when long versus short chain-of-thought (CoT) prompting is preferable for large reasoning models at the instance level. It first reports an empirical analysis across five models and a range of datasets (math, knowledge, fact, social, sentiment, creative), concluding that long CoT gives the largest accuracy gains on hard math and knowledge tasks, that its advantage diminishes with model scale, and that more tokens do not uniformly buy accuracy. It then proposes SwitchCoT, a two-stage framework: a fine-tuned copy of the generator (DeepSeek-R1-Distill-Qwen-7B) predicts whether to use short or long CoT for each question, optionally conditioned on a token budget, and the original model generates the answer with the selected prompt. The main reported results are a roughly 50% token reduction relative to always-long CoT with comparable or slightly better aggregate accuracy, plus strong performance under limited token budgets in Figure 7 and on out-of-distribution datasets in Table 2.

Significance. If the empirical findings hold, the paper is a useful empirical contribution: it documents an instance-level token-accuracy tradeoff with explicit confusion-matrix evidence, shows that fixed long CoT is often wasteful, and proposes a lightweight, prompt-based router that requires no generation-time parameter changes. The paper ships code and evaluates on a broad set of benchmarks, and the qualitative claims in Section 3 are broadly consistent with the detailed tables in Appendices B. The central limitation is that the router is trained and evaluated with the same generator's own oracle labels, so the reported cost savings are a self-imitation result; cross-model and oracle-upper-bound checks are needed before the method can be considered broadly applicable as claimed in the abstract and introduction.

major comments (4)
  1. [Appendix C, Section 6.1, Tables 1–2] The selector's training labels are computed by running DeepSeek-R1-Distill-Qwen-7B with short and long prompts, and the same model is used for generation in evaluation. This makes the reported 52% token reduction (Table 1: 556 vs 1174 tokens on ALL) a self-imitation result; it does not demonstrate that SwitchCoT transfers across generators or that the router captures task properties rather than model-specific quirks. Add (i) a cross-generator experiment (e.g., labels from one model and generation from another, or at least a Qwen3-8B generator with a selector trained on its own labels), and (ii) an oracle upper bound that always selects the strategy with higher accuracy per instance, so the reader can see how far the learned router is from the achievable optimum. The limitation section (Section 7) discusses only the binary-strategy restriction and does not mention this label-transfer risk; without these checks, the 'broadly applicable' claim in the abstract and Section 1 is not established.
  2. [Appendix C (labeling), Section 6.2, Figure 7] The False-False rule ('discard') is specified for training data, but the evaluation protocol does not state whether False-False test instances are included in Tables 1–2 and Figure 7. If they are included, the selector is asked to route examples from a region it has never seen, and both strategies are known to fail on those items, so accuracy there is not informative while token cost is still affected. If they are excluded, the reported accuracy and token numbers are not directly comparable to full-benchmark numbers from other work. Please specify the evaluation inclusion rule and report both full-set and 'routable' numbers, because the 50% cost-saving claim depends on what happens on exactly these hard, token-hungry instances.
  3. [Appendix C (budget-aware labeling), Section 5.1, Figure 7] The budget-aware label generation is underspecified. The text defines a dataset-specific threshold τ_i and samples budgets from b_d plus ten uniformly sampled offsets δ_i, but never defines the function that maps an (instance, budget) pair to a ground-truth strategy; the phrase 'satisfies the budget constraint while maintaining accuracy' is ambiguous when both strategies fail under the budget, when both succeed, or when a truncated long CoT produces no extractable answer. Since Figure 7 and the budget-aware variant of SwitchCoT depend on these labels, provide the precise decision rule and the values or distributions of τ_i, b_d, and δ_i; otherwise the budget-aware experiments are not reproducible.
  4. [Tables 1–2, Figure 7] No uncertainty quantification is reported. Several benchmarks are very small (AIME25 n=30, AMC n=40, GPQA n=198), and the headline 'maintaining high accuracy' rests on differences such as 88.9 vs 88.2 on ALL (Table 1), 60.2 vs 62.1 on Fact (Table 2), and 74.9 vs 74.8 on Sentiment (Table 2), which are within a plausible noise range. Report bootstrap confidence intervals or standard errors at least for the small benchmarks, and state the number of fine-tuning seeds used for the selector. Without this, the claim that SwitchCoT 'matches or even surpasses' long CoT is not statistically supported.
minor comments (6)
  1. [Table 3] The dataset category 'Soicial' is a typo and should be 'Social'.
  2. [Section 3.2, Table 5] The text says math and knowledge gains 'reach up to 20%', but Table 5 shows AMC23 with DeepSeek-R1-Distill-Qwen-7B gains of 89.4 − 62.4 = 27.0 percentage points; please reconcile the stated maximum.
  3. [Section 2.1] The phrase 'explicitly generating explicit intermediate reasoning steps' repeats 'explicit'; this should be rephrased.
  4. [Appendix C] The cross-entropy loss formula contains '−NX i=1' where a summation symbol appears to be missing; the equation is malformed as printed.
  5. [Figure 4] The caption describes 'orange-highlighted top-right cells (True-False)' but the figure layout is hard to read; please clarify which cells are orange and ensure the caption matches the visual layout.
  6. [Appendix D] In the short-CoT MATH-500 example, the model answers 4320 while the ground truth is 720; if this is intended to illustrate short CoT failing on a math problem, state that explicitly, since otherwise the example appears to contradict the efficiency claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: SwitchCoT's headline results are measured against external gold labels on held-out and out-of-distribution benchmarks, not derived from its own training labels.

full rationale

The only self-referential element is the label generation in Appendix C: strategy labels are produced by running the same base model, DeepSeek-R1-Distill-Qwen-7B, with short and long prompts, and False-False samples are discarded. This is a data-generation choice that limits cross-model generality, but it does not make the reported results circular. The central claims—up to 50% token reduction and accuracy comparable to or exceeding long CoT—are evaluated on held-out and out-of-distribution datasets in Section 6 (Tables 1 and 2) against external gold answers, with token counts measured from actual generated outputs. No fitted parameter is renamed as a prediction: the selector is trained with cross-entropy to oracle strategy labels, and the paper's success metric is downstream accuracy, which is not equal to the training objective by construction. No load-bearing self-citations, imported uniqueness theorems, or renamed known results appear. Therefore, under the required standard of exhibiting a specific reduction, no circular step can be identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a trained selector whose labels are produced by the same base model that generates answers. The method is controlled by prompt templates and a truncation scheme, both of which are assumptions about model behavior rather than derived facts. No new physical entities are introduced. The main fitted quantities are the per-dataset budget thresholds and offset sampling, which are not reported.

free parameters (2)
  • dataset-specific budget threshold tau_i = not reported
    Appendix C defines a per-dataset threshold tau_i to separate short and long CoT in budget-aware labeling. The threshold is a fitted quantity, and its values are not given, so the budget-aware training labels depend on it.
  • budget offset sampling distribution (base budget b_d plus ten uniformly sampled offsets delta_i) = not reported
    Appendix C samples budget values around b_d per instance. The choice of base budget and offset scale affects which strategy is labeled optimal, and no values are reported.
assumptions (4)
  • domain assumption The placeholder think block ('Okay, I think I have finished thinking.') reliably produces short-CoT behavior in the evaluated models.
    Section 3.1 and Appendix D use this prompt to control reasoning style; if it changes output quality beyond length, the short-CoT baseline and SwitchCoT comparisons are biased.
  • domain assumption Budget Force truncation with an inserted closing think tag is a fair controlled way to compare strategies under token limits.
    Section 4.2 adopts this method from Muennighoff et al. 2025; truncation may favor certain answer styles, and the fairness of inserting a closing tag is assumed rather than validated.
  • ad hoc to paper Samples where both short and long CoT are incorrect (False-False) can be discarded from selector training without biasing evaluation.
    Appendix C removes all False-False cases; this concentrates training on decidable instances and may inflate apparent routing quality.
  • domain assumption The fine-tuned selector's strategy predictions transfer to held-out and out-of-distribution datasets.
    Section 6.1 and Table 2 evaluate transfer, but the selector is trained only on math, knowledge, and social datasets, and generalization to fact, sentiment, and creative tasks is assumed to rely on shared question features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Long or short CoT? Investigating Instance-level Switch of Large Reasoning Models." pith.science (2026). https://pith.science/paper/TA6SZDH6

@misc{pith2026250604182,
  author       = {Pith},
  title        = {Pith review of: Long or short CoT? Investigating Instance-level Switch of Large Reasoning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TA6SZDH6}},
  note         = {Machine review of arXiv:2506.04182}
}
read the original abstract

With the rapid advancement of large reasoning models, long Chain-of-Thought (CoT) prompting has demonstrated strong performance on complex tasks. However, this often comes with a significant increase in token usage. In this paper, we conduct a comprehensive empirical analysis comparing long and short CoT strategies. Our findings reveal that while long CoT can lead to performance improvements, its benefits are often marginal relative to its significantly higher token consumption. Specifically, long CoT tends to outperform when ample generation budgets are available, whereas short CoT is more effective under tighter budget constraints. These insights underscore the need for a dynamic approach that selects the proper CoT strategy based on task context and resource availability. To address this, we propose SwitchCoT, an automatic framework that adaptively chooses between long and short CoT strategies to balance reasoning accuracy and computational efficiency. Moreover, SwitchCoT is designed to be budget-aware, making it broadly applicable across scenarios with varying resource constraints. Experimental results demonstrate that SwitchCoT can reduce inference costs by up to 50% while maintaining high accuracy. Notably, under limited token budgets, it achieves performance comparable to, or even exceeding, that of using either long or short CoT alone.

Figures

Figures reproduced from arXiv: 2506.04182 by the authors.

Figure 1
Figure 1. Performance gains from using long CoT across different datasets and representative models. Datasets are ordered by average accuracy. The highest accuracies are observed in mathematical reasoning tasks (AIME25, AMC23, MATH500, GSM8K), followed by knowledge-based tasks (GPQA, MMLU, ARC), both of which generally involve higher reasoning complexity. Different data types As illustrated in the top-left panel of [PITH_FUL… view at source ↗
Figure 2
Figure 2. Trade-off between Accuracy and Token Usage. Each arrow indicates the change from short CoT to long CoT across datasets and DeepSeek-distilled model variants. 0 3000 6000 9000 12000 0 2000 4000 6000 8000 Δtoken 0 400 800 1200 1600 Math Others Knowledge [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Confusion matrices for various datasets, where the cells correspond to: top-left = True￾True, top-right = True-False, bottom-left = False-True, and bottom-right = False-False. Here, the first value denotes the correctness of the short CoT prediction, and the second denotes that of the long CoT. The orange-highlighted top-right cells (True-False) indicate cases where long CoT harms performance, which are the primary … view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Token-efficient performance of short vs. long CoT. Short CoT excels under tight token budgets, while long CoT performs better when more tokens are available. 4.2 Budget Aware The effectiveness of long CoT reasoning is influenced by the available computational budget, s…
Figure 6
Figure 6. Figure 6: The framework of SwitchCoT. Given a question, it first selects the optimal reasoning strategy using a fine-tuned selector model, taking into account instance characteristics and, optionally, a budget constraint (Stage I). Then in Stage II, a generation model produces t…
Figure 7
Figure 7. Figure 7: Experiment results under different token budget. The red line represents our method. It consistently achieves the best performance across different token budgets. 6.1 Instance-Level Selection Results To evaluate the effectiveness of our strategy selector at the individ…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Strategic Reflectivism In Intelligent Systems

    cs.AI 2025-05 conditional novelty 4.0 of 10

    Strategic Reflectivism holds that intelligent systems should allocate reflective reasoning tactically, weighing its benefits against its costs.

  2. Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.

Reference graph

Works this paper leans on

35 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    Aime 2025 dataset

    AIME. Aime 2025 dataset. URL https://huggingface.co/datasets/opencompass/AIME2025

    work page 2025

  2. [2]

    Amc 2023 dataset, 2023

    AMC. Amc 2023 dataset, 2023. URL https://huggingface.co/datasets/math-ai/amc23

    work page 2023

  3. [3]

    Training language models to reason efficiently, 2025

    Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025. URL https://arxiv.org/abs/2502.04463

    arXiv 2025

  4. [4]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 17682--17690, 2024

    2024

  5. [5]

    Do not think that much for 2+3=? on the overthinking of o1-like llms

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms. ArXiv, abs/2412.21187, 2024. URL https://api.semanticscholar.org/CorpusID:275133600

    arXiv 2024

  6. [6]

    Compressed chain of thought: Efficient reasoning through dense representations

    Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint arXiv:2412.13171, 2024

    arXiv 2024

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    arXiv 2018

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 35 references
  1. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, and Zhuoshu Li. Deepseek-r1: Incentivizing reasoning capability in llm...

    2025 arXiv

  2. [10]

    Token-budget-aware llm reasoning

    Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547, 2024

    2024 arXiv

  3. [11]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  4. [12]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  5. [13]

    How well do llms compress their own chain-of-thought? a token complexity approach

    Ayeong Lee, Ethan Che, and Tianyi Peng. How well do llms compress their own chain-of-thought? a token complexity approach. arXiv preprint arXiv:2503.01141, 2025

    2025 arXiv

  6. [14]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  7. [15]

    Can language models learn to skip steps? arXiv preprint arXiv:2411.01855, 2024

    Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Can language models learn to skip steps? arXiv preprint arXiv:2411.01855, 2024

    2024 arXiv

  8. [16]

    Thought manipulation: External thought can be efficient for large reasoning models

    Yule Liu, Jingyi Zheng, Zhen Sun, Zifan Peng, Wenhan Dong, Zeyang Sha, Shiwen Cui, Weiqiang Wang, and Xinlei He. Thought manipulation: External thought can be efficient for large reasoning models. arXiv preprint arXiv:2504.13626, 2025

    2025 arXiv

  9. [17]

    Reasoning models can be effective without thinking

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. 2025 a . URL https://api.semanticscholar.org/CorpusID:277781570

    2025

  10. [18]

    Reasoning models can be effective without thinking

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.09858, 2025 b

    2025 arXiv

  11. [19]

    Zettlemoyer, Percy Liang, Emmanuel J

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei Li, Hanna Hajishirzi, Luke S. Zettlemoyer, Percy Liang, Emmanuel J. Candes, and Tatsunori Hashimoto. s1: Simple test-time scaling. ArXiv, abs/2501.19393, 2025. URL https://api.semanticscholar.org/CorpusID:276079693

    2025 arXiv

  12. [20]

    Learning to reason with llms., 2024

    OpenAI. Learning to reason with llms., 2024. URL https://openai.com/index/learning-to-reason-with-llms

    2024

  13. [21]

    Opentriviaqa dataset, 2020

    OpenTriviaQA. Opentriviaqa dataset, 2020. URL https://github.com/uberspot/OpenTriviaQA/tree/master

    2020

  14. [22]

    poetry dataset, 2024

    poetry. poetry dataset, 2024. URL https://huggingface.co/datasets/merve/poetry

    2024

  15. [23]

    Qwen3: Think deeper, act faster, 2025

    Qwen. Qwen3: Think deeper, act faster, 2025. URL https://qwenlm.github.io/blog/qwen3/

    2025

  16. [24]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  17. [25]

    The benefits of a concise chain of thought on problem-solving in large language models

    Matthew Renze and Erhan Guven. The benefits of a concise chain of thought on problem-solving in large language models. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pp.\ 476--483. IEEE, 2024

    2024

  18. [26]

    Socialiqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  19. [27]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Proces...

    2013

  20. [28]

    Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, ...

    2023

  21. [29]

    Token assorted: Mixing latent and text tokens for improved language model reasoning

    DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint arXiv:2502.03275, 2025

    2025 arXiv

  22. [30]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu...

    2022

  23. [31]

    When more is less: Understanding chain-of-thought length in llms

    Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. ArXiv, abs/2502.07266, 2025. URL https://api.semanticscholar.org/CorpusID:276259519

    2025 arXiv

  24. [32]

    Tokenskip: Controllable chain-of-thought compression in llms

    Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502.12067, 2025

    2025

  25. [33]

    Chain of draft: Thinking faster by writing less

    Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint arXiv:2502.18600, 2025

    2025 arXiv

  26. [34]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 0 11809--11822, 2023

    2023

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.