Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The reasoning strategies inside long chain-of-thought traces can be extracted bottom-up from the model's own outputs, predicted from the question alone, and steered by prompting to improve accuracy and safety.

desk verdict A genuinely bottom-up taxonomy of reasoning strategies, but the headline accuracy-gain claim rests on a correlational definition of 'optimal' that needs a manipulation check before it carries the weight the paper puts on it. read the letter →

arxiv 2505.10185 v1 pith:XMU6DAUV submitted 2025-05-15 cs.CL cs.AI

Seongyun Lee , Seungone Kim , Minju Seo , Yongrae Jo , Dongyoung Go , Hyeonbin Hwang , Jinho Park , Xiang Yue
show 4 more authors
Sean Welleck Graham Neubig Moontae Lee Minjoon Seo
This is my paper · ORCID
classification cs.CLcs.AI
keywords chain-of-thoughtreasoningstrategiesbottom-uptaxonomystrategypredictionpromptsteeringtrainingdataformatmodelmergingLLMsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long chain-of-thought (the intermediate reasoning a model writes before answering) is not an opaque monolith, this paper argues: it decomposes into identifiable strategies—top-down versus bottom-up, hypothesis-driven versus data-driven, iterative versus immediate—that a pipeline can extract directly from model outputs without any human-predefined category. The paper claims these discovered strategies distinguish models far more sharply than standard predefined cognitive behaviors, can be predicted from the input question alone, and can be used as prompts: telling a model to adopt the strategy most likely to produce a correct answer improved accuracy and safety on five benchmarks, with gains of 2.5–8.3% across three reasoning models. A second set of controlled training experiments claims that the format of the training data—multiple-choice versus free-form—shapes reasoning strategies with effect sizes up to 1.5, far more than the data's domain ($d < 0.2$), and that linear weight interpolation between format-trained models smoothly interpolates their reasoning styles. If these claims hold, reasoning analysis becomes a real-time control knob for accuracy, safety, and training-data design rather than a post-hoc description.

What carries the argument

The load-bearing object is the CoT Encyclopedia taxonomy itself, built in five stages: (1) an LLM generates free-form contrastive reasoning criteria from raw chain-of-thought responses; (2) each criterion's text is embedded into a semantic space; (3) hierarchical agglomerative clustering with cosine distance compresses thousands of criteria into six interpretable dimensions, each represented by its medoid; (4) an LLM writes detailed contrastive rubrics for each dimension; and (5) each response is classified under every rubric, producing a binary strategy profile. The control claim rides on a second mechanism: a classifier that maps a question to its predicted strategy profile, combined with a Bayes-rule estimate $P(\text{Correct} \mid \text{Pattern})$ computed from responses that were initially correct, so the prompt can prescribe the strategy with the highest conditional success probability. The predictive claim is anchored by an $R^2 = 0.405$ correlation between question similarity and strategy similarity, used to justify transferring strategy labels from similar previously seen questions to unseen ones.

What would settle it

Run the paper's control pipeline on a benchmark, then compare the predicted-optimal prompt against a placebo prompt of equal length and format that describes a reasoning style never observed in any correct response, such as a strategy invented by shuffling rubric poles. If the placebo matches the optimal prompt's accuracy, the specific strategy content is not what drives the gain; the causal claim survives only if the optimal prompt wins consistently across benchmarks and models.

Watch

Extended reading notes

Core claim

The paper's central claim is that the reasoning strategies behind a model's long chain-of-thought can be discovered automatically, named, predicted, and controlled. The CoT Encyclopedia pipeline prompts an LLM to brainstorm contrastive reasoning criteria from model-generated traces (4,057 criteria such as "Top-down vs. Bottom-up"), embeds them in a semantic space, clusters them into six high-level rubrics (analytical perspective, scope of approach, reasoning type, idea development, verification focus, clarification approach), and classifies each response under each rubric. Human judges found the resulting analyses reasonable in 92–97% of cases, versus 51% for a predefined four-behavior taxonomy. Using these rubrics, the paper computes $P(\text{Correct} \mid \text{Pattern})$ for each strategy pole, trains a classifier that predicts the strategy a model will use from the question alone, and prompts the model to follow the strategy with the highest estimated success probability; across five benchmarks this improves accuracy and safety for three reasoning models (GPQA-Diamond accuracy 72.7% to 79.1%; XSTest safety 91.1% to 94.0%; WildGuard 89.1% to 92.9%). The paper presents this as, to its knowledge, the first demonstration that controlling a model's high-level reasoning strategy directly enhances accuracy. It further claims that training data format, not domain, dominates the reasoning style a model adopts, and that model-weight interpolation between format-trained models interpolates strategies without further training.

Load-bearing premise

The paper labels as optimal whatever reasoning pattern appears in responses that happened to be correct, then steers the model toward that pattern; if a pattern is merely correlated with correctness rather than the cause of it, the reported prompting gains could reflect generic instruction-following instead of genuinely better reasoning.

Editorial extensions

If this is right

  • Prompting a model with its predicted optimal strategy improves accuracy by 2.5–8.3% across three reasoning models and improves safety on XSTest and WildGuard (e.g., GPQA-Diamond accuracy rises from 72.7% to 79.1%).
  • The bottom-up taxonomy distinguishes models more finely than predefined cognitive behaviors (Cohen's $d$ up to 0.44, versus roughly 0.1 for the baseline), giving a more sensitive instrument for comparing reasoning models.
  • Training data format shifts reasoning strategy with effect sizes up to 1.5, so choosing multiple-choice versus free-form data is a deliberate lever over the model's later reasoning style rather than a neutral packaging choice.
  • Linear weight interpolation between format-trained models produces smooth strategy transitions without fine-tuning, a cheap way to tune reasoning behavior toward task needs.
  • Because similar questions elicit similar strategies ($R^2 = 0.405$), strategy classifiers trained on one benchmark transfer to others, supporting question-level adaptive prompting at inference time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because 'optimal' strategies are defined as the patterns found in responses that happened to be correct, the control gains rest on a correlation-is-causation step; a direct test would pit the predicted-optimal prompt against a matched placebo instruction describing a plausible strategy never observed in a correct response.
  • The $R^2 = 0.405$ ceiling implies that for question types far outside the training distribution, the predicted strategy may be near-random; the method's practical reach is bounded by how well the benchmark distribution covers the deployment space.
  • The format effect suggests a training-time corollary the paper does not pursue: deliberately mixing formats within one curriculum could cultivate hybrid reasoning styles directly, rather than only merging already-trained models.
  • Because the strategy predictor works from the question alone, a natural untested deployment is mid-generation intervention: classify the emerging trace, detect a known-bad pattern, and inject a corrective prompt before the final answer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript introduces the CoT Encyclopedia, a bottom-up framework that extracts contrasting reasoning criteria from long chain-of-thought traces via GPT-4o, embeds and clusters them into high-level dimensions, generates interpretable rubrics, and classifies new responses. The authors validate the taxonomy with human evaluation and then use it for two further purposes: predicting each question's 'optimal' reasoning strategy and steering models toward it by prompting, and analyzing how RLVR training-data format versus domain shapes reasoning strategies, including model-merge interpolation. On the control side, the paper reports accuracy and safety improvements from dataset-wide and question-specific optimal patterns across five benchmarks. On the analysis side, it reports that training format (multiple-choice versus free-form) has a much larger effect on reasoning strategies than training domain, with model merging producing smooth interpolations between formats.

Significance. If the control claims hold, the paper offers a practical method for steering long-CoT reasoning and a taxonomy that is more sensitive to model differences than predefined cognitive behaviors. The human evaluation is a genuine strength, as are the controlled RLVR experiments in Section 5, which carefully separate format from domain, and the cross-domain classifier design in Section 4.3. The model-merging interpolation results are also interesting and do not depend on the correlational label issue that affects the control experiments. The main weakness is that the central 'optimal strategy' notion is defined by correlation with correctness rather than by causal evidence; this limits the strength of the accuracy-control claims and the novelty assertion attached to them.

major comments (4)
  1. [§4.1, §4.3 (optimal strategy labels)] The definition of an 'optimal strategy' as the reasoning pattern observed in responses that are correct is correlational and is load-bearing for the entire control claim. In §4.1 the paper computes P(Correct | Pattern) from generated responses, and in §4.3 it trains the strategy predictor using 'questions initially answered correctly to derive optimal strategies.' Correctness and strategy share likely common causes (question difficulty, model knowledge, output length), so a pattern associated with correct answers need not be the cause of correctness. Prompting a model to adopt that pattern therefore does not establish direct control over reasoning quality. To support the causal claim, the authors should (i) select optimal patterns on a held-out development set, (ii) report a manipulation check showing that prompted responses actually increase the frequency of the intended rubric pattern relative to a no-instruction baseline, and (iii) include a control that instructs a matched-length, task-relevant but non-strategy behavior to exclude generic instruction-following.
  2. [§4.1 (dataset-wide optimal pattern selection)] The dataset-wide optimal patterns in §4.1 appear to be selected from the same benchmark responses on which the improvement is then measured: P(Correct | Pattern) and P(Safe | Pattern) are computed over the test sets of GPQA-Diamond, MMLU-Redux, MATH-500, XSTest, and WildGuard, and Figure 4 reports 'optimal-dataset' results on those same benchmarks. This is an in-sample selection procedure, so the reported gains may reflect selection bias rather than a transferable property of the strategy. The paper should either move pattern selection to a separate validation set or evaluate the full pipeline cross-validated, and it should report how many dataset-wide patterns were examined and whether any multiple-comparison correction was applied.
  3. [§4.1, §4.3 (confounds and manipulation check)] The prompting comparisons do not control for instruction length or for the amount of generated text. The optimal-strategy prompt in Appendix E contains six rubric descriptions and instructs the model to align its entire process, while the 'no instruction' baseline is a short task prompt; the 'random' and 'unoptimal' conditions also differ in content and length from the optimal condition. If the optimal instruction causes longer, more exploratory CoTs, part or all of the measured accuracy/safety gain could be test-time compute or verbosity rather than the specific strategy. The paper should measure output length in each condition, include a token-length-matched neutral instruction, and verify through the framework's own rubrics that prompted responses follow the intended pattern rather than only echoing the instruction.
  4. [Introduction, §4.3 (scope of the control claim)] The abstract and §4.3 claim that the method enables real-time adaptive control and, to the authors' knowledge, is the first demonstration that controlling high-level reasoning strategies directly enhances accuracy. Given the correlational definition of 'optimal' and the absence of a manipulation check, this is broader than the evidence supports. A more defensible claim is that prompting with strategies associated with correct responses on a training split improves accuracy on held-out questions; the causal interpretation should be stated as an open question or supported by the additional experiments above. In addition, the sentence in §4.3 reporting 'accuracy gains of 81.0%, 92.3%, and 80.7%' appears to report post-control accuracy rates, not gains; please reword to distinguish final rates from gains relative to the no-instruction baseline, and reconcile these values with the ranges in Figure 13.
minor comments (4)
  1. [§4.2] The text describes a 'regression analysis' between question similarity and strategy similarity, but only a correlation or R² is mentioned; please report the fitted slope, confidence intervals, and the number of pairs used so the reader can assess the predictive claim.
  2. [Tables 5 and 10] Several p-values are reported as '0.00'; please report exact values or use a format such as p < 0.001, since a p-value cannot be exactly zero.
  3. [Appendix A.3] The human evaluation uses 100 responses and four annotators with binary questions, but no inter-annotator agreement statistic is reported; adding Cohen's kappa or per-annotator consistency would strengthen the interpretability claim.
  4. [Throughout] There are several typographical and formatting issues: 'NA VER' in the author block, 'LLMo' in §3.2, 'Wild guard' in Figure 11, and the incomplete reference URL for NuminaMath in [10]; these should be cleaned up in revision.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; optimal-strategy labels are a heuristic but the control experiments measure real outcomes.

full rationale

The paper's main derivation is bottom-up and self-contained: criteria are extracted from model CoTs, clustered, and human-validated; the control experiments in Sec. 4.1-4.3 compare prompted outcomes (final-answer accuracy and safety scores) against no-instruction, random-pattern, and unoptimal-pattern baselines, so the reported gains are not equal to the fitted P(Correct|Pattern) values by construction. The 'optimal' strategy labels are derived from correct responses (Sec. 4.1: 'We compute P(Correct | Pattern)...'; Sec. 4.3: 'we use questions initially answered correctly to derive optimal strategies'), which is a heuristic rather than a causal proof; this is a validity limitation, acknowledged in Appendix D ('our reasoning strategy classification relies on GPT-4o outputs as an evaluator'), not a circular reduction. The cross-domain classifier setting provides out-of-sample evidence. The only self-citation is [27] (co-authored by two current authors) used as background for long-CoT scaling, and it is not load-bearing. Therefore the central claims do not reduce to their inputs by definition.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces the CoT Encyclopedia framework and six derived reasoning criteria, but no new physical or ontological entities. The main unvalidated inputs are the automated judgments of GPT-4o, the casual definition of optimal strategies, and the synthetic data assumptions.

free parameters (1)
  • Number of clusters k = 6
    Chosen via silhouette scores in Section 3.2 and Appendix A.1. This modeling choice determines the six high-level reasoning criteria that all subsequent analysis depends on.
assumptions (5)
  • domain assumption GPT-4o's reasoning-strategy classifications are valid proxies for human judgment.
    The entire pipeline uses GPT-4o to generate criteria, rubrics, and per-response labels. Human validation covers only 100 responses from three models, and the RL-trained model analyses are not human-validated. See Section 3.2 and Appendix A.3.
  • domain assumption The pattern used in a correct response is the optimal strategy for that question.
    Section 4.1 and 4.3 define optimal strategies as patterns from correct responses, treating correlation with correctness as sufficient for causal control. No intervention or counterfactual analysis is provided.
  • domain assumption LLaMA-Guard-3 8B labels accurately measure safety.
    All safety measurements in Section 4 use LLaMA-Guard-3 8B as the judge, without validation against human safety judgments. See Appendix A.4.
  • domain assumption The synthetic multiple-choice conversion of NuminaMath preserves content and isolates format.
    Section 5.1 states the conversion 'control[s] for content while isolating presentation format,' but the synthetic options may introduce stylistic or content cues that affect reasoning beyond format.
  • standard math Standard statistical tools (chi-squared, Cohen's d) are applied correctly to categorical pattern data.
    Chi-squared tests and Cohen's d are standard, but the data are per-response binary labels produced by an automated judge, so the tests inherit the judge's reliability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think." pith.science (2026). https://pith.science/paper/XMU6DAUV

@misc{pith2026250510185,
  author       = {Pith},
  title        = {Pith review of: The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMU6DAUV}},
  note         = {Machine review of arXiv:2505.10185}
}
read the original abstract

Long chain-of-thought (CoT) is an essential ingredient in effective usage of modern large language models, but our understanding of the reasoning strategies underlying these capabilities remains limited. While some prior works have attempted to categorize CoTs using predefined strategy types, such approaches are constrained by human intuition and fail to capture the full diversity of model behaviors. In this work, we introduce the CoT Encyclopedia, a bottom-up framework for analyzing and steering model reasoning. Our method automatically extracts diverse reasoning criteria from model-generated CoTs, embeds them into a semantic space, clusters them into representative categories, and derives contrastive rubrics to interpret reasoning behavior. Human evaluations show that this framework produces more interpretable and comprehensive analyses than existing methods. Moreover, we demonstrate that this understanding enables performance gains: we can predict which strategy a model is likely to use and guide it toward more effective alternatives. Finally, we provide practical insights, such as that training data format (e.g., free-form vs. multiple-choice) has a far greater impact on reasoning behavior than data domain, underscoring the importance of format-aware model design.

Figures

Figures reproduced from arXiv: 2505.10185 by the authors.

Figure 1
Figure 1. Comparison between conventional reasoning analysis and the CoT Encyclopedia. Traditional methods use fixed criteria to identify strategies but offer limited guidance for improving reasoning. The CoT Encyclopedia takes a bottom-up approach, uncovering diverse, task-specific strategies and enabling flexible analysis and actionable insights to enhance model performance. focus largely on short or medium-length CoTs and … view at source ↗
Figure 2
Figure 2. Overview of the COT ENCYCLOPEDIA. The framework constructs a taxonomy of reasoning strategies through five key stages: (1) Classification Criteria Identification – diverse reasoning criteria are identified from model-generated CoTs; (2) Classification Criteria Embedding – these criteria are converted into semantic embeddings; (3) Criteria Compression via Hierarchical Clustering – semantically similar criteria are cl… view at source ↗
Figure 3
Figure 3. Human Evaluation Results for CoT Encyclopedia. Human annotators found the gener￾ated criteria plausible, their mapping to high-level dimensions sensible, and the overall analysis rea￾sonable. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Impact of Pattern-Based Instructions on Model Performance. Five approaches are compared: not instructed, unoptimal instructions, random patterns, dataset-wide optimal patterns, and question-specific optimal patterns. Results show that optimal patterns improve performan…
Figure 5
Figure 5. Figure 5: Analysis of relationships between question similarity and reasoning strategy similarity across multiple benchmarks. Relationship between question similarity and reasoning strategy similarity. (a) Scatter plot showing positive correlation between question similarity and…
Figure 6
Figure 6. Figure 6: Comparison of effect sizes showing how question format and domain influence reasoning strategies across three benchmarks. Format differences (purple bars) consistently demonstrate substantially larger effects on reasoning strategies than domain differences (green bars)…
Figure 7
Figure 7. Figure 7: Qualitative and quantitative comparison between models trained on Multiple-choice (MC) and Free-form (FF) data formats. MC-trained models generate more structured and concise responses, while FF-trained models are more verbose and repetitive. These differences appear b…
Figure 8
Figure 8. Figure 8: Interpolation of reasoning strategies through model merging. reasoning strategy dynamics as models are merged from Multiple-Choice (MC) to Free-Form (FF) training formats across three benchmarks. multiple solution paths early on—similar to breadth-first search—whereas …
Figure 9
Figure 9. Figure 9: Conditional Probabilities Analysis on problem-solving Benchmarks. The bars represent different reasoning strategies categorized by Analytical Perspective (blue), Scope of Approach (yellow), Reasoning Type (green), Idea Development (orange), Verification Focus (purple),…
Figure 10
Figure 10. Figure 10: Conditional Probabilities Analysis on Safety Benchmarks. The bars represent different reasoning strategies categorized by Guidance Strategy (blue), Ethical Consideration (yellow), Safety Precedence (green), Idea Development (orange), Content Handling (purple), Risk As…
Figure 11
Figure 11. Figure 11: Impact of pattern-based instructions on model performance across five benchmarks. For all benchmarks, instructing models to follow question-specific optimal reasoning strategies yields the highest performance (17.5-68.5%), followed by dataset-wide optimal patterns (14…
Figure 12
Figure 12. Figure 12: Fine-grained safety response ratio analysis across XSTest and WildGuard bench￾marks. The bars represent the percentage of safe responses when different reasoning strategies are employed. ‘Benevolent’ reasoning achieves the highest safety scores (84.0% in XSTest, 81.0%…
Figure 13
Figure 13. Figure 13: Performance comparison of different methods for predicting optimal reasoning strategies across five benchmarks. Trained classifiers (both in-domain and cross-domain) signifi￾cantly outperform prompting-based methods across all benchmarks. In-domain classifiers achieve…
Figure 14
Figure 14. Figure 14: Hierarchical Clustering Across Benchmarks. Six different benchmarks: XSTest (top left), WildGuard (top right), Arena-Hard (mid left), and MMLU-Redux (mid right), MATH-500 (bottom left), GPQA-Diamond (bottom right). Stars indicate the representative embeddings (medoids…
Figure 15
Figure 15. Figure 15: Hierarchical Clustering of reasoning strategies with PCA Projection on problem￾solving Benchmarks. Comparing four different representative embedding selection methods: medoid-based (default, top-left), silhouette-based (top-right), frequency-based (bottom-left), and d…
Figure 16
Figure 16. Figure 16: Pattern ratio distributions across three different sizes of the Distill-R1 model family (32B, 7B, and 1.5B). The x-axis shows six reasoning strategy criteria: analytical perspective (ap), clarification approach (sa), reasoning type (rt), idea development (id), verific…
Figure 17
Figure 17. Figure 17: Pairwise Cohen’s d effect size measurements comparing reasoning strategy distribu￾tions across different sizes of the Distill-R1 model family. The x-axis displays the six reasoning strategy criteria: analytical perspective (ap), scope of approach (sa), reasoning type …
Figure 18
Figure 18. Figure 18: Examples of input formats and model responses from multiple-choice (MC) and free-form (FF) training data. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: Prompt template for identifying fine-grained reasoning criteria. This template guides the systematic extraction of diverse reasoning strategies from model responses by instructing the analysis to: (1) identify meaningful criteria with descriptive names that reflect ge…
Figure 20
Figure 20. Figure 20: Prompt template for rubric generation in reasoning strategy analysis. This template outlines the structured approach for creating assessment rubrics that distinguish between contrasting reasoning strategies. The prompt requests (1) concise definitions capturing each p…
Figure 21
Figure 21. Figure 21: Prompt template for comprehensive reasoning strategy analysis report generation. This template guides the systematic evaluation of model responses against predefined reasoning strategies. It establishes a structured analytical process involving (1) careful examination…
Figure 22
Figure 22. Figure 22: Prompt template for multiple-choice question assessment with structured reasoning. This template presents a standardized format for evaluating reasoning models on multiple-choice QA benchmarks (GPQA-Diamon, MMLU-Redux). Example prompt for steering models toward optima…
Figure 23
Figure 23. Figure 23: Example prompt for steering models toward optimal reasoning strategies. This template demonstrates how specific reasoning strategies can be explicitly prescribed to guide problem￾solving approaches. Six distinct reasoning strategies are presented: top-down, broad, ind…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reasoning Fine-Tuning Induces Persistent Latent Policy States

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Reasoning fine-tuning reorganizes chain-of-thought into more differentiated latent switching states, and pruning with those states beats self-consistency in 11 of 12 settings.

Reference graph

Works this paper leans on

53 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Skill-based few-shot selection for in-context learning.arXiv preprint arXiv:2305.14210, 2023

    Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. Skill-based few-shot selection for in-context learning.arXiv preprint arXiv:2305.14210, 2023

    arXiv 2023

  2. [2]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  3. [3]

    Metacognitive capabilities of llms: An exploration in mathematical problem solving.Advances in Neural Information Processing Systems, 37:19783–19812, 2024

    Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael C Mozer, and Sanjeev Arora. Metacognitive capabilities of llms: An exploration in mathematical problem solving.Advances in Neural Information Processing Systems, 37:19783–19812, 2024

    work page 2024

  4. [4]

    CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts Reasoning

    Yuanheng Fang, Guoqing Chao, Wenqiang Lei, Shaobo Li, and Dianhui Chu. Cdw-cot: Clustered distance-weighted chain-of-thoughts reasoning.arXiv preprint arXiv:2501.12226, 2025

    work page Pith review arXiv 2025

  5. [5]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    arXiv 2025

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  7. [7]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms.arXiv preprint arXiv:2406.18495, 2024

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms.arXiv preprint arXiv:2406.18495, 2024

    arXiv 2024

  8. [8]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

Show all 53 references
  1. [9]

    Qasc: A dataset for question answering via sentence composition

    Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qasc: A dataset for question answering via sentence composition. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090, 2020

    2020

  2. [10]

    Numinamath

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://huggingface.co/AI-MO/NuminaMath...

    2024

  3. [11]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939, 2024

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  4. [12]

    Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    2023

  5. [13]

    Deepseek-r1 thoughtology: Let’s< think> about llm reasoning.arXiv preprint arXiv:2504.07128, 2025

    Sara Vera Marjanovi ´c, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lù, et al. Deepseek-r1 thoughtology: Let’s< think> about llm reasoning.arXiv preprint arXiv:2504.07128, 2025

    2025

  6. [14]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018. 11

    2018 arXiv

  7. [15]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  8. [16]

    Modern hierarchical, agglomerative clustering algorithms.arXiv preprint arXiv:1109.2378, 2011

    Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms.arXiv preprint arXiv:1109.2378, 2011

    2011 arXiv

  9. [17]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models.arXiv preprint arXiv:2308.01263, 2023

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models.arXiv preprint arXiv:2308.01263, 2023

    2023 arXiv

  10. [18]

    Approximating kl divergence.John Schulman’s Homepage, 2020

    John Schulman. Approximating kl divergence.John Schulman’s Homepage, 2020

    2020

  11. [19]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  12. [20]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Langu...

    2019 doi

  13. [21]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  14. [22]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  15. [23]

    Crowdsourcing multiple choice science questions.arXiv preprint arXiv:1707.06209, 2017

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions.arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  16. [24]

    Thinkpatterns-21k: A systematic study on the impact of thinking patterns in llms.arXiv preprint arXiv:2503.12918, 2025

    Pengcheng Wen, Jiaming Ji, Chi-Min Chan, Juntao Dai, Donghai Hong, Yaodong Yang, Sirui Han, and Yike Guo. Thinkpatterns-21k: A systematic study on the impact of thinking patterns in llms.arXiv preprint arXiv:2503.12918, 2025

    2025

  17. [25]

    Lars: Latent reasoning skills for chain-of-thought reasoning

    Zifan Xu, Haozhu Wang, Dmitriy Bespalov, Xian Wu, Peter Stone, and Yanjun Qi. Lars: Latent reasoning skills for chain-of-thought reasoning. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 3624–3643, 2024

    2024

  18. [26]

    Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  19. [27]

    Demystifying long chain-of-thought reasoning in llms.ICML, 2025

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms.ICML, 2025

    2025

  20. [28]

    Evaltree: Profiling language model weaknesses via hierarchical capability trees.arXiv preprint arXiv:2503.08893, 2025

    Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, and Pang Wei Koh. Evaltree: Profiling language model weaknesses via hierarchical capability trees.arXiv preprint arXiv:2503.08893, 2025

    2025 arXiv

  21. [29]

    Least-to-most prompting enables complex reasoning in large language models.arXiv preprint arXiv:2205.10625, 2022

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models.arXiv preprint arXiv:2205.10625, 2022. 12 A Implementation de...

    2022 arXiv

  22. [30]

    Each criterion should have a clear and descriptive name that reflects a real aspect of the reasoning process

    Identify multiple *meaningful criteria* that differentiate reasoning strategies. Each criterion should have a clear and descriptive name that reflects a real aspect of the reasoning process. **Do not use generic placeholders like ‘Criterion 1’.**

  23. [31]

    *Outcome-first*, or *Concrete* vs

    For each criterion, describe two contrasting *pattern types* (e.g., *Step-by-step* vs. *Outcome-first*, or *Concrete* vs. *Abstract*)

  24. [32]

    Pattern B) Descriptive Criterion Name (Pattern A vs

    Present your analysis in the following format, using <patterns> and </patterns> tags to enclose the list: <patterns> Descriptive Criterion Name (Pattern A vs. Pattern B) Descriptive Criterion Name (Pattern A vs. Pattern B) ... Descriptive Criterion Name (Pattern A vs. Pattern ...

  25. [33]

    Do not include any explanations or commentary within the <patterns> tags

  26. [34]

    You are encouraged to define your own diverse and insightful pattern criteria based on the given response

    The example format above is only a guide. You are encouraged to define your own diverse and insightful pattern criteria based on the given response. Response: {answer} Figure 19:Prompt template for identifying fine-grained reasoning criteria.This template guides the systematic...

  27. [35]

    A clear, concise definition (2-3 sentences) that captures the essence of this reasoning strategy

  28. [36]

    3-4 key characteristics that distinguish this pattern

  29. [37]

    The def- initions and examples should help evaluators quickly categorize model responses without ambiguity

    2 concrete examples of responses that demonstrate this pattern (keep examples brief, about 2-3 sentences each) Focus on making the distinctions between patterns clear and easily identifiable. The def- initions and examples should help evaluators quickly categorize model respon...

  30. [38]

    A rubric describing two distinct reasoning strategies (Pattern A and Pattern B)

  31. [39]

    Analysis Process:

    A model response to analyze Your task is to create a detailed analysis report that determines which pattern the response exhibits. Analysis Process:

  32. [40]

    Carefully examine the response against both pattern definitions in the rubric

  33. [41]

    Identify specific elements, structures, and linguistic features in the response that align with either pattern

  34. [42]

    Note any mixed signals or elements that span both patterns

  35. [43]

    Determine which pattern (A or B) the response most closely matches Report Structure:

  36. [44]

    **Initial Observations** (2-3 sentences summarizing key features of the reasoning approach)

  37. [45]

    **Evidence for Pattern A**: - If applicable, quote 1-2 specific segments from the response that demonstrate Pattern A - Explain how these segments match characteristics described in the rubric

  38. [46]

    **Evidence for Pattern B**: - If applicable, quote 1-2 specific segments from the response that demonstrate Pattern B - Explain how these segments match characteristics described in the rubric

  39. [47]

    **Pattern Determination**: - Explain which pattern (A or B) is most dominant and why - Address any aspects that show characteristics of both patterns

  40. [48]

    Final pattern determination: [PATTERN NAME]

    **Conclusion**: - Clearly state the final pattern determination using the format: "Final pattern determination: [PATTERN NAME]" Focus on concrete evidence from the response that matches specific elements from the rubric patterns. Rubric: {rubric} Response to analyze: {response...

  41. [49]

    Fresnel Drag Formula: The formula for the speed of light in a moving medium is given by: u= c n +v 1 + v nc Substitutingc= 1into the formula: u= 1 n +v 1 + v n

  42. [50]

    Simplification: Simplify the expression by multiplying the numerator and denominator byn: u= 1 n +v n 1 + v n n = 1 +nv n+v

  43. [51]

    Understanding Method (Conceptual Insight vs. Formula Application)

    Relativistic Velocity Addition: Alternatively, using the relativistic velocity addition formula, whereu′ = 1 n is the speed of light in the glass frame moving at velocityv relative to the lab frame: u= u′ +v 1 +u′v Substitutingu′ = 1 n: u= 1 n +v 1 + v n This simplifies to the...

  44. [52]

    Backtracking

    The author starts by recalling foundational concepts related to light in a medium and how it relates to refractive index and motion. 2. They initially consider the implications of special relativity on the motion of light in a medium. 3. As they derive the formula for the spee...

  45. [53]

    Backtracking

    **Comfort with Ambiguity**: The presence of various interpretations throughout (e.g., the potential for a carboxylic acid, possible structures aligning with the degrees of unsaturation, and the consideration of different functional groups) reflects a methodical approach to mak...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.