{"id":"fd26201b-c086-4ae7-81ef-511a0c2e4a2b","arxiv_id":"2505.10185","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A bottom-up, clustering-based taxonomy of LLM reasoning strategies enables predicting and steering model behavior, with training data format mattering more than domain.","lead":"This paper introduces a bottom-up framework that automatically discovers the reasoning strategies hidden inside long chain-of-thought traces of large language models. It shows these strategies can be predicted and steered to improve accuracy and safety, and that training data format shapes reasoning more than content domain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'optimal strategy' label is inferred from correct responses rather than causal intervention; without held-out selection and a manipulation check, the reported accuracy gains may reflect in-sample fitting or generic instruction-following.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: optimal strategies are defined by correlation with correctness, not by causal intervention. This is the right point to stress because the paper's headline contribution is the claim that controlling high-level reasoning strategies directly enhances accuracy, and that claim depends on the labels being causally valid and on the prompting effect being attributable to strategy rather than to confounds. The paper has real strengths: the bottom-up taxonomy is human-evaluated, the format-versus-domain finding comes from controlled RLVR training, and the cross-domain classifier is a plausible generalization check. However, those strengths do not settle the causal question. The in-sample selection of dataset-wide optimal patterns in §4.1 is particularly risky because the evaluation split is used to choose the very pattern that is then applied to that split. The question-specific classifier is more convincing, but its training labels are still derived from correct responses on a different split, so the same confounding can propagate. A concrete held-out evaluation with a length-matched control and a manipulation check would either validate the central claim or show that the gains are generic instruction-following. Since the reader already assigns CONDITIONAL, this stress-test does not change the verdict, but it sharpens the condition under which acceptance would be justified.","tokens_in":28913,"tokens_out":4699,"duration_ms":50686,"concrete_test":"Re-run the §4.3 pipeline on a question-level train/test split: select optimal patterns (dataset-wide and per-question classifier) using only the training split, then evaluate on the held-out split. Add two controls: (a) a length-matched generic 'think carefully' prompt, and (b) an instruction to use the predicted strategy while verifying via an independent rubric and human annotation that the final response actually exhibits the strategy. Report accuracy gains and output-length differences for each condition. If gains disappear or match the generic prompt, the strategy-control claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim in §4 rests on labeling as 'optimal' the strategy observed in responses that happened to be correct. §4.1 computes P(Correct | Pattern) over generated responses and selects dataset-wide optimal patterns; §4.3 trains question-specific predictors using 'questions initially answered correctly to derive optimal strategies.' This is a correlational definition: correctness and strategy share common causes such as question difficulty, model knowledge, and output length. Prompting a model to adopt the pattern associated with correct responses is not equivalent to intervening on strategy while holding other factors fixed. The comparisons against 'random' and 'unoptimal' patterns do not control for instruction length or for the possibility that the chosen pattern simply encourages more verbose, exploratory reasoning that increases test-time compute. Moreover, the dataset-wide optimal pattern in §4.1 is selected on the same benchmark responses used for evaluation, so the reported gain may be in-sample selection. No manipulation check is reported to verify that prompted responses actually exhibit the intended strategy rather than merely echoing the instruction. The cross-domain classifier in §4.3 is a stronger design, but its training labels are still correlation-derived; if the pattern-correctness association is confounded by question difficulty, the predicted 'optimal' strategy for hard questions may be wrong.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces the CoT Encyclopedia, a bottom-up framework that extracts contrasting reasoning criteria from long chain-of-thought traces via GPT-4o, embeds and clusters them into high-level dimensions, generates interpretable rubrics, and classifies new responses. The authors validate the taxonomy with human evaluation and then use it for two further purposes: predicting each question's 'optimal' reasoning strategy and steering models toward it by prompting, and analyzing how RLVR training-data format versus domain shapes reasoning strategies, including model-merge interpolation. On the control side, the paper reports accuracy and safety improvements from dataset-wide and question-specific optimal patterns across five benchmarks. On the analysis side, it reports that training format (multiple-choice versus free-form) has a much larger effect on reasoning strategies than training domain, with model merging producing smooth interpolations between formats.","tokens_in":29080,"tokens_out":5643,"duration_ms":57302,"significance":"If the control claims hold, the paper offers a practical method for steering long-CoT reasoning and a taxonomy that is more sensitive to model differences than predefined cognitive behaviors. The human evaluation is a genuine strength, as are the controlled RLVR experiments in Section 5, which carefully separate format from domain, and the cross-domain classifier design in Section 4.3. The model-merging interpolation results are also interesting and do not depend on the correlational label issue that affects the control experiments. The main weakness is that the central 'optimal strategy' notion is defined by correlation with correctness rather than by causal evidence; this limits the strength of the accuracy-control claims and the novelty assertion attached to them.","major_comments":[{"comment":"The definition of an 'optimal strategy' as the reasoning pattern observed in responses that are correct is correlational and is load-bearing for the entire control claim. In §4.1 the paper computes P(Correct | Pattern) from generated responses, and in §4.3 it trains the strategy predictor using 'questions initially answered correctly to derive optimal strategies.' Correctness and strategy share likely common causes (question difficulty, model knowledge, output length), so a pattern associated with correct answers need not be the cause of correctness. Prompting a model to adopt that pattern therefore does not establish direct control over reasoning quality. To support the causal claim, the authors should (i) select optimal patterns on a held-out development set, (ii) report a manipulation check showing that prompted responses actually increase the frequency of the intended rubric pattern relative to a no-instruction baseline, and (iii) include a control that instructs a matched-length, task-relevant but non-strategy behavior to exclude generic instruction-following.","section":"§4.1, §4.3 (optimal strategy labels)"},{"comment":"The dataset-wide optimal patterns in §4.1 appear to be selected from the same benchmark responses on which the improvement is then measured: P(Correct | Pattern) and P(Safe | Pattern) are computed over the test sets of GPQA-Diamond, MMLU-Redux, MATH-500, XSTest, and WildGuard, and Figure 4 reports 'optimal-dataset' results on those same benchmarks. This is an in-sample selection procedure, so the reported gains may reflect selection bias rather than a transferable property of the strategy. The paper should either move pattern selection to a separate validation set or evaluate the full pipeline cross-validated, and it should report how many dataset-wide patterns were examined and whether any multiple-comparison correction was applied.","section":"§4.1 (dataset-wide optimal pattern selection)"},{"comment":"The prompting comparisons do not control for instruction length or for the amount of generated text. The optimal-strategy prompt in Appendix E contains six rubric descriptions and instructs the model to align its entire process, while the 'no instruction' baseline is a short task prompt; the 'random' and 'unoptimal' conditions also differ in content and length from the optimal condition. If the optimal instruction causes longer, more exploratory CoTs, part or all of the measured accuracy/safety gain could be test-time compute or verbosity rather than the specific strategy. The paper should measure output length in each condition, include a token-length-matched neutral instruction, and verify through the framework's own rubrics that prompted responses follow the intended pattern rather than only echoing the instruction.","section":"§4.1, §4.3 (confounds and manipulation check)"},{"comment":"The abstract and §4.3 claim that the method enables real-time adaptive control and, to the authors' knowledge, is the first demonstration that controlling high-level reasoning strategies directly enhances accuracy. Given the correlational definition of 'optimal' and the absence of a manipulation check, this is broader than the evidence supports. A more defensible claim is that prompting with strategies associated with correct responses on a training split improves accuracy on held-out questions; the causal interpretation should be stated as an open question or supported by the additional experiments above. In addition, the sentence in §4.3 reporting 'accuracy gains of 81.0%, 92.3%, and 80.7%' appears to report post-control accuracy rates, not gains; please reword to distinguish final rates from gains relative to the no-instruction baseline, and reconcile these values with the ranges in Figure 13.","section":"Introduction, §4.3 (scope of the control claim)"}],"minor_comments":[{"comment":"The text describes a 'regression analysis' between question similarity and strategy similarity, but only a correlation or R² is mentioned; please report the fitted slope, confidence intervals, and the number of pairs used so the reader can assess the predictive claim.","section":"§4.2"},{"comment":"Several p-values are reported as '0.00'; please report exact values or use a format such as p < 0.001, since a p-value cannot be exactly zero.","section":"Tables 5 and 10"},{"comment":"The human evaluation uses 100 responses and four annotators with binary questions, but no inter-annotator agreement statistic is reported; adding Cohen's kappa or per-annotator consistency would strengthen the interpretability claim.","section":"Appendix A.3"},{"comment":"There are several typographical and formatting issues: 'NA VER' in the author block, 'LLMo' in §3.2, 'Wild guard' in Figure 11, and the incomplete reference URL for NuminaMath in [10]; these should be cleaned up in revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the taxonomy-building contribution is solid and well within the journal's scope, and the RLVR format-versus-domain analysis is a strong, controlled result. My main reservation is the causal language around 'optimal' strategies in Section 4; this is fixable with a validation split, a manipulation check, and length-matched controls, so I am recommending major revision rather than rejection. I would also encourage the authors to separate the interpretability contribution (which is largely supported) from the control contribution in the revised framing, and to temper the 'first demonstration' novelty claim unless the additional controls are provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, the paper's backbone—a bottom-up clustering of model-generated reasoning criteria into contrastive rubrics—is genuinely new, and the human evaluation suggests it captures reasoning differences that predefined categories miss. Second, the headline \"we can control how a model thinks and improve accuracy\" rests on a correlational definition of \"optimal\": a strategy is optimal because it appears in responses that were correct. That is not a causal result, and the paper sometimes writes as if it were. The gap is real but fixable, and the rest of the paper is strong enough that it deserves serious refereeing.\n\nWhat's actually new: the five-stage pipeline (criteria identification, embedding, hierarchical clustering to six medoid criteria, rubric generation, report generation) produces a taxonomy with contrastive rubrics like Top-down vs. Bottom-up, Focused vs. Broad. Human annotators judge it reasonable at 92–97%, versus 51% for the predefined behaviors from Gandhi et al. That result alone is a useful contribution. The format-vs-domain experiments are also well designed: they hold content fixed by converting NuminaMath into multiple-choice form, train 7B models with RLVR, and show format moves reasoning patterns far more than domain (Cohen's d up to ~1.5 vs <0.2). The model-merging interpolation is a nice bonus.\n\nWhere the soft spots are, in proportion: the central performance-control claim in §4 has a load-bearing weakness. P(Correct | Pattern) is a correlation, and correctness and strategy share common causes—question difficulty, model knowledge, output length, test-time compute. Prompting a model to adopt the \"optimal\" pattern may just make it more verbose or more exploratory; there is no manipulation check that the prompt actually changes the strategy rather than echoing the instruction. The dataset-wide optimal pattern in §4.1 is selected on the same benchmarks used for evaluation, so some of the gain could be in-sample selection. The cross-domain classifier in §4.3 is a better design—trained on correct responses, tested on initially incorrect ones—but the training labels still come from the same correlation. No error bars are reported, and no code or data is released. The paper's own Limitations section acknowledges the GPT-4o reliance and the dependence on instruction-following, but it doesn't confront the correlation–causation issue directly.\n\nNone of this kills the taxonomy or the format finding. But the accuracy-gain claim, as presented, needs stronger evidence: a manipulation check that prompted responses actually exhibit the intended pattern, controls for verbosity and sampling compute, and ideally a train/test split for the dataset-wide patterns too.\n\nMy take: bring it to the reading group, cite it for the bottom-up taxonomy and the format effect, and send it to peer review with a request for causal validation and code release. I would not let the current §4 headline stand without revision.","headline":"A genuinely bottom-up taxonomy of reasoning strategies, but the headline accuracy-gain claim rests on a correlational definition of 'optimal' that needs a manipulation check before it carries the weight the paper puts on it.","tokens_in":29685,"tokens_out":3555,"would_cite":true,"duration_ms":33452,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The reasoning strategies inside long chain-of-thought traces can be extracted bottom-up from the model's own outputs, predicted from the question alone, and steered by prompting to improve accuracy and safety.","keywords":["chain-of-thought","reasoning strategies","bottom-up taxonomy","strategy prediction","prompt steering","training data format","model merging","LLM safety"],"falsifier":"Run the paper's control pipeline on a benchmark, then compare the predicted-optimal prompt against a placebo prompt of equal length and format that describes a reasoning style never observed in any correct response, such as a strategy invented by shuffling rubric poles. If the placebo matches the optimal prompt's accuracy, the specific strategy content is not what drives the gain; the causal claim survives only if the optimal prompt wins consistently across benchmarks and models.","tokens_in":28679,"feed_emoji":"🧠","tokens_out":13053,"duration_ms":107249,"temperature":0.7,"pith_summary":"Long chain-of-thought (the intermediate reasoning a model writes before answering) is not an opaque monolith, this paper argues: it decomposes into identifiable strategies—top-down versus bottom-up, hypothesis-driven versus data-driven, iterative versus immediate—that a pipeline can extract directly from model outputs without any human-predefined category. The paper claims these discovered strategies distinguish models far more sharply than standard predefined cognitive behaviors, can be predicted from the input question alone, and can be used as prompts: telling a model to adopt the strategy most likely to produce a correct answer improved accuracy and safety on five benchmarks, with gains of 2.5–8.3% across three reasoning models. A second set of controlled training experiments claims that the format of the training data—multiple-choice versus free-form—shapes reasoning strategies with effect sizes up to 1.5, far more than the data's domain ($d < 0.2$), and that linear weight interpolation between format-trained models smoothly interpolates their reasoning styles. If these claims hold, reasoning analysis becomes a real-time control knob for accuracy, safety, and training-data design rather than a post-hoc description.","feed_headline":"Steering a reasoning model's strategy boosts accuracy and safety","feed_subtitle":"A bottom-up map of chain-of-thought traces lets researchers predict and prompt better strategies on five benchmarks.","key_machinery":"The load-bearing object is the CoT Encyclopedia taxonomy itself, built in five stages: (1) an LLM generates free-form contrastive reasoning criteria from raw chain-of-thought responses; (2) each criterion's text is embedded into a semantic space; (3) hierarchical agglomerative clustering with cosine distance compresses thousands of criteria into six interpretable dimensions, each represented by its medoid; (4) an LLM writes detailed contrastive rubrics for each dimension; and (5) each response is classified under every rubric, producing a binary strategy profile. The control claim rides on a second mechanism: a classifier that maps a question to its predicted strategy profile, combined with a Bayes-rule estimate $P(\\text{Correct} \\mid \\text{Pattern})$ computed from responses that were initially correct, so the prompt can prescribe the strategy with the highest conditional success probability. The predictive claim is anchored by an $R^2 = 0.405$ correlation between question similarity and strategy similarity, used to justify transferring strategy labels from similar previously seen questions to unseen ones.","core_discovery":"The paper's central claim is that the reasoning strategies behind a model's long chain-of-thought can be discovered automatically, named, predicted, and controlled. The CoT Encyclopedia pipeline prompts an LLM to brainstorm contrastive reasoning criteria from model-generated traces (4,057 criteria such as \"Top-down vs. Bottom-up\"), embeds them in a semantic space, clusters them into six high-level rubrics (analytical perspective, scope of approach, reasoning type, idea development, verification focus, clarification approach), and classifies each response under each rubric. Human judges found the resulting analyses reasonable in 92–97% of cases, versus 51% for a predefined four-behavior taxonomy. Using these rubrics, the paper computes $P(\\text{Correct} \\mid \\text{Pattern})$ for each strategy pole, trains a classifier that predicts the strategy a model will use from the question alone, and prompts the model to follow the strategy with the highest estimated success probability; across five benchmarks this improves accuracy and safety for three reasoning models (GPQA-Diamond accuracy 72.7% to 79.1%; XSTest safety 91.1% to 94.0%; WildGuard 89.1% to 92.9%). The paper presents this as, to its knowledge, the first demonstration that controlling a model's high-level reasoning strategy directly enhances accuracy. It further claims that training data format, not domain, dominates the reasoning style a model adopts, and that model-weight interpolation between format-trained models interpolates strategies without further training.","pith_inferences":["Because 'optimal' strategies are defined as the patterns found in responses that happened to be correct, the control gains rest on a correlation-is-causation step; a direct test would pit the predicted-optimal prompt against a matched placebo instruction describing a plausible strategy never observed in a correct response.","The $R^2 = 0.405$ ceiling implies that for question types far outside the training distribution, the predicted strategy may be near-random; the method's practical reach is bounded by how well the benchmark distribution covers the deployment space.","The format effect suggests a training-time corollary the paper does not pursue: deliberately mixing formats within one curriculum could cultivate hybrid reasoning styles directly, rather than only merging already-trained models.","Because the strategy predictor works from the question alone, a natural untested deployment is mid-generation intervention: classify the emerging trace, detect a known-bad pattern, and inject a corrective prompt before the final answer."],"forward_implications":["Prompting a model with its predicted optimal strategy improves accuracy by 2.5–8.3% across three reasoning models and improves safety on XSTest and WildGuard (e.g., GPQA-Diamond accuracy rises from 72.7% to 79.1%).","The bottom-up taxonomy distinguishes models more finely than predefined cognitive behaviors (Cohen's $d$ up to 0.44, versus roughly 0.1 for the baseline), giving a more sensitive instrument for comparing reasoning models.","Training data format shifts reasoning strategy with effect sizes up to 1.5, so choosing multiple-choice versus free-form data is a deliberate lever over the model's later reasoning style rather than a neutral packaging choice.","Linear weight interpolation between format-trained models produces smooth strategy transitions without fine-tuning, a cheap way to tune reasoning behavior toward task needs.","Because similar questions elicit similar strategies ($R^2 = 0.405$), strategy classifiers trained on one benchmark transfer to others, supporting question-level adaptive prompting at inference time."],"supporting_citations":[{"why":"Supplies the top-down four-behavior baseline (verification, backtracking, subgoal setting, backward chaining) that the bottom-up taxonomy must beat in interpretability and sensitivity.","marker":"[5]"},{"why":"Provides the DeepSeek-R1 long-CoT context and the GRPO objective used to train the 7B models in the format-versus-domain experiments.","marker":"[6]"},{"why":"NuminaMath is the free-form math dataset that the paper converts into multiple-choice format to isolate the effect of presentation format from content.","marker":"[10]"},{"why":"s1: Simple test-time scaling produces one of the three reasoning models (s1.1-32B) whose strategies are analyzed and compared.","marker":"[15]"},{"why":"Supplies the hierarchical agglomerative clustering algorithm used to compress thousands of extracted criteria into six rubrics.","marker":"[16]"},{"why":"XSTest is one of the two safety benchmarks on which strategy-guided prompting is evaluated for harmlessness.","marker":"[17]"},{"why":"WildGuard is the second safety benchmark and provides the basis for safe/unsafe classification of responses.","marker":"[7]"},{"why":"Arena-Hard is the instruction-following benchmark used to characterize the differing reasoning styles of multiple-choice versus free-form trained models.","marker":"[11]"},{"why":"Chain-of-thought prompting is the object under study; the paper analyzes and steers the reasoning that this technique elicits.","marker":"[22]"}],"fun_headline_variants":["Reasoning strategies mapped, predicted, and steered for better accuracy","Training format shapes reasoning style more than data domain","CoT encyclopedia predicts and steers model thought for gains","Automatic map of reasoning strategies boosts accuracy and safety","Steer a reasoning model's thought by mapping its CoT patterns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper labels as optimal whatever reasoning pattern appears in responses that happened to be correct, then steers the model toward that pattern; if a pattern is merely correlated with correctness rather than the cause of it, the reported prompting gains could reflect generic instruction-following instead of genuinely better reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning strategies mapped, predicted, and steered for better accuracy","Training format shapes reasoning style more than data domain","CoT encyclopedia predicts and steers model thought for gains","Automatic map of reasoning strategies boosts accuracy and safety","Steer a reasoning model's thought by mapping its CoT patterns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000574,"raw_usage":{"total_tokens":2762,"prompt_tokens":1049,"completion_tokens":1713,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":1631}},"tokens_in":665,"tokens_out":1713,"duration_ms":12159,"temperature":1.0,"reasoning_tokens":1631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:14:05.925028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's control pipeline on a benchmark, then compare the predicted-optimal prompt against a placebo prompt of equal length and format that describes a reasoning style never observed in any correct response, such as a strategy invented by shuffling rubric poles. If the placebo matches the optimal prompt's accuracy, the specific strategy content is not what drives the gain; the causal claim survives only if the optimal prompt wins consistently across benchmarks and models.","supporting_citations":[],"review_version":1}