{"id":"a206fed3-76cf-48f6-ab2e-d9180a39a33c","arxiv_id":"2501.12226","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CDW-CoT groups a reasoning dataset into clusters, learns a prompt distribution per cluster, and interpolates these distributions by embedding distance for each new query, reporting higher exact-match accuracy than three chain-of-thought baselines on six benchmarks.","lead":"This paper proposes CDW-CoT, a prompting method that clusters reasoning questions, learns a separate prompt distribution for each cluster, and blends those distributions for each new question based on its distance to cluster centers. The authors report accuracy gains over standard chain-of-thought methods on six benchmarks with LLaMA2 and LLaMA3.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may be an artifact of selecting temperature and pool size on the test sets, so the per-instance weighting is not yet evaluated out-of-sample.","rationale":"The reader's verdict of REJECT is appropriate, and I agree that the evidence as presented does not support the central claim. However, the single most load-bearing concern is not the interpolation assumption in Eq. (7); the ablation in Table 3 provides at least some empirical support for distance weighting, even without variance estimates. The more direct threat is the apparent selection of T and S on the same test sets used to report improvements. The sensitivity figures coincide with final test accuracies, indicating that at least four reported results are post-selection rather than out-of-sample. This alone invalidates the headline claim of consistent superiority, independent of how strong the underlying method might be. The proposed concrete test—selecting hyperparameters on the eval split and recomputing test accuracy—would settle whether the reported gains survive out-of-sample evaluation. Since this concern reinforces the reader's rejection rather than changing it, the verdict remains unchanged.","tokens_in":10154,"tokens_out":7220,"duration_ms":77112,"concrete_test":"Re-run the full CDW-CoT pipeline on all six datasets with T and S chosen exclusively on the eval split (with K fixed by a rule that does not use test labels), then report test accuracy for that fixed configuration. If the resulting MultiArith, CSQA, or StrategyQA accuracies fall below the baselines in Table 1, the claimed consistent improvement is an artifact of test-set tuning. Include the Automate-CoT baseline under the same eval-selected protocol to test the state-of-the-art claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in Table 1 requires that reported accuracies are out-of-sample evaluations of a fixed configuration. The paper does not satisfy this: the sensitivity analyses in 'Sensitivity Analysis of Temperature' and 'Impact of Pool Size on CDW-CoT' select T=0.3 and S=40 on the same datasets whose test accuracies are later reported. Specifically, Figure 4 shows MultiArith (LLaMA2-13B) reaching 85.56 at S=40, which is exactly the test-set value in Table 1; Figure 3 selects T=0.3 as optimal on StrategyQA and MultiArith, and that same T is then used in the headline table. Because the final numbers for MultiArith, CSQA, and StrategyQA are thereby tuned on the reporting test sets, the claim of consistent outperformance over baselines is not established for unseen instances. A validation-based hyperparameter protocol could change the ranking. The reader's Eq. (7) concern is real, but Table 3 gives the distance-weighting component some empirical support; the more immediate threat to the central claim is this test-set selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CDW-CoT, a chain-of-thought prompting method that clusters the training data, learns an optimized prompt probability distribution per cluster via a black-box policy-gradient update (referred to as BBPL), and at inference time constructs a per-instance prompt distribution as a temperature-scaled, distance-weighted combination of the cluster distributions. The method is evaluated on six reasoning datasets (CSQA, StrategyQA, Letter, Coin, MultiArith, AQuA) with LLaMA2-13B and LLaMA3-8B, reporting exact-match accuracies that claim consistent improvement over Zero-Shot-CoT, Auto-CoT, and Manual-CoT.","tokens_in":10365,"tokens_out":4771,"duration_ms":49639,"significance":"If the central empirical claim were properly supported, the idea of instance-adaptive prompt distributions would be a useful contribution to automatic chain-of-thought prompting: the ablation study (Table 3) gives some preliminary evidence that distance weighting and clustering each contribute to accuracy, and the prompt budget is kept small. However, the manuscript as written does not establish the headline claim of consistent out-of-sample improvements, because the key hyperparameters (temperature and pool size) appear to be selected on the same test sets used for the main results, and the reported accuracies come from a single run without statistical significance testing. The method description also contains internal inconsistencies in the optimization equations that prevent reproduction. The core idea is salvageable, but the current evidence is not sufficient.","major_comments":[{"comment":"The main results in Table 1 are not out-of-sample evaluations of a fixed configuration. In the section 'Sensitivity Analysis of Temperature', T=0.3 is selected using accuracy on StrategyQA and MultiArith, and in 'Impact of Pool Size on CDW-CoT', S=40 is selected on CommonsenseQA and MultiArith; these exact settings are then used for the headline results in Table 1. Concretely, Figure 4 reports MultiArith accuracy of 85.56 at S=40, which is identical to the Table 1 value, and Figure 3 selects T=0.3 on datasets that later appear in Table 1. This means the reported improvements for MultiArith, CSQA, and StrategyQA are at least partly tuned on the reporting test sets. The authors must either select hyperparameters on a held-out validation split that is disjoint from the final test set, or report results for configurations fixed before any test-set evaluation.","section":"Sensitivity Analysis of Temperature / Impact of Pool Size on CDW-CoT / Table 1"},{"comment":"All accuracy tables report a single run with no standard deviations, no number of seeds, and no significance tests. Some differences are very small, for example StrategyQA LLaMA3 (66.72 to 67.44, a 0.72-point increase) and CSQA LLaMA3 (69.57 to 72.15, a 2.58-point increase), yet the text describes them as consistent improvements. The central empirical claim requires either multiple random seeds with reported variance or a paired significance test (e.g., bootstrap over instances) to rule out noise. Please add these statistics or temper the claim accordingly.","section":"Experiments and Results / Table 1"},{"comment":"The optimization equations are internally inconsistent as written. Equation (1) defines δ = -1/p(i), a scalar quantity, but Eq. (2) then indexes it as δk,m,n and sets it to -δk,m,n if the prompt is selected and δk,m,n otherwise, without defining how the scalar δ becomes indexed or what the sign flip is supposed to accomplish. Equation (3) multiplies the loss deviation by δk, but δk has not been introduced. As written, this procedure cannot be implemented or checked. Please give a coherent derivation of the policy-gradient update, following the notation of the cited BBPL/BDPL references, or provide a precise algorithmic specification of the update.","section":"Optimizing Prompt Probability Distributions for Clusters, Eqs. (1)-(5)"},{"comment":"The core assumption of the method—that a test instance's optimal prompt distribution is the convex combination of per-cluster prompt distributions weighted by a temperature-scaled softmax of Euclidean distances to cluster centers—is asserted without empirical or theoretical justification. The ablation in Table 3 shows that distance weighting outperforms nearest-cluster assignment on these datasets, which is supportive, but it does not validate the specific functional form of Eq. (7), particularly for instances near cluster boundaries where the interpolation is most sensitive to T. Please provide an analysis of the learned weights, a comparison to alternative interpolation schemes, or at least a validation-based sensitivity analysis with error bars.","section":"Distance-Weighted Prompt Selection and Inference, Eq. (7)"}],"minor_comments":[{"comment":"The pre-trained sentence transformer used for embeddings is never named; please identify the exact model so the experiments are reproducible.","section":"Cluster-Based Prompt Candidate Pool Initialization"},{"comment":"The number of clusters is manually adjusted per dataset (e.g., from the default 8 to 3 for Coin and MultiArith), but the criterion for adjustment is not stated. Please clarify whether these choices were made on a validation split and report their sensitivity.","section":"Data Split and Number of Clusters Identification, Table 2"},{"comment":"The claim that CDW-CoT uses 'no more prompts than baselines' cannot be verified because the prompt counts for Zero-Shot-CoT, Auto-CoT, and Manual-CoT are not reported. Please provide the number of prompts used for each baseline.","section":"Models and Baselines"},{"comment":"Several baseline numbers are unusually low compared with published results for these models and datasets (e.g., Auto-CoT MultiArith 44.17 and Manual-CoT Letter 15.80 on LLaMA2-13B). Please clarify the evaluation settings or report these baselines with the same decoding and prompt-count conditions as the proposed method.","section":"Experiments and Results"},{"comment":"Figures 3 and 4 report accuracies without error bars or repeated runs, so it is difficult to assess whether the reported trends are stable; please add multiple runs and variance information.","section":"Sensitivity Analysis of Temperature / Impact of Pool Size on CDW-CoT"},{"comment":"There is a typo in the description of Auto-CoT ('clustering similar questions and and using'), and the acronyms BDPL and BBPL are used inconsistently; please unify the terminology.","section":"Related Works and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The test-set hyperparameter selection is the most serious issue and will require re-running the experiments with a proper validation protocol. If the authors do so, the paper could become publishable, but as it stands the central claim is not supported. I would also encourage the editor to require code or detailed hyperparameter settings, since the method description currently lacks enough detail to be reproduced."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is reasonable and genuinely a small step beyond Auto-CoT/Automate-CoT: instead of one prompt distribution for the whole dataset, cluster the training set, learn a per-cluster prompt distribution, then at inference blend the per-cluster distributions with a softmax over distances to cluster centers. The ablation in Table 3 gives some support to the distance-weighting component (Dist-W beats Near-C on five of six datasets, and beats No-Clust everywhere). That part is the paper's best evidence and it is worth taking seriously.\n\nThe problems are all in the evaluation. Table 1 reports single-run exact-match accuracies with no standard deviation, no seeds, no significance tests. Some baseline numbers are oddly low (Auto-CoT at 44.17 on MultiArith for both LLaMA2 and LLaMA3, when the published Auto-CoT numbers are typically higher), and the most relevant baseline, Automate-CoT, is missing even though it is cited in the intro. The optimization equations are also sloppy as written: Eq. (2) is inconsistent with Eq. (1) in the sign conventions, and Eq. (3)'s gradient term is not derived clearly. That is fixable but it makes the method hard to reproduce from the text alone.\n\nThe bigger issue, which the stress-test note gets right, is that the temperature and pool size are selected on the same datasets whose test accuracies are then reported. Figure 4 shows MultiArith (LLaMA2-13B) hitting 85.56 at S=40, exactly the value in Table 1; Figure 3 picks T=0.3 on StrategyQA and MultiArith, and that same T is then used in the headline table. So the final numbers for MultiArith, CSQA, and StrategyQA are not honest out-of-sample evaluations of a fixed configuration. The per-cluster distributions are trained on the train split, but the configuration itself is tuned on the test split. A validation-based protocol could change the ranking, especially for the smaller gains (StrategyQA LLaMA3, for instance, is only +0.72).\n\nI also share the reader's concern about Eq. (7): the paper gives no evidence that convex combination in SentenceTransformer embedding space tracks prompt usefulness, and the sentence-transformer model is unnamed, so the whole distance-weighting mechanism is underspecified. The ablation helps, but it is on the same test sets and still lacks variance.\n\nBottom line: the paper proposes a plausible incremental method, but the evidence as presented does not support the claim of consistent outperformance. It deserves a referee because the idea is not a non-starter and the ablation suggests something real is happening, but it needs a code release, multiple seeds, validation-based hyperparameter selection, and a proper comparison to Automate-CoT before the claim can be trusted.","headline":"Plausible incremental method, but the headline numbers are tuned on the test sets and the evaluation section has no variance estimates, so the central claim is not yet supported.","tokens_in":10883,"tokens_out":698,"would_cite":false,"duration_ms":7950,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CDW-CoT claims that per-instance prompt distributions built from cluster-specific optimized prompts consistently beat fixed CoT prompts across six reasoning benchmarks.","keywords":["chain-of-thought prompting","prompt optimization","clustering","distance-weighted inference","large language models","reasoning benchmarks","black-box prompt learning","LLaMA"],"falsifier":"Swap the distance-weighted blend $p = \\sum_i w_i p^{(i)}$ for a uniform blend of the same per-cluster distributions while holding clusters, pool, temperature, and sampling fixed; if uniform blending matches or beats distance weighting on the six benchmarks, the claim that proximity to cluster centers drives the improvement is falsified.","tokens_in":9940,"feed_emoji":"🎯","tokens_out":10369,"duration_ms":85945,"temperature":0.7,"pith_summary":"CDW-CoT aims to show that the common practice of using one prompt set for an entire dataset is a bottleneck for chain-of-thought reasoning. The method clusters the training questions, builds a candidate pool of prompts per cluster, learns an optimal prompt probability distribution for each cluster, and at inference blends those distributions according to a test question's distance to cluster centers. The paper reports that this consistently beats Zero-Shot-CoT, Auto-CoT, and Manual-CoT on six commonsense, symbolic, and mathematical reasoning datasets, with average gains over Manual-CoT of 25.34% on LLaMA2-13B and 15.72% on LLaMA3-8B while using no more prompts than the baselines. If true, it means meaningful per-instance adaptation can be obtained from a fixed prompt pool plus cheap distance-weighted averaging, without per-example prompt search.","feed_headline":"Distance-weighted prompts beat fixed chain-of-thought by up to 25%","feed_subtitle":"Clustering plus cluster-specific prompt distributions lifts reasoning accuracy on six benchmarks for LLaMA2 and LLaMA3.","key_machinery":"The carrying object is the distance-weighted prompt probability distribution. The candidate pool is initialized from the questions closest to each K-means cluster center, and each cluster $i$ is assigned an optimized probability vector $p^{(i)}$ over that pool, trained by black-box prompt learning. For a test instance, the effective distribution is $p = \\sum_{i=1}^{K} w_i p^{(i)}$, with weights $w_i = \\exp(-d_i/T) / \\sum_j \\exp(-d_j/T)$, where $d_i$ is the Euclidean distance to cluster center $i$ and $T$ is a temperature. This identity converts geometric proximity in embedding space into a per-instance blend of cluster-specific prompt strategies; the paper uses $T=0.3$ and a pool size of 40 for its main runs.","core_discovery":"The central claim, stated on the paper's own terms, is that a single global prompt distribution leaves accuracy on the table, while a prompt distribution optimized per cluster and then combined for each test instance by distance weighting does not. The paper reports exact-match accuracy gains on all six datasets for both LLaMA2-13B and LLaMA3-8B; the largest single jump is Letter, which rises from 30.80% to 82.67% on LLaMA2. The ablation results attribute part of the gain to clustering and part to distance weighting, with the full model beating both a nearest-cluster assignment and a no-clustering global distribution on every dataset.","pith_inferences":["Not tested in the paper: comparing Eq. (7)'s distance-weighted blend against a uniform blend of the same per-cluster distributions would isolate whether proximity to cluster centers, rather than mixing itself, drives the gains.","An implicit bet is that the sentence-transformer embedding space aligns with prompt usefulness; swapping the embedder and measuring accuracy drift would test whether the clusters, not the downstream model, carry the signal.","Because cluster distributions are optimized independently, the method could be extended to streaming data by updating only the distributions of clusters that receive new instances.","Boundary instances are where the blend is most sensitive to $T$; a testable extension is to set $T$ per instance from the entropy of the distance weights."],"forward_implications":["Prompt count no longer has to scale with instance count: a fixed candidate pool plus per-cluster distributions serves an entire dataset.","Per-cluster optimization can be done offline, and inference adds only an embedding lookup and a weighted average, so the same training effort supports any number of test instances.","Datasets with heterogeneous subpopulations benefit most; the Letter jump from 30.80% to 82.67% on LLaMA2 suggests symbolic tasks are especially sensitive to prompt-instance matching.","Temperature and pool size are material hyperparameters: the paper finds best accuracy at $T=0.3$ and pool size 40, so deployments should tune them rather than defaulting.","Gains appear on both LLaMA2-13B and LLaMA3-8B, suggesting the mechanism is not tied to one model family."],"supporting_citations":[{"why":"introduces Manual-CoT, the main comparison baseline, and supplies the Letter and Coin symbolic tasks.","marker":"(Wei et al. 2022)"},{"why":"defines Zero-Shot-CoT and the \"let's think step by step\" prompt used to generate prompt candidates.","marker":"(Kojima et al. 2022)"},{"why":"defines Auto-CoT, the automatic baseline CDW-CoT adapts, including default cluster counts.","marker":"(Zhang et al. 2022)"},{"why":"supplies the black-box prompt learning procedure that optimizes each cluster's prompt distribution.","marker":"(Diao et al. 2022)"},{"why":"provides LLaMA2-13B, one of the two models on which CDW-CoT is evaluated.","marker":"(Touvron et al. 2023)"},{"why":"provides LLaMA3-8B, the second model on which CDW-CoT is evaluated.","marker":"(Dubey et al. 2024)"},{"why":"provides the CommonsenseQA benchmark used in the commonsense reasoning evaluation.","marker":"(Talmor et al. 2018)"},{"why":"provides the StrategyQA benchmark used in the commonsense reasoning evaluation.","marker":"(Geva et al. 2021)"},{"why":"provides the MultiArith benchmark used in the mathematical reasoning evaluation.","marker":"(Roy and Roth 2016)"},{"why":"provides the AQuA benchmark used in the mathematical reasoning evaluation.","marker":"(Ling et al. 2017)"}],"fun_headline_variants":["Cluster-aware prompts beat static CoT by 25% on reasoning","Distance-weighted prompt picks lift CoT accuracy up to 25%","Adaptive prompt distribution boosts reasoning on six datasets","Per-cluster prompt tuning improves chain-of-thought by 25%","Why fixed prompts underperform: clustered CoT gains 25%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ideal prompt for a question can be approximated by a weighted average of the prompt distributions of nearby clusters, with weights determined by embedding distance; if embedding distance does not track prompt usefulness, the reported gains may come from per-cluster optimization rather than from distance weighting.","fun_headline_variants_meta":{"raw":{"variants":["Cluster-aware prompts beat static CoT by 25% on reasoning","Distance-weighted prompt picks lift CoT accuracy up to 25%","Adaptive prompt distribution boosts reasoning on six datasets","Per-cluster prompt tuning improves chain-of-thought by 25%","Why fixed prompts underperform: clustered CoT gains 25%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1313,"prompt_tokens":938,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":287}},"tokens_in":554,"tokens_out":375,"duration_ms":4612,"temperature":1.0,"reasoning_tokens":287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:22:37.769781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swap the distance-weighted blend $p = \\sum_i w_i p^{(i)}$ for a uniform blend of the same per-cluster distributions while holding clusters, pool, temperature, and sampling fixed; if uniform blending matches or beats distance weighting on the six benchmarks, the claim that proximity to cluster centers drives the improvement is falsified.","supporting_citations":[],"review_version":1}