{"id":"fb6e1043-adc9-4282-825f-635036008ae1","arxiv_id":"2607.14386","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A decoupled exploration-selection framework for test-time scaling lifts data-science agent accuracy by up to roughly 12 percentage points over matched baselines.","lead":"CIPHER is an AI data-science agent that generates many candidate plans, picks a subset to run in parallel, and combines the results, beating existing agents on two benchmarks. It offers an early map of which exploration and selection choices matter when you scale AI agents with extra computation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DES gains are not empirically separated from more compute plus a stronger aggregator: no equivalent-token repeated-sampling baseline is run, so the central causal claim rests on an untested mechanism.","rationale":"The reader's weakest assumption correctly identifies the missing compute-matched baseline as load-bearing; I agree with the CONDITIONAL verdict. I focused on the compute-matched baseline because it is the minimal control that would distinguish DES from ordinary repeated sampling with a stronger aggregator. The paper is transparent about token usage and provides extensive ablations, but Appendix B's explicit admission means the central causal claim is a plausible conjecture rather than a measured effect. The secondary test-set-selection issue is real and could bias the specific design recommendation (Ensemble+Goal-Align), but it does not by itself undermine the existence of a compute-scaling effect. Since the reader's verdict already conditions on these concerns, my analysis does not move the verdict; it strengthens the reason for the condition. I set agreement_with_reader to 'partial' because the reader also cited test-set peeking, which I treat as secondary rather than the primary load-bearing concern.","tokens_in":18468,"tokens_out":6311,"duration_ms":66741,"concrete_test":"Implement a repeated-sampling control: run R = 5 independent CIPHER(1,1) executions using the same base prompt, temperature 0.5, and coding loop, with total input tokens approximately matching CIPHER†(10,5) on each benchmark (86K on Infi-DA-Bench, 102K on InsightBench). Aggregate the five final answers with the same Sonnet-3.7 leader prompt used in CIPHER†, and evaluate on the same 3 seeds under the official Infi-DA-Bench and InsightBench metrics. Compare mean accuracy against CIPHER†'s 81.06% and 49.53%. If the repeated-sampling control is within ~1pp, the DES decoupling contributes little beyond repeated sampling plus a stronger aggregator; if it trails by more than ~3pp, the current conclusion is materially supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that decoupling exploration (generating N plans) from selection (executing M) is what drives CIPHER's gains. The evidence for this specific mechanism is incomplete. The non-DES comparators are single-execution baselines using far fewer tokens (Table 4: DataWise uses 16K input tokens vs. CIPHER†(10,5) at 86K), and Appendix B explicitly concedes: 'A fully compute-matched baseline (e.g., five independent CIPHER(1,1) executions aggregated via majority voting) is not evaluated here.' The paper's rebuttal—that CIPHER's gains come from structurally different plans rather than repeated sampling—is asserted, not measured. Moreover, Section 5.1 reports that the generation-strategy advantage emerges only under leader aggregation; under self-aggregation, the Friedman test is non-significant (χ²=0.88, p=0.65). This is exactly the pattern expected if the gains come from more executions plus a stronger aggregator, not from the exploration-selection decoupling. Without a repeated-sampling control at equal inference budget, the +7.0pp (Infi-DA-Bench) and +10.9pp (InsightBench) gains cannot be attributed to DES as opposed to additional compute and a more capable aggregator. A secondary concern is that the recommended configuration is selected on the same test panels where it is then reported (Section 5.2 vs. 5.3), which may inflate the reported numbers, but the compute-matched baseline is the more decisive missing control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CIPHER, a data science agent that applies test-time scaling via Decoupled Exploration-Selection (DES): it first generates N candidate initial plans, selects M for parallel execution, and aggregates the results. The framework is instantiated with three generation modes (base/independent, conditional, ensemble), four selection strategies (random, maxent, clustering, goal-align), and two aggregation modes (self vs leader). On Infi-DA-Bench and InsightBench, the paper reports that the recommended configuration CIPHER†(10,5) (ensemble generation, goal-align selection, leader aggregation) improves over the matched-model single-agent baselines: 81.06% vs 74.06% on Infi-DA-Bench and 49.53% vs 38.63% on InsightBench. The paper also presents an extensive ablation of 24 configurations across two budgets and analyzes generation/selection/aggregation choices, with statistical tests and token-usage reporting.","tokens_in":18856,"tokens_out":7056,"duration_ms":67925,"significance":"The empirical measurements are valuable: 357 tasks, two benchmarks, 24 configurations, three seeds, and significance tests. The paper is candid about limitations, including the absence of a compute-matched repeated-sampling baseline (Appendix B). However, this missing control is decisive for the central causal claim. The results clearly show that spending more inference compute (10 plans, 5 executions, leader aggregator) improves accuracy over single-execution baselines, but they do not show that the decoupling of exploration and selection—as opposed to merely sampling more executions and using a stronger aggregator—is responsible. If the control were added and showed equal accuracy at equal tokens, the DES contribution would reduce to a compute-allocation strategy. The design-space analysis (entropy measurements, Friedman tests) is useful but does not substitute for this control.","major_comments":[{"comment":"The central claim that the DES decoupling drives the reported gains is not supported. The non-DES baselines use substantially fewer input tokens (DataWise 16K, Agent-Poirot 51K) than CIPHER† (86K and 102K on the two benchmarks). Appendix B explicitly states that a fully compute-matched baseline (e.g., five independent CIPHER(1,1) executions aggregated via majority voting) is not evaluated. The rebuttal that an equivalent-token replay would not exploit plan diversity is an assertion, not a measurement. Furthermore, the Friedman test in §5.1—generation strategy is non-significant under self-aggregation and significant only under leader aggregation—is consistent with a 'more compute plus stronger aggregator' explanation. Please add the compute-matched control (or an equivalent analysis at matched token budget), or reframe the contribution as an empirical study of a specific high-compute all","section":"§5.3, Tables 4/5, Appendix B"},{"comment":"The recommended configuration CIPHER†(10,5) is selected by ranking the 12 strategies on the same four evaluation panels (Infi-DA-Bench M=3/M=5 and InsightBench M=3/M=5) that are then used to report the headline results in §5.3. This is test-set selection; the reported +7.0pp and +10.9pp gains are optimistically biased relative to the performance one would expect from the same configuration chosen a priori. Please validate the recommendation on a hold-out split or with nested cross-validation, or explicitly label the numbers as 'best-on-test' and soften the claim that Ensemble+Goal-Align is a robust default.","section":"§5.2 vs §5.3"}],"minor_comments":[{"comment":"The 'Alignment selection' strategy in §3.2 is renamed 'Goal Align' in Appendix C.1, and the judge model/temperature differ (Claude 3.5 Sonnet, temperature 0.5) from the InsightBench correctness judge (Claude 3.5 Haiku, temperature 0). Please clarify the naming and configuration to avoid confusion.","section":"§3.2, Appendix C.1"},{"comment":"The 'Consistency' metric is described as 'top-3 agreement of the selected plans across seeds' but the aggregation procedure is not defined. Specify whether this is Jaccard overlap, average pairwise agreement, or another measure, and how it enters the composite ranking.","section":"§5.2"},{"comment":"The paper reports Cohen's d alongside Mann-Whitney U tests. Mann-Whitney is a rank-based test; consider reporting a rank-based effect size (e.g., rank-biserial correlation or Cliff's delta) or justify the use of Cohen's d on the ranks.","section":"§5.1"},{"comment":"The benchmark name is inconsistently capitalized: 'InFi-DA-Bench' in Table 2 and 'Infi-DA-Bench' elsewhere. Standardize to one spelling.","section":"Tables 2–5"},{"comment":"The statement that strategies are 'outperformed by strategies based Ensemble generation and entropy based selection techniques' is misleading because the Maxent selection strategy produces the highest selected-plan entropy but does not yield the highest accuracy. Rephrase to reflect that entropy-based selection is not sufficient on its own.","section":"§6 Conclusion"},{"comment":"The reference 'Kang il Lee et al.' should be formatted in standard style (e.g., 'K. il Lee et al.' or 'Lee et al.') to avoid appearing as a surname 'il Lee'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is from an industrial lab and relies on proprietary Titan embeddings and closed Claude models, which limits reproducibility but does not invalidate the measurements. The decisive issue is the missing compute-matched repeated-sampling baseline; the paper honestly acknowledges it in Appendix B, but the contribution framing and headline claims depend on it. I believe this is fixable: either add the baseline or explicitly reframe the paper as a study of a compute-heavy pipeline. The test-set selection of the recommended configuration should also be addressed by relabeling or validation. Given the otherwise thorough empirical study, major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is an honest, well-executed empirical paper about a data-science agent with test-time scaling. The contribution is real but narrower than the title suggests. What is genuinely new is the explicit decoupling of plan generation (N candidates) from plan selection (M executed), and the 24-configuration ablation on two benchmarks (Infi-DA-Bench, InsightBench) telling you which design choices matter. The measurements look solid: three seeds, standard deviations reported, matched-model comparisons against Agent-Poirot and DataWise, and a sensible finding that ensemble generation is the only source of real plan-space entropy. The paper is also unusually candid about its own limitations, which counts for something.\n\nWhere it gets soft: the central mechanism is asserted, not shown. The gains over baselines are +7pp and +10.9pp, but the comparisons are not compute-matched (Table 4 token counts: 16K vs 86K), and Appendix B explicitly concedes that a repeated-sampling baseline at equal tokens was not run. The rebuttal in Appendix B — that gains come from structurally different plans rather than repeated sampling — is exactly the thing you would need to measure. And the Friedman test in Section 5.1 shows that under self-aggregation the generation strategies are statistically indistinguishable; the advantage appears only when the aggregator is upgraded to a leader model. That pattern is consistent with the alternative story: more executions, more diverse prompts, and a stronger aggregator. The true novelty of 'selection' — whether picking M out of N beats just executing M or N independently — is never isolated.\n\nSecond soft spot: the recommended configuration (Ensemble+Goal-Align, leader) is selected on the same benchmarks where it is then reported as the headline result (Section 5.2 vs 5.3). The authors partially acknowledge this through the 'recommended vs absolute best' distinction, and the gap is small (<1.5pp), but it still means the headline numbers carry some selection optimism. This is a minor-to-moderate concern, not a fatal one.\n\nThe math and citation pattern look fine. No formal derivations are promised and none are needed here. The limitations section is honest. The paper is worth engaging with: the DES design space mapping and the recommendation that M is a more reliable lever than N, and that leader aggregation unlocks diversity gains, will be useful to practitioners and to anyone working on agentic test-time scaling.\n\nRecommendation: send to peer review. It is a serious empirical system paper with reproducible experiments and an addressable missing control. The referee should push for a compute-matched repeated-sampling baseline before acceptance; if that baseline matches CIPHER, the paper still has value as a design-space study, just with a more modest claim.","headline":"A useful empirical system paper with a genuine design-space ablation, but the headline causal claim about 'decoupled exploration-selection' is not yet separated from a stronger aggregator plus more compute; deserves refereeing, not desk rejection.","tokens_in":19330,"tokens_out":1359,"would_cite":true,"duration_ms":12155,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that test-time scaling for data-science agents should be split into two phases—generate many candidate starting plans, then select a subset to run in parallel—and that this split lets a small model close most of the gap to","keywords":["test-time scaling","data science agents","initial state generation","plan selection","ensemble prompting","parallel agent execution","aggregation","Decoupled Exploration-Selection"],"falsifier":"Run a compute-matched control on the same 357 tasks: five independent single-plan executions with the same base model and the same total token budget, aggregated by majority voting. If that control matches or exceeds CIPHER's 81.06% / 49.53% accuracy, the decoupling claim reduces to a compute-budget effect.","tokens_in":18355,"feed_emoji":"📊","tokens_out":8042,"duration_ms":68425,"temperature":0.7,"pith_summary":"CIPHER's central claim is that how test-time compute is allocated matters as much as how much is spent: an agent that first generates many candidate starting plans and then deliberately selects a few to execute in parallel outperforms an agent that commits to one plan from the start. On two public data-science benchmarks, the authors report that this decoupled exploration-selection (DES) design lifts accuracy by about 7 to 11 percentage points over matched single-route baselines using the same small base model, and brings the agent close to much larger-model systems. The empirical study isolates the responsible design choices: mixing several planner prompts (ensemble generation), selecting plans by task alignment, and upgrading only the final aggregation step to a stronger model. A sympathetic reader would care because this pinpoints the initial plan as a high-leverage intervention and gives a concrete recipe for spending test-time compute. The paper is transparent that a fully compute-matched baseline was not evaluated, so the claim that gains come from structural plan diversity rather than extra tokens is asserted rather than fully demonstrated.","feed_headline":"Generate 10 plans, run 5: agent accuracy jumps 7–11 pts","feed_subtitle":"Decoupling exploration from selection beats single-plan agents with the same small model on open and closed-ended data tasks.","key_machinery":"The central object is the Decoupled Exploration-Selection (DES) framework: an explicit two-phase loop that first samples N candidate initial states (high-level plans) from a planner, then selects M of them for parallel execution, and finally aggregates the M execution transcripts into one answer. Its work is to turn the planner's output distribution into a controllable search space: the generation mode controls how broad that space is, the selection mode controls which points get executed, and the aggregation mode controls how much the final answer can exploit the diversity. The paper's key causal mechanism is the interaction between these choices—diversity from the generation step only conv","core_discovery":"The paper's discovery is that decoupling exploration from selection in test-time scaling works for data-science agents. With a small base model, CIPHER generates N=10 candidate plans, selects M=5 of them, executes those routes in parallel, and aggregates the results with a stronger model. In matched-model comparisons this configuration reaches 81.06% on the closed-ended benchmark (versus 74.06% for the specialized single-route baseline) and 49.53% on the open-ended benchmark (versus 38.63%); both differences are statistically significant. The authors further find that the advantage appears only when the aggregator is a stronger model: with self-aggregation no generation or selection strategy","pith_inferences":["Editorial inference: the decisive missing control is a token-matched baseline (e.g., five independent single-plan runs with majority voting), which the paper itself flags in Appendix B; until that runs, the structural-diversity interpretation is an inference rather than a demonstrated mechanism.","Editorial inference: if the DES mechanism is real, it should transfer to other agent domains where the first plan conditions a long trajectory, such as web navigation, code repair, or scientific discovery; the framework's generation/selection/aggregation axes provide a ready-made ablation template for testing that transfer.","Editorial inference: the finding that maximum-entropy selection does not improve accuracy suggests a testable variant—a selection score combining goal alignment with a tunable diversity term—which might outperform either pure alignment or pure diversity on open-ended tasks.","Editorial inference: because gains are larger for weaker models on closed tasks but for stronger models on open tasks, the optimal N/M budget likely depends on task openness; an adaptive budget that grows N for open-ended tasks and weak base models would be a natural extension."],"forward_implications":["Scaling the selection budget M produces reliable, diminishing-return gains on both closed- and open-ended tasks: +7.2pp and +3.3pp from one to three plans, then +1.9pp and +1.4pp from three to five.","Ensemble generation is the only tested strategy that raises the entropy of the generated plan set, and under leader aggregation it outperforms base and conditional generation by 1.25–1.79pp.","Upgrading just the aggregator node—with all execution logs identical—adds roughly +2.8pp on closed-ended and +4.5pp on open-ended tasks, and is required for the framework's advantage to emerge.","With the recommended configuration, a small base model reaches 81.06% and 49.53% on the two benchmarks, closing most of the gap to a much larger model running a fixed single-route plan.","The recommended default (Ensemble + Goal-Align + leader) is within 1.5pp of the empirically best configuration on both benchmarks, so a single setting transfers across task types."],"fun_headline_variants":["CIPHER: generate 10 plans, select 5, beat single-plan agents","Decouple exploration from selection: small agent gains 7-11 pts","Multi-plan with strong aggregator: 7-11 pt boost over single plan","Small data agent beats larger ones: decouple exploration-selection","Generate 10 plans, execute 5: 7-11 pt accuracy gain"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim depends on the assumption that the accuracy gains come from generating and selecting structurally different plans rather than from spending roughly five times more tokens; the paper acknowledges in Appendix B that a fully compute-matched baseline (e.g., five independent single-plan executions aggregated by majority voting) was not evaluated.","fun_headline_variants_meta":{"raw":{"variants":["CIPHER: generate 10 plans, select 5, beat single-plan agents","Decouple exploration from selection: small agent gains 7-11 pts","Multi-plan with strong aggregator: 7-11 pt boost over single plan","Small data agent beats larger ones: decouple exploration-selection","Generate 10 plans, execute 5: 7-11 pt accuracy gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3498,"prompt_tokens":739,"completion_tokens":2759,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":2658}},"tokens_in":483,"tokens_out":2759,"duration_ms":18805,"temperature":1.0,"reasoning_tokens":2658,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:13:23.085584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a compute-matched control on the same 357 tasks: five independent single-plan executions with the same base model and the same total token budget, aggregated by majority voting. If that control matches or exceeds CIPHER's 81.06% / 49.53% accuracy, the decoupling claim reduces to a compute-budget effect.","supporting_citations":[],"review_version":1}