Pith. sign in

REVIEW 2 major objections 6 minor 62 references

This paper claims that test-time scaling for data-science agents should be split into two phases—generate many candidate starting plans, then select a subset to run in parallel—and that this split lets a small model close most of the gap to

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:13 UTC pith:UHQGQ473

load-bearing objection A useful empirical system paper with a genuine design-space ablation, but the headline causal claim about 'decoupled exploration-selection' is not yet separated from a stronger aggregator plus more compute; deserves refereeing, not desk rejection. the 2 major comments →

arxiv 2607.14386 v1 pith:UHQGQ473 submitted 2026-07-15 cs.AI

CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents

Maxime Heuillet , Sharadind Peddiraju This is my paper
classification cs.AI
keywords test-time scalingdata science agentsinitial state generationplan selectionensemble promptingparallel agent executionaggregationDecoupled Exploration-Selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CIPHER's central claim is that how test-time compute is allocated matters as much as how much is spent: an agent that first generates many candidate starting plans and then deliberately selects a few to execute in parallel outperforms an agent that commits to one plan from the start. On two public data-science benchmarks, the authors report that this decoupled exploration-selection (DES) design lifts accuracy by about 7 to 11 percentage points over matched single-route baselines using the same small base model, and brings the agent close to much larger-model systems. The empirical study isolates the responsible design choices: mixing several planner prompts (ensemble generation), selecting plans by task alignment, and upgrading only the final aggregation step to a stronger model. A sympathetic reader would care because this pinpoints the initial plan as a high-leverage intervention and gives a concrete recipe for spending test-time compute. The paper is transparent that a fully compute-matched baseline was not evaluated, so the claim that gains come from structural plan diversity rather than extra tokens is asserted rather than fully demonstrated.

Core claim

The paper's discovery is that decoupling exploration from selection in test-time scaling works for data-science agents. With a small base model, CIPHER generates N=10 candidate plans, selects M=5 of them, executes those routes in parallel, and aggregates the results with a stronger model. In matched-model comparisons this configuration reaches 81.06% on the closed-ended benchmark (versus 74.06% for the specialized single-route baseline) and 49.53% on the open-ended benchmark (versus 38.63%); both differences are statistically significant. The authors further find that the advantage appears only when the aggregator is a stronger model: with self-aggregation no generation or selection strategy

What carries the argument

The central object is the Decoupled Exploration-Selection (DES) framework: an explicit two-phase loop that first samples N candidate initial states (high-level plans) from a planner, then selects M of them for parallel execution, and finally aggregates the M execution transcripts into one answer. Its work is to turn the planner's output distribution into a controllable search space: the generation mode controls how broad that space is, the selection mode controls which points get executed, and the aggregation mode controls how much the final answer can exploit the diversity. The paper's key causal mechanism is the interaction between these choices—diversity from the generation step only conv

Load-bearing premise

The central claim depends on the assumption that the accuracy gains come from generating and selecting structurally different plans rather than from spending roughly five times more tokens; the paper acknowledges in Appendix B that a fully compute-matched baseline (e.g., five independent single-plan executions aggregated by majority voting) was not evaluated.

What would settle it

Run a compute-matched control on the same 357 tasks: five independent single-plan executions with the same base model and the same total token budget, aggregated by majority voting. If that control matches or exceeds CIPHER's 81.06% / 49.53% accuracy, the decoupling claim reduces to a compute-budget effect.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Scaling the selection budget M produces reliable, diminishing-return gains on both closed- and open-ended tasks: +7.2pp and +3.3pp from one to three plans, then +1.9pp and +1.4pp from three to five.
  • Ensemble generation is the only tested strategy that raises the entropy of the generated plan set, and under leader aggregation it outperforms base and conditional generation by 1.25–1.79pp.
  • Upgrading just the aggregator node—with all execution logs identical—adds roughly +2.8pp on closed-ended and +4.5pp on open-ended tasks, and is required for the framework's advantage to emerge.
  • With the recommended configuration, a small base model reaches 81.06% and 49.53% on the two benchmarks, closing most of the gap to a much larger model running a fixed single-route plan.
  • The recommended default (Ensemble + Goal-Align + leader) is within 1.5pp of the empirically best configuration on both benchmarks, so a single setting transfers across task types.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the decisive missing control is a token-matched baseline (e.g., five independent single-plan runs with majority voting), which the paper itself flags in Appendix B; until that runs, the structural-diversity interpretation is an inference rather than a demonstrated mechanism.
  • Editorial inference: if the DES mechanism is real, it should transfer to other agent domains where the first plan conditions a long trajectory, such as web navigation, code repair, or scientific discovery; the framework's generation/selection/aggregation axes provide a ready-made ablation template for testing that transfer.
  • Editorial inference: the finding that maximum-entropy selection does not improve accuracy suggests a testable variant—a selection score combining goal alignment with a tunable diversity term—which might outperform either pure alignment or pure diversity on open-ended tasks.
  • Editorial inference: because gains are larger for weaker models on closed tasks but for stronger models on open tasks, the optimal N/M budget likely depends on task openness; an adaptive budget that grows N for open-ended tasks and weak base models would be a natural extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces CIPHER, a data science agent that applies test-time scaling via Decoupled Exploration-Selection (DES): it first generates N candidate initial plans, selects M for parallel execution, and aggregates the results. The framework is instantiated with three generation modes (base/independent, conditional, ensemble), four selection strategies (random, maxent, clustering, goal-align), and two aggregation modes (self vs leader). On Infi-DA-Bench and InsightBench, the paper reports that the recommended configuration CIPHER†(10,5) (ensemble generation, goal-align selection, leader aggregation) improves over the matched-model single-agent baselines: 81.06% vs 74.06% on Infi-DA-Bench and 49.53% vs 38.63% on InsightBench. The paper also presents an extensive ablation of 24 configurations across two budgets and analyzes generation/selection/aggregation choices, with statistical tests and token-usage reporting.

Significance. The empirical measurements are valuable: 357 tasks, two benchmarks, 24 configurations, three seeds, and significance tests. The paper is candid about limitations, including the absence of a compute-matched repeated-sampling baseline (Appendix B). However, this missing control is decisive for the central causal claim. The results clearly show that spending more inference compute (10 plans, 5 executions, leader aggregator) improves accuracy over single-execution baselines, but they do not show that the decoupling of exploration and selection—as opposed to merely sampling more executions and using a stronger aggregator—is responsible. If the control were added and showed equal accuracy at equal tokens, the DES contribution would reduce to a compute-allocation strategy. The design-space analysis (entropy measurements, Friedman tests) is useful but does not substitute for this control.

major comments (2)
  1. [§5.3, Tables 4/5, Appendix B] The central claim that the DES decoupling drives the reported gains is not supported. The non-DES baselines use substantially fewer input tokens (DataWise 16K, Agent-Poirot 51K) than CIPHER† (86K and 102K on the two benchmarks). Appendix B explicitly states that a fully compute-matched baseline (e.g., five independent CIPHER(1,1) executions aggregated via majority voting) is not evaluated. The rebuttal that an equivalent-token replay would not exploit plan diversity is an assertion, not a measurement. Furthermore, the Friedman test in §5.1—generation strategy is non-significant under self-aggregation and significant only under leader aggregation—is consistent with a 'more compute plus stronger aggregator' explanation. Please add the compute-matched control (or an equivalent analysis at matched token budget), or reframe the contribution as an empirical study of a specific high-compute all
  2. [§5.2 vs §5.3] The recommended configuration CIPHER†(10,5) is selected by ranking the 12 strategies on the same four evaluation panels (Infi-DA-Bench M=3/M=5 and InsightBench M=3/M=5) that are then used to report the headline results in §5.3. This is test-set selection; the reported +7.0pp and +10.9pp gains are optimistically biased relative to the performance one would expect from the same configuration chosen a priori. Please validate the recommendation on a hold-out split or with nested cross-validation, or explicitly label the numbers as 'best-on-test' and soften the claim that Ensemble+Goal-Align is a robust default.
minor comments (6)
  1. [§3.2, Appendix C.1] The 'Alignment selection' strategy in §3.2 is renamed 'Goal Align' in Appendix C.1, and the judge model/temperature differ (Claude 3.5 Sonnet, temperature 0.5) from the InsightBench correctness judge (Claude 3.5 Haiku, temperature 0). Please clarify the naming and configuration to avoid confusion.
  2. [§5.2] The 'Consistency' metric is described as 'top-3 agreement of the selected plans across seeds' but the aggregation procedure is not defined. Specify whether this is Jaccard overlap, average pairwise agreement, or another measure, and how it enters the composite ranking.
  3. [§5.1] The paper reports Cohen's d alongside Mann-Whitney U tests. Mann-Whitney is a rank-based test; consider reporting a rank-based effect size (e.g., rank-biserial correlation or Cliff's delta) or justify the use of Cohen's d on the ranks.
  4. [Tables 2–5] The benchmark name is inconsistently capitalized: 'InFi-DA-Bench' in Table 2 and 'Infi-DA-Bench' elsewhere. Standardize to one spelling.
  5. [§6 Conclusion] The statement that strategies are 'outperformed by strategies based Ensemble generation and entropy based selection techniques' is misleading because the Maxent selection strategy produces the highest selected-plan entropy but does not yield the highest accuracy. Rephrase to reflect that entropy-based selection is not sufficient on its own.
  6. [References] The reference 'Kang il Lee et al.' should be formatted in standard style (e.g., 'K. il Lee et al.' or 'Lee et al.') to avoid appearing as a surname 'il Lee'.

Circularity Check

1 steps flagged

No derivation is circular; the one constructional element is the Maxent self-check, with test-set selection and a missing compute-matched control as secondary methodological limitations.

specific steps
  1. self definitional [Section 5.1, 'The Maxent selection strategy is uniquely effective...' paragraph]
    "Across all generation methods and both benchmarks, the selection strategy Maxent consistently produces the highest selected-plan entropy. The ordering Maxent≥Clustering≥Random≥Goal-Align is preserved on both benchmarks independently, validating that Maxent’s entropy-based objective successfully identifies maximally diverse subsets from the candidate pool."

    Maxent is defined in Section 3.2 as an entropy-based selection method: 'a greedy maximin algorithm maximizes the selection’s entropy using cosine distance.' The claimed evidence is the selected-plan entropy computed from the same embeddings and cosine geometry (Tables 2-3). Therefore, 'Maxent produces the highest selected-plan entropy' is true by construction: it is the very objective the selector optimizes. This is a sanity check on the implementation, not an empirical discovery, and it is not load-bearing for the main accuracy comparisons.

full rationale

The core of the paper is an empirical evaluation rather than a derivation chain. CIPHER is measured against external, fixed baselines (DataWise, Agent-Poirot) on two public benchmarks, and the DES design-space study is an ablation over operationally defined generation, selection, and aggregation modes. No equation is derived from another, and no fitted parameter is renamed as a prediction. There is no load-bearing self-citation: the only author self-citation (Heuillet et al., 2021) is background AutoML context; the aggregator prompt is attributed to external work [Zhang et al., 2025]; the diversity-bottleneck citation [Maryanskyy, 2026] is not by the authors. The genuine constructional element is the Maxent self-check described above: it validates the selector against its own objective. Two limitations weaken the causal interpretation but are not circularity. First, Appendix B explicitly concedes, 'A fully compute-matched baseline (e.g., five independent CIPHER(1,1) executions aggregated via majority voting) is not evaluated here, and we acknowledge this as a limitation,' so the claim that gains come from structurally different plans rather than more compute is under-tested. Second, Section 5.2 selects the recommended and 'empirically best' configurations using the same benchmark panels on which Section 5.3 reports their accuracy, creating test-set selection bias; however, this is benchmark overfitting rather than a definitional reduction. Because the principal comparisons are against external benchmarks and fixed baselines, the circularity score is low.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

All empirical claims rest on the two benchmarks, the LLM judge, and hyperparameters tuned on the same tasks; no new physical or mathematical entity is introduced.

free parameters (8)
  • N (generation budget) = 10 (and 30 in scaling panels)
    Number of candidate initial states generated; set by user/hyperparameter, not optimized but explored.
  • M (selection budget) = 3 and 5
    Number of states executed; central experimental variable.
  • E (number of ensemble prompts) = 2
    Set in Appendix C because E<M so the prompt set is likely covered; chosen by hand.
  • p (prompt sampling distribution) = [0.5, 0.5]
    Uniform over the two ensemble prompts; chosen by hand.
  • max_iterations coding loop = 3
    Appendix D: set based on observed retry distribution under the baseline (86.9%/59.7% success at iteration 0, >=99% by iteration 2). This is fitting a design hyperparameter to the evaluation tasks.
  • sampling temperature/top-p/top-k = 0.5, 0.99, 500
    Section 4.3: selected after a sensitivity study on four configurations; fit to the benchmarks.
  • Goal Align judge and parameters = Claude 3.5 Sonnet temp 0.5, 200 tokens
    Choice of selection oracle; not compared to other judge models.
  • Leader aggregator configuration = Sonnet 3.7 + thinking, 5k reasoning budget, temp 1
    Chosen for leader aggregation; not swept.
axioms (5)
  • domain assumption LLM-as-a-judge accuracy on InsightBench reflects true insight retrieval
    Section 4.1 uses Claude 3.5 Haiku as judge; no human agreement study is reported.
  • domain assumption Ground-truth insights in benchmarks are complete and correct
    Accuracy is measured against them.
  • domain assumption Embedding cosine distance / k-means clusters capture plan diversity
    Section 3.2 uses Titan embeddings; no validation that embedding diversity corresponds to task-relevant diversity.
  • domain assumption Official Infi-DA-Bench rule-based evaluator is a valid measure
    Section 4.2 adopts the official strict rule-based evaluation.
  • domain assumption Three random seeds are enough to support p-values and effect sizes
    Statistical tests across 3 seeds and M=3,5; no power analysis is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 18091 in / 15611 out tokens · 143894 ms · 2026-08-02T02:13:23.085584+00:00 · methodology

0 comments
read the original abstract

Data science tasks span from closed-ended information extraction to open-ended analysis, presenting significant challenges for automation. Recent AI agents powered by language models show promise for handling such complex tasks. However, existing agents typically rely on a single initial state that conditions the entire agent's execution, making them vulnerable to cascading errors initiated by a suboptimal initial state. To mitigate this, we present CIPHER, an automated data science agent that leverages test-time scaling through the generation and selection of multiple initial states for concurrent execution. Unlike existing works on test-time scaling of AI agents, CIPHER explicitly decouples the generation of candidate initial states from their strategic selection for parallel execution. Through extensive evaluation on two benchmarks (closed-form and open-form tasks), we demonstrate that CIPHER exceeds state-of-the-art performance in matched-model comparisons, and remains competitive against larger-model baselines despite relying on a substantially smaller base LM. Our empirical study characterizes the design space of the Decoupled Exploration-Selection (DES) framework: we quantify how generation strategy, selection strategy, and aggregator model capacity contribute to overall performance, and derive actionable design recommendations for practitioners.

Figures

Figures reproduced from arXiv: 2607.14386 by Maxime Heuillet, Sharadind Peddiraju.

Figure 1
Figure 1. Figure 1: CIPHER execution route. The system generates N initial states, selects M for concurrent execution, and then aggregates the information for the output. Research question and contributions: We hypothesize that in test-time scaling, the gen￾eration of initial states and their selection for execution jointly influence the performance of the agent. To validate this hypothesis, we make three main contributions: … view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy gain (pp) over the single-agent baseline CIPHER(N=1, M=1). Results averaged [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy gain (pp) over the single-agent baseline CIPHER(N=1, M=1). Results for [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of iterations needed to reach executable code across [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Table: Strategy rankings across benchmarks (Infi-DA-Bench, InsightBench) and selection [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy gain (pp) over the single-agent baseline CIPHER(1,1) stratified by question [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Example execution of CIPHER on question 254 of Infi-DA-Bench. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example execution of CIPHER on question 16 of Insightbench. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 6 linked inside Pith

  1. [1]

    2024 , eprint=

    DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning , author=. 2024 , eprint=

    2024

  2. [2]

    2024 , eprint=

    Data Interpreter: An LLM Agent For Data Science , author=. 2024 , eprint=

    2024

  3. [3]

    arXiv preprint arXiv:2409.07703 , year=

    DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? , author=. arXiv preprint arXiv:2409.07703 , year=

    Pith/arXiv arXiv

  4. [4]

    2025 , eprint=

    InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation , author=. 2025 , eprint=

    2025

  5. [5]

    arXiv preprint arXiv:2312.13876 , year=

    Capture the Flag: Uncovering Data Insights with Large Language Models , author=. arXiv preprint arXiv:2312.13876 , year=

    Pith/arXiv arXiv

  6. [6]

    2025 , eprint=

    DABstep: Data Agent Benchmark for Multi-step Reasoning , author=. 2025 , eprint=

    2025

  7. [7]

    arXiv preprint arXiv:2508.05002 , year=

    Agenticdata: An agentic data analytics system for heterogeneous data , author=. arXiv preprint arXiv:2508.05002 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2506.23719 , year=

    DABstep: Data Agent Benchmark for Multi-step Reasoning , author=. arXiv preprint arXiv:2506.23719 , year=

    Pith/arXiv arXiv

  9. [9]

    2024 , eprint=

    DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models , author=. 2024 , eprint=

    2024

  10. [10]

    2025 , eprint=

    DSCodeBench: A Realistic Benchmark for Data Science Code Generation , author=. 2025 , eprint=

    2025

  11. [11]

    2025 , eprint=

    DatawiseAgent: A Notebook-Centric LLM Agent Framework for Automated Data Science , author=. 2025 , eprint=

    2025

  12. [12]

    2025 , publisher=

    Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems , author=. 2025 , publisher=

    2025

  13. [13]

    arXiv preprint arXiv:2401.05507 , year=

    Infiagent-dabench: Evaluating agents on data analysis tasks , author=. arXiv preprint arXiv:2401.05507 , year=

    Pith/arXiv arXiv

  14. [14]

    First Conference on Language Modeling , year=

    Autogen: Enabling next-gen LLM applications via multi-agent conversations , author=. First Conference on Language Modeling , year=

  15. [15]

    2025 , eprint=

    MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement , author=. 2025 , eprint=

    2025

  16. [16]

    2025 , eprint=

    AutoMind: Adaptive Knowledgeable Agent for Automated Data Science , author=. 2025 , eprint=

    2025

  17. [17]

    2024 , eprint=

    Large Language Monkeys: Scaling Inference Compute with Repeated Sampling , author=. 2024 , eprint=

    2024

  18. [18]

    Flash-Searcher: Fast and Effective Web Agents via

    Tianrui Qin and Qianben Chen and Sinuo Wang and He Xing and King Zhu and He Zhu and Dingfeng Shi and Xinxin Liu and Ge Zhang and Jiaheng Liu and Xitong Gao and Yuchen Eleanor Jiang and Wangchunshu Zhou , booktitle=. Flash-Searcher: Fast and Effective Web Agents via. 2026 , url=

    2026

  19. [19]

    The Fourteenth International Conference on Learning Representations , year=

    Generalized Parallel Scaling with Interdependent Generations , author=. The Fourteenth International Conference on Learning Representations , year=

  20. [20]

    The Fourteenth International Conference on Learning Representations , year=

    Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning , author=. The Fourteenth International Conference on Learning Representations , year=

  21. [21]

    2025 , eprint=

    Optimizing Sequential Multi-Step Tasks with Parallel LLM Agents , author=. 2025 , eprint=

    2025

  22. [22]

    International workshop on multiple classifier systems , pages=

    Ensemble methods in machine learning , author=. International workshop on multiple classifier systems , pages=. 2000 , organization=

    2000

  23. [23]

    Journal of computer and system sciences , volume=

    A decision-theoretic generalization of on-line learning and an application to boosting , author=. Journal of computer and system sciences , volume=. 1997 , publisher=

    1997

  24. [24]

    2025 , eprint=

    Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling , author=. 2025 , eprint=

    2025

  25. [25]

    2025 , eprint=

    Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning , author=. 2025 , eprint=

    2025

  26. [26]

    2025 , eprint=

    Think Deep, Think Fast: Investigating Efficiency of Verifier-free Inference-time-scaling Methods , author=. 2025 , eprint=

    2025

  27. [27]

    2025 , eprint=

    Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search , author=. 2025 , eprint=

    2025

  28. [28]

    2025 , eprint=

    Generating Diverse Hypotheses for Inductive Reasoning , author=. 2025 , eprint=

    2025

  29. [29]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  30. [30]

    2025 , eprint=

    Scaling Test-time Compute for LLM Agents , author=. 2025 , eprint=

    2025

  31. [31]

    2025 , eprint=

    DS-STAR: Data Science Agent via Iterative Planning and Verification , author=. 2025 , eprint=

    2025

  32. [32]

    International Conference on Machine Learning , pages=

    Coresets for data-efficient training of machine learning models , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  33. [33]

    Titan Text Embeddings v2 , year =

  34. [34]

    Mathematical programming , volume=

    An analysis of approximations for maximizing submodular set functions—I , author=. Mathematical programming , volume=. 1978 , publisher=

    1978

  35. [35]

    , title =

    MacQueen, J. , title =. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability , volume =. 1967 , publisher =

    1967

  36. [36]

    Frontiers of Computer Science , volume=

    A survey on large language model based autonomous agents , author=. Frontiers of Computer Science , volume=. 2024 , publisher=

    2024

  37. [37]

    2015 IEEE international conference on data science and advanced analytics (DSAA) , pages=

    Deep feature synthesis: Towards automating data science endeavors , author=. 2015 IEEE international conference on data science and advanced analytics (DSAA) , pages=. 2015 , organization=

    2015

  38. [38]

    Advances in Neural Information Processing Systems , volume=

    Large language models for automated data science: Introducing caafe for context-aware automated feature engineering , author=. Advances in Neural Information Processing Systems , volume=

  39. [39]

    8th ICML Workshop on Automated Machine Learning (AutoML) , year=

    Sequential Automated Machine Learning: Bandits-driven Exploration using a Collaborative Filtering Representation , author=. 8th ICML Workshop on Automated Machine Learning (AutoML) , year=

  40. [40]

    2024 , eprint=

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters , author=. 2024 , eprint=

    2024

  41. [41]

    Proceedings of the genetic and evolutionary computation conference 2016 , pages=

    Evaluation of a tree-based pipeline optimization tool for automating data science , author=. Proceedings of the genetic and evolutionary computation conference 2016 , pages=

    2016

  42. [42]

    2024 , month = dec, organization =

    Schluntz, Erik and Zhang, Barry , title =. 2024 , month = dec, organization =

    2024

  43. [43]

    LangGraph: Build language agents as graphs , year =

  44. [44]

    2025 , eprint=

    Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation , author=. 2025 , eprint=

    2025

  45. [45]

    ACM Computing Surveys (CSUR) , volume=

    Data science: a comprehensive overview , author=. ACM Computing Surveys (CSUR) , volume=. 2017 , publisher=

    2017

  46. [46]

    2024 , eprint=

    ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models , author=. 2024 , eprint=

    2024

  47. [47]

    2025 , eprint=

    Risk Analysis Techniques for Governed LLM-based Multi-Agent Systems , author=. 2025 , eprint=

    2025

  48. [48]

    arXiv preprint arXiv:2304.05128 , year=

    Teaching large language models to self-debug , author=. arXiv preprint arXiv:2304.05128 , year=

    Pith/arXiv arXiv

  49. [49]

    Proceedings of th e 10th international conference on artificial intelligence and soft computing, Sydney, Australia , year=

    Evaluating persona prompting for question answering tasks , author=. Proceedings of th e 10th international conference on artificial intelligence and soft computing, Sydney, Australia , year=

  50. [50]

    2020 , publisher=

    Bandit algorithms , author=. 2020 , publisher=

    2020

  51. [51]

    2025 , eprint=

    Context Length Alone Hurts LLM Performance Despite Perfect Retrieval , author=. 2025 , eprint=

    2025

  52. [52]

    2026 , eprint=

    SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science , author=. 2026 , eprint=

    2026

  53. [53]

    2026 , eprint=

    When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines , author=. 2026 , eprint=

    2026

  54. [54]

    2025 , eprint=

    Flash-Searcher: Fast and Effective Web Agents via DAG-Based Parallel Execution , author=. 2025 , eprint=

    2025

  55. [55]

    The Eleventh International Conference on Learning Representations , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. The Eleventh International Conference on Learning Representations , year=

  56. [56]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  57. [57]

    and Mordatch, Igor , title =

    Du, Yilun and Li, Shuang and Torralba, Antonio and Tenenbaum, Joshua B. and Mordatch, Igor , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  58. [58]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Self-Refine: Iterative Refinement with Self-Feedback , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  59. [59]

    Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =

    Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and Gupta, Shashank and Majumder, Bodhisattwa Prasad and Hermann, Katherine and Welleck, Sean and Yazdanbakhsh, Amir and Clark, Peter , title =. Proceedings of the 37th Internatio...

    2023

  60. [60]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Reflexion: language agents with verbal reinforcement learning , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  61. [61]

    Charlie Victor Snell and Jaehoon Lee and Kelvin Xu and Aviral Kumar , booktitle=. Scaling. 2025 , url=

    2025

  62. [62]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , title =. Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =. 2023 , publisher =

    2023