{"id":"e672989b-9f60-46e8-a075-695d8cf7bec1","arxiv_id":"2608.08958","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Idea Search, a tree search variant that samples and updates a dynamically scored bank of atomic design ideas, modestly improves LLM-generated single-cell batch integration code over a pure tree search baseline.","lead":"A method called Idea Search stores atomic 'ideas' from known methods in a bank, samples them to guide an LLM as it writes and mutates code, then updates the bank from execution results. On a single-cell RNA sequencing batch integration task, it modestly beats a strong tree search baseline, though the gains are close to run-to-run noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dynamic bank update is never ablated; the reported gains may be fully attributable to static idea injection plus prompt conditioning.","rationale":"I considered the statistical fragility first: with five trials per configuration and mean gains (about 0.02) comparable to the reported SD (0.008-0.018), the headline 'reliably' is easy to challenge, and the paper's own Scope paragraph concedes this. But that concern is explicitly acknowledged and is addressable by more trials; it does not target the mechanism. The more load-bearing gap is that the defining component of Idea Search, the dynamic execution-verified bank, is never isolated. The algorithm's novelty has three steps in the Abstract, and step (3), 'dynamically updating the bank with new ideas discovered through execution,' is what separates Idea Search from static plan or prompt-perturbation methods. Section 2 explicitly motivates this by contrasting with approaches that 'remain static over time.' Yet Section 5 only varies parameters of the bank and search; it never turns the update off. The close correspondence of the random and bandit final means (0.697 vs 0.697) is suggestive that the score-feedback mechanism may not be what matters, since random ignores scores. The 0.728 max also arises under random sampling. Thus even if all reported numbers are accurate, the evidence does not establish that the dynamic mechanism causes the improvement. The proposed frozen-bank ablation is a single, decisive experiment: if static matches dynamic, the paper's central contribution reduces to a prompt-engineering effect; if dynamic wins, the mechanism is supported. This is why I keep the reader's CONDITIONAL verdict: the issue is a missing control that can be supplied, not a demonstrated failure.","tokens_in":10865,"tokens_out":10016,"duration_ms":101684,"concrete_test":"Run an ablation that freezes the Idea Bank at initialization: remove Algorithm 1 lines 13-16 (no decomposition of new solutions, no addition of unseen ideas, no updates to s_j or r_j) while keeping all other settings identical. Execute both Random and Bandit sampling with the frozen Expert-Only and Augmented banks for at least 20 trials per configuration, and compare final mean and max scores to the corresponding dynamic runs. If a frozen configuration matches its dynamic counterpart within one standard error, the dynamic update is not load-bearing; if dynamic runs clearly improve, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's defining novelty is the dynamic Idea Bank: Algorithm 1 lines 13-16 add newly discovered ideas and update each idea's execution-verified score/rank, and the Discussion credits 'systematically injecting execution-verified ideas.' Yet no reported experiment disables these updates. All comparisons in Section 5 vary sampling (Random vs Bandit), initial bank (Expert-Only vs Augmented), prompting (Conservative vs Exploratory), and alpha; none compares the full dynamic loop to a frozen initial bank. This matters because random sampling, which ignores all score feedback, also reaches the 0.697 mean in Section 5.1, and the augmented-bank random configuration produces the 0.728 max. If the score-update/new-idea-insertion machinery is inert, the improvement reduces to a static curated list plus prompt conditioning, and the central claim that a dynamically updated bank drives the gain is not established. This is a missing control for the mechanism, not an internal inconsistency. It also sharpens the reader's concern about idea fidelity: even if decomposition is faithful, the execution-verified feedback loop may be superfluous.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Idea Search, a tree-search framework for automated scientific code generation in which an explicit, human-readable \"Idea Bank\" is used to guide LLM mutations. Expert methods are decomposed into atomic ideas, an idea is sampled (randomly or via a rank-based UCB bandit) and injected into the mutation prompt, and the bank is dynamically updated by decomposing newly generated solutions and updating each idea's execution-verified score. The method is evaluated on scRNA-seq batch integration against a pure Tree Search baseline from Aygün et al. (2026). The central empirical claims are that Idea Search reliably breaks the baseline plateau (mean score from 0.678 to roughly 0.697, best score 0.728), and that design choices interact: bank augmentation helps bandit sampling but not random sampling, exploratory prompting produces rare high-scoring outliers, and increasing the bandit exploration parameter alpha is counterproductive. The manuscript includes an explicit Scope paragraph and a Limitations section acknowledging the small effect size, single task, single LLM, and coarse credit assignment.","tokens_in":11058,"tokens_out":3598,"duration_ms":36219,"significance":"If the central claim holds, Idea Search is a useful step toward making conceptual exploration a first-class component of test-time scaling: it maintains an explicit, executable, and dynamically updated representation of scientific ideas, and the paper includes enough detail in the appendix (prompts, full idea lists, scoring protocol reference) to reproduce the setup. The authors are commendably explicit about the narrowness of the evidence and the small effect size relative to trial-to-trial variance. The main value is the formulation and the clean identification of design-choice interactions, not a definitive effect-size estimate. However, the paper does not isolate the dynamic-update mechanism from static idea injection, and the statistical support for \"reliably\" is weak given five trials per configuration. If the dynamic mechanism is inert, the contribution shrinks to a static curated list plus prompt conditioning, so the missing control is load-bearing rather than cosmetic.","major_comments":[{"comment":"The defining novelty of the method is the dynamic Idea Bank: Algorithm 1 lines 13-16 insert newly discovered ideas and update each idea's score and rank from executed solutions, and the Discussion credits \"systematically injecting execution-verified ideas.\" Yet no experiment in Section 5 ablates these updates. Every reported configuration varies the sampling rule, the initial bank, the prompting style, or alpha, but none compares the full dynamic loop to a frozen initial bank with identical sampling and prompting. This matters because random sampling, which ignores all score feedback, reaches the same mean as the bandit in Section 5.1, and the augmented-bank random configuration produces the single best score of 0.728. A missing control for the mechanism is not an internal inconsistency, but it means the central claim that a dynamically updated bank drives the gain is not established by the reported experiments. I request an experiment that freezes the initial bank and disables score updates, matching all other components, so the contribution of new-idea insertion and execution-verified re-ranking can be separated from static idea prompting.","section":"Algorithm 1 and Section 5"},{"comment":"The statistical support for the headline claim is weak. The mean improvement over baseline is about 0.02 while the reported standard deviations across five trials are 0.008-0.018, and the paper's own Scope paragraph concedes that the gain is comparable to trial-to-trial variance. The statement that Idea Search \"reliably breaks the plateau\" is not supported by any significance test, confidence interval, or power analysis, and with n=5 per configuration it is hard to distinguish a true shift from sampling noise. In addition, the alpha-comparison in Section 5.3 reports a default mean of 0.712 +/- 0.012, which matches the Random-sampling Expert-Only configuration in Section 5.2 rather than the bandit means reported in Sections 5.1 and 5.2; the precise configuration for the alpha sweep is not identified. Please report the exact configuration for each comparison, add confidence intervals or a paired test across matched trials, and, because the paper argues the signal is in rare outliers, report the distribution of best-of-n scores rather than only the mean.","section":"Sections 5.1, 5.2, and 5.3; Scope paragraph"},{"comment":"The execution-verified idea-score mechanism relies on the assumption that the mean score over all solutions containing an idea reflects that idea's contribution, but the paper's Limitations section correctly notes that credit is assigned coarsely when several ideas co-occur and that the decomposition inherits LLM bias. This is a genuine correctness risk for the bandit mechanism: if the decomposition is unfaithful or credit is diluted by co-occurrence, the bandit's rank-based scores may simply track prompt artifacts rather than the actual causal effect of the idea. The manuscript does not provide any sensitivity analysis, such as measuring decomposition stability across repeated LLM calls, ablating the score-update term while keeping new-idea insertion, or comparing rank-based UCB against a version that uses raw mean scores. I ask for at least one such analysis so that the reader can gauge how much of the reported effect is attributable to the idea-credit assignment rather than to the prompt's general encouragement to try new concepts.","section":"Algorithm 1 lines 13-16 and Limitations"}],"minor_comments":[{"comment":"The exploration term is written as alpha/N times the square-root term; please clarify whether N is the current bank size and how the rank r_i is defined for ties and for unseen ideas, since the paper states that unseen ideas get maximum score but does not specify the corresponding rank used in the 1/r_i term.","section":"Section 4.2, Eq. (1)"},{"comment":"The experimental setup does not state the mutation budget b, the number of tree nodes or LLM calls per trial, the seed solution, or the exact tree-expansion policy; without these values the reported curves and the comparison across designs are not reproducible.","section":"Section 4 and Section 5"},{"comment":"The Augmented Bank list contains duplicate entries (e.g., \"Minimizing mutual information between latent space and source ID\" and \"Regressing out the source ID from principal components\" each appear twice), which should be deduplicated because duplicates effectively change sampling weights in random sampling.","section":"Appendix A.2"},{"comment":"The text says the baseline \"typically plateaus early (around 300 nodes)\" while Idea Search breaks the plateau \"around 500 nodes,\" but the figure is not visible in the text; please either include the plot in the main text or describe the node-axis and the number of trials per curve in the caption.","section":"Section 5.1, Figure 2"},{"comment":"The prompt examples in Appendix A.3 are helpful, but the manuscript would benefit from stating whether the sampled ideas are always all injected into the prompt or whether the LLM is allowed to ignore incompatible ideas; this affects how faithfully \"the idea is implemented\" is tracked in the score update.","section":"Section 3, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations and the effect is small, but the missing static-bank ablation is a central gap for a paper whose title and abstract center on the dynamic Idea Bank. The statistical presentation also needs work before the headline \"reliably breaks the plateau\" is justified. I would support a revised version that adds the frozen-bank control, reports confidence intervals or a small repeated-trials analysis, and clarifies the alpha-sweep configuration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a clean, readable paper with a genuine missing control. Injecting a curated idea bank into tree search does seem to nudge scores up on this benchmark, but the paper never tests whether the dynamic score-updating machinery matters. That leaves the core novelty claim under-supported.\n\nWhat it does well: the decomposition of expert methods into atomic ideas, the execution-scored rank/UCB sampling, and the conservative-vs-exploratory prompt split are all well-motivated and clearly described. The paper is also unusually honest. The \"Scope of the results\" paragraph says outright that the gains are modest relative to trial-to-trial variance, and the Limitations section flags single-task, single-LLM, coarse credit assignment, and the small number of trials. That candor earns real credit.\n\nThe main soft spot: there is no ablation that disables the dynamic updates. Both random and bandit sampling use the expanding bank and updated scores; random sampling just ignores the scores for selection but still receives newly discovered ideas. Without a frozen-initial-bank control, the reported improvement could come entirely from having a static curated list of ideas plus prompt conditioning. The paper's title and framing emphasize the dynamic bank, so this is a load-bearing omission, not a nitpick.\n\nSecondary concerns, in proportion: the mean gain over baseline is about 0.019, comparable to the standard deviation across five trials (0.008–0.018). The abstract's \"reliably breaks the plateau\" overstates that; the body's own scope note is more accurate. Five trials per configuration is low power, and the paper does not release code, which makes the results hard to check. There are also a few duplicate ideas in the appendix bank, which is sloppy but minor.\n\nOverall: the paper is worth engaging with seriously. The missing ablation is fixable with a single additional experimental condition, and the honesty and clarity of the writing make it a good candidate for review. I would send it to peer review and ask for the frozen-bank control and at least a few more seeds before publication.","headline":"A plausible, honestly-reported method with a real missing control: the 'dynamic' idea bank is never compared to a frozen one, so the central mechanism is unproven despite a small but consistent score gain.","tokens_in":11602,"tokens_out":2133,"would_cite":false,"duration_ms":22902,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Guiding Tree Search with a dynamic Idea Bank of decomposed method components reliably breaks its plateau on scRNA-seq batch integration, lifting the mean score from 0.678 to 0.697 and the best solution to 0.728.","keywords":["Idea Search","Tree Search","test-time scaling","LLM code generation","scRNA-seq batch integration","Idea Bank","bandit sampling","automated scientific discovery"],"falsifier":"Run the same Idea Search loop but replace the sampled textual ideas with semantically irrelevant or shuffled labels while holding prompts, bank size, and update dynamics fixed; if the plateau-breaking and outlier gains persist, the effect is not driven by idea content. Conversely, an ideal test would show score gains vanish when ideas are randomly misassigned to solutions.","tokens_in":10664,"feed_emoji":"🧬","tokens_out":5367,"duration_ms":45193,"temperature":0.7,"pith_summary":"Idea Search claims that test-time scaling of LLMs by Tree Search can be systematically guided by a dynamic 'Idea Bank' of atomic method components, and that doing so breaks the performance plateau of an unguided strong Tree Search baseline. On single-cell RNA-sequencing batch integration, the framework raises the mean score from 0.678 to 0.697 (bandit and random sampling both reach about 0.697) and the best discovered solution from 0.694 to 0.728. The paper argues that the gains are attributable to the idea mechanism rather than to uniform extra computation, since design choices reshape where gains appear: augmenting the bank helps bandit sampling but not random sampling, an exploratory prompt surfaces rare best solutions, and increasing bandit exploration (alpha equals 4) hurts. The evidence is deliberately narrow: one task, one backbone LLM, validation-split scoring, and five trials per configuration.","feed_headline":"Idea-guided tree search breaks the plateau in cell-data integration","feed_subtitle":"A dynamic idea bank lifts mean score from 0.678 to 0.697 and best score from 0.694 to 0.728.","key_machinery":"The load-bearing object is the Idea Bank: a growing set of short textual descriptions of atomic, recombinable design choices (for example, 'condition the decoder on batch ID'), each maintained with an execution-verified score. Sampling is either uniform or bandit-based using a rank-adjusted UCB score $S_i = 1/r_i + \\alpha \\sqrt{N \\ln(1+T)/(1+t_i)}$, where $r_i$ is the rank of idea $i$ by mean score, $t_i$ is its implementation count, $T = \\sum_j t_j$, and $\\alpha$ controls exploration. The bank is initialized by decomposing expert methods (optionally augmented by LLM brainstorming) and expanded by decomposing every newly discovered solution. This machinery carries the argument because it turns code-level search into explicitly idea-level search, and because the paper's ablations vary exactly these components—sampling strategy, augmentation, and prompting—to locate the source of the gains.","core_discovery":"The central claim is that decomposing expert methods into reusable atomic ideas, storing them in a bank, and feeding sampled ideas into mutation prompts closes the loop between conceptual search and code search, enabling an automated framework to escape plateaus that pure Tree Search cannot. Idea Search runs a tree search in which each mutation is conditioned on one or more sampled ideas; after the child solution is executed and scored, an LLM decomposes it into ideas and updates the bank, with each idea's score stored as the average score of all solutions containing it. Under this loop, the paper reports reliable plateau-breaking: mean 0.678 ± 0.011 to 0.697, and best 0.728 versus the baseline's 0.694. The paper further claims that the design choices interact: the augmented bank benefits bandit sampling but not random sampling, exploratory prompting yields rare outlier solutions, and higher sampling-level exploration (alpha equals 4) is counterproductive.","pith_inferences":["A natural extension beyond the paper would be to replace the coarse average-score credit with counterfactual or Shapley-style credit across co-occurring ideas, which could sharpen which idea actually drove a gain.","If idea-level guidance is what matters, the same bank could be reused across tasks or domains; a cheap test would be to run Idea Search on a second benchmark using the same bank without retraining.","The finding that exploratory prompting plus random sampling produced the best outlier suggests that novelty is generated by implementation-level freedom, not by more bandit exploration; an untested design would fix alpha very low and rely on prompts for diversity.","The mechanism resembles a form of representation learning over method space: the bank is a compressed, human-readable latent space where each dimension is an executable idea; one could test this by measuring whether bank growth correlates with diversity of discovered solutions."],"forward_implications":["If the claim is right, a pure Tree Search baseline's plateau is not a fixed ceiling; injecting execution-verified conceptual guidance from a dynamic bank shifts both the average and the best case.","The interaction findings imply that the best configuration is not simply 'more ideas' or 'more exploration': bank augmentation pays off only with score-guided sampling, and sampling-level exploration should be kept low while prompting-level exploration is increased.","Because gains concentrate in rare outliers, evaluations of such systems should track best-of-n and distributional tails, not just mean scores, to capture the discovery value.","The same loop could apply wherever a scientific method can be summarized as a recombination of ideas and scored by execution, making the framework a candidate general test-time architecture for automated method development.","The paper's own scope caveats mean these are directional effects on one validation benchmark, not calibrated effect sizes."],"supporting_citations":[{"why":"Supplies the pure Tree Search baseline, the scRNA-seq batch integration task, and the expert method summaries that seed the bank; the plateau this paper breaks is defined relative to that system.","marker":"(Aygün et al., 2026)"},{"why":"Provides the OpenProblems v2.0.0 scoring protocol used for all validation-split scores.","marker":"(Luecken et al., 2025)"},{"why":"Supplies the finite-time multi-armed bandit framework underlying the UCB sampling strategy.","marker":"(Auer et al., 2002)"},{"why":"ComBat is one of the expert methods decomposed into atomic ideas (additive and multiplicative batch effects, empirical Bayes), and the baseline conceptually resembles it.","marker":"(Johnson et al., 2007)"},{"why":"Cited as evidence that Tree Search can enter unproductive loops, motivating the plateau-breaking goal.","marker":"(Antoniades et al., 2024)"},{"why":"Cited as evidence that Tree Search can get trapped in greedy local optima, motivating idea guidance.","marker":"(Jiang et al., 2025)"},{"why":"Provides the CZ CELLxGENE Discover dataset used for hill climbing during the search.","marker":"(Program et al., 2025)"}],"fun_headline_variants":["Idea bank guides tree search to escape local optima","Dynamic ideas improve automated scientific coding","Tree search plus idea bank beats plateau in cell data","Idea-guided tree search boosts cell-batch integration","Idea bank unlocks better tree search for science"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM's decomposition of a method into atomic ideas is faithful enough that the average score of all solutions containing an idea measures that idea's true contribution; if the decomposition is arbitrary or biased, the bank's guidance degenerates into random prompt perturbation.","fun_headline_variants_meta":{"raw":{"variants":["Idea bank guides tree search to escape local optima","Dynamic ideas improve automated scientific coding","Tree search plus idea bank beats plateau in cell data","Idea-guided tree search boosts cell-batch integration","Idea bank unlocks better tree search for science"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1690,"prompt_tokens":943,"completion_tokens":747,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":675}},"tokens_in":559,"tokens_out":747,"duration_ms":7604,"temperature":1.0,"reasoning_tokens":675,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:18:39.007846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same Idea Search loop but replace the sampled textual ideas with semantically irrelevant or shuffled labels while holding prompts, bank size, and update dynamics fixed; if the plateau-breaking and outlier gains persist, the effect is not driven by idea content. Conversely, an ideal test would show score gains vanish when ideas are randomly misassigned to solutions.","supporting_citations":[],"review_version":1}