{"id":"65731e9d-8bb2-4a07-89ba-a99eedd5abff","arxiv_id":"2607.17100","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Inner-loop AI-scientist choices survive a held-out test on 9 of 10 Matbench tasks; frozen feature plus model code combines for a 26.3% mean outer-holdout improvement.","lead":"The paper tests whether AI research agents make decisions that still hold up on data the search loop never sees. It runs separate searches over features, models, representations, and data on ten materials benchmarks, freezes each winner, and finds nine of ten inner-loop choices remain best on a held-out test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM pretraining may have leaked Matbench fold-0 test labels, so the 'held-out' evidence may not be unseen; the 9-of-10 transfer count could be inflated by memorization rather than generalization.","rationale":"I agree with the reader's weakest assumption: the outer holdout may be contaminated via LLM pretraining. This is the most load-bearing concern because the paper's headline empirical claim is the 9-of-10 transfer result; without that, the methodological contribution remains plausible but undemonstrated. The paper is otherwise carefully designed: axis isolation, freeze-before-evaluation, permutation test, and once-only outer scoring are all sound. The official-fold replay is correctly labeled as post-selection context, and the F+M compatibility test is explicitly excluded from the primary claim. However, the absence of code/data release and the lack of any contamination analysis leave the central evidence dependent on the unexamined assumption that DeepSeek-V4-Pro did not memorize public Matbench labels. A single direct memorization probe would test this cheaply. Given the reader already conditioned acceptance on resolving this, my recommendation is UNCHANGED (i.e., remain CONDITIONAL).","tokens_in":11065,"tokens_out":8532,"duration_ms":87286,"concrete_test":"Probe DeepSeek-V4-Pro for direct memorization of the outer holdout: sample 100-200 random entries from Matbench fold-0 (inputs only), and ask the model via API (no tools, no code execution) to state the target value, e.g., 'The experimental band gap of [composition] in the Matbench test set is ...' without revealing the label. Score exact or numerically close matches against ground truth. If the model reproduces a non-trivial fraction (e.g., >5%) of the holdout targets, the 'evidence the loop never sees' claim is false and the transfer results are potentially inflated. If the model cannot, the contamination concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: inner five-fold feedback selects the best tested intervention on nine of ten outer holdouts, with 0.228 pp regret and 89.3% ordering agreement. This depends on the outer holdout being genuinely unseen by the decision process. The protocol (Sections 3-4) only prevents the agent process from loading outer-holdout labels; it does not prevent the pretrained LLM (DeepSeek-V4-Pro) from having memorized Matbench fold-0 targets, which are public. If the model's code suggestions are implicitly conditioned on that memorized data, the outer-holdout 'audit' is not an out-of-sample test. The 9-of-10 count and ordering agreement could then reflect test-set contamination rather than transfer. The paper provides no contamination check (no training-data cutoff, no memorization probe, no de-contamination). This weakens the empirical demonstration of the methodology, though not the methodological framework itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces intervention-centered Auto Research, a workflow in which an LLM agent searches independently along Feature, Model, Representation, and Data axes using inner five-fold feedback, freezes each selected intervention, and then evaluates all frozen interventions on a Matbench outer holdout. The central empirical claim is that inner feedback selects the best tested intervention on nine of ten endpoints, with 0.228 percentage-point mean selection regret and 89.3% pairwise ordering agreement. The paper also reports a structure Representation intervention that inner feedback endorsed but the outer holdout rejected, and a post-hoc Feature+Model compatibility assembly that improves mean outer-holdout MAE from 19.0% to 26.3%.","tokens_in":11352,"tokens_out":7697,"duration_ms":76178,"significance":"If the 9-of-10 transfer result holds up, this is a meaningful methodological contribution: it provides a clean separation between adaptive search feedback and validation, makes individual research decisions the unit of evidence, and demonstrates an audit that rejects an overfit intervention. The protocol's code-freeze/SHA-256 manifest, once-only outer-holdout evaluation, and exact blocked permutation test are careful and creditable design choices. The negative Representation result is a useful illustration that outer validation can overturn inner feedback. The main uncertainty is whether the outer holdout is genuinely unseen by the pretrained LLM, which the current manuscript does not establish.","major_comments":[{"comment":"The core transfer claim depends on the outer holdout being genuinely unseen. The protocol only prevents the agent process from loading outer-holdout labels; it does not address memorization by DeepSeek-V4-Pro, a pretrained LLM. Matbench fold-0 targets are public and could appear in training data, so code suggestions could be implicitly conditioned on those targets. No contamination check (model cutoff, memorization probe, or decontamination) is reported. Thus the 9-of-10 count, 0.228 pp regret, and 89.3% agreement are not yet established as out-of-sample evidence. This is fixable by a private holdout or a documented cutoff/audit.","section":"§4, 'Outer-holdout protocol' and 'Agent and compute'"},{"comment":"The F+M compatibility result is selected after inspecting the outer-holdout matrix: the complementary profiles in that matrix motivate the deterministic union, and the same matrix is then used to score it. This is selection on the test set, so the 26.3% mean improvement is an optimistic post-selection estimate, not a fresh held-out result. The paper excludes it from the primary 9-of-10 claim, but the Abstract and Table 3 still present it as a headline. It should be reframed as exploratory, or validated on a new split if it is to support a compatibility claim.","section":"§5, 'Feature and Model Changes Remain Compatible' and Table 3"}],"minor_comments":[{"comment":"The structure baseline is described as 140 inputs (132 Magpie descriptors plus an eight-descriptor block), while the controlled comparison uses 141 inputs (132 plus nine light structure descriptors). Please reconcile this discrepancy.","section":"§4 'Materials Instantiation' vs §5 'Input type, not input dimension'"},{"comment":"The phrase 'evidence the loop never sees' is overstrong given the pretraining-leakage concern. Suggest 'evidence never loaded into the agent process' or an explicit caveat, in addition to addressing the underlying issue.","section":"Abstract and §3"},{"comment":"The text reports 701 attempts and 699 scored attempts; this is clear in the body, but consider noting it in the relevant figure/table caption to avoid an apparent inconsistency.","section":"§5 'Search dynamics'"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the LLM-pretraining leakage issue; if the authors can supply a contamination check or re-run with a private holdout, the core 9-of-10 claim would be convincing. The F+M 26.3% result should be demoted to post-selection/exploratory evidence. I do not see a fundamental flaw in the intervention-centered framework itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: read this before it disappears into the AI-scientist stack. The paper's actual contribution is a protocol—separate each editable axis, search with inner five-fold feedback, freeze code, then compare every frozen alternative on one untouched outer holdout. That gives you a matrix of selection regret, ordering agreement, and breadth instead of a single terminal score. The empirical vehicle is Matbench: 701 attempts across ten endpoints, inner feedback picks the best outer intervention on nine of them, with a permutation test over 2,880 relabelings (P=0.0014). The authors also report all 699 scored attempts, including failures, and they get a clean negative result: a structure Representation intervention that gains 1.4% inside the loop loses 3.0% on the outer holdout. A terminal-pipeline evaluation would have missed that sign reversal. The design really is new relative to the two anonymous prior works it cites—one certified a single intervention, the other evaluated a terminal recipe.\n\nNow the soft spots, in proportion. The biggest one is the one the stress-test note flags: Matbench fold-0 labels are public, the agent is a pretrained LLM, and the paper only prevents the agent process from loading outer labels. It never checks whether DeepSeek-V4-Pro memorized those labels during pretraining. If the model's proposal distribution is contaminated, the 9-of-10 count is not strong evidence of transfer. The protocol still works as a methodology—the selection is still based on inner CV, so the audit logic is intact—but the empirical claim needs a contamination probe or a re-run on a fresh holdout before I'd trust it. This is a moderate weakness, not a fatal one.\n\nSecond, the compatibility test (Feature+Model, 26.3%) is chosen after inspecting the same outer matrix. The paper clearly labels it post hoc and excludes it from the primary 9-of-10 result, but the abstract headlines it. That is selection-on-the-test-set, and the framing should be more honest.\n\nThird, no code or data is shipped, which is odd for a paper whose pitch is frozen, replayable, auditable interventions.\n\nBottom line: the protocol idea is solid, the empirical evidence is mostly careful, and the materials findings are plausible. This deserves a real referee, not a desk reject. I'd ask for a contamination analysis, a revised framing of the compatibility result, and a code release. I'd also bring it to a reading group focused on AI-scientist evaluation.","headline":"A genuinely useful audit protocol for AI-scientist search; the empirical claim is probably right but needs a contamination check and a more honest framing of the compatibility test.","tokens_in":11779,"tokens_out":4289,"would_cite":true,"duration_ms":40444,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Intervention-centered Auto Research—searching each possible change separately, freezing its code, then validating it on data the search loop never saw—survives on nine of ten materials-prediction benchmarks.","keywords":["intervention-centered Auto Research","held-out transfer","selection regret","axis-isolated search","outer-holdout matrix","materials property prediction","language-model agents","feedback overfitting"],"falsifier":"Re-run the same seven campaigns against a freshly collected materials-property holdout that was published after the agent's training cutoff, or use an agent whose pretraining provably excludes the benchmark labels; if the 9-of-10 transfer count collapses to chance on the replacement holdout, the reported transfer is leakage—if it persists, the audit is real.","tokens_in":10983,"feed_emoji":"🧪","tokens_out":3598,"duration_ms":36391,"temperature":0.7,"pith_summary":"The paper introduces intervention-centered Auto Research, a workflow where a language-model agent proposes changes confined to one declared axis—features, model, representation, or data—and each axis is searched independently under inner five-fold cross-validation. The best code on each axis is frozen, and then all alternatives are validated together on an outer holdout that the search loop never saw. Across ten materials-prediction benchmarks and 701 agent-executed attempts, the inner feedback selected the best tested intervention on nine outer holdouts, with 0.228 percentage-point mean selection regret and 89.3% agreement between inner and outer orderings. The audit also rejected one structure representation change that inner feedback alone would have endorsed. This design makes the research decision, not the terminal pipeline score, the unit of evidence, and it exposes which technical changes remain reusable after adaptive search.","feed_headline":"9 of 10 agent-selected research decisions pass a held-out audit","feed_subtitle":"Freezing each proposed change before testing on unseen data separates reusable discoveries from overfitted tweaks.","key_machinery":"The central mechanism is the frozen-code outer-holdout matrix. Each campaign searches one declared axis (Feature, Model, Representation, or Data) using only inner five-fold feedback; the selected code, configuration, and a SHA-256 manifest are frozen before any outer-holdout label is evaluated. Then a once-only outer-holdout matrix compares every frozen alternative against the baseline, producing selection regret, pairwise ordering agreement, endpoint breadth, and compatibility. This design makes adaptive search auditable because the validation evidence is structurally separated from the feedback that shaped the candidate.","core_discovery":"On the paper's own terms: a method that isolates interventions into four axes, freezes each selected code change, and evaluates the full set of frozen alternatives on a once-only outer holdout can separate search-driven overfitting from genuine transfer. The evidence shows that inner five-fold feedback picks the best tested intervention on nine of ten endpoints, with mean regret of 0.228 percentage points and pairwise ordering agreement of 89.3%. The one disagreement—two-dimensional exfoliation, the smallest structure holdout—produced 2.279 percentage-point regret. The audit also found an information-dependent hierarchy: on composition-only tasks, several axes worked; on structure-informed t","pith_inferences":["If replicated in other domains, this decision-level audit could become a standard evaluation pattern for agentic research, replacing single terminal scores with selection regret and ordering checks.","A direct extension would compare these language-model agents against fixed random or heuristic search on the same axes to measure how much of the transfer comes from agent proposals versus the validation protocol itself.","The sign-reversal example suggests practical Auto Research should always reserve a small untouched audit set before trusting any agent-endorsed change, especially when task samples are small.","Because the official fold-0 replay reuses examples from the search pool, one could rerun the audit on multiple genuinely fresh holdouts to estimate variance in the 9-of-10 transfer count."],"forward_implications":["Terminal pipeline scores can hide which technical decision carried a gain; the outer-holdout matrix attributes improvement to a specific axis.","Nine of ten inner-selected interventions remained best on untouched holdouts, implying that five-fold inner feedback transfers despite repeated reuse.","On structure tasks, separately found Feature and Model changes are complementary: their union improved all six structure endpoints and raised mean improvement from 19.0% to 26.3%.","The rejected Representation intervention—positive inside the loop, negative outside—is a concrete example of how single-loop validation can endorse an overfit change.","Information availability, not input dimension, predicts which interventions help: additional composition features did not replace sparse structural descriptors."],"fun_headline_variants":["Held-out audit passes 9 of 10 AI research decisions","AI research audit: 9 of 10 choices survive unseen data","Unseen data confirms 9 of 10 agent-selected research moves","Intervention audit: 9 of 10 AI decisions transfer, 1 rejected"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire audit rests on the claim, stated in the outer-holdout protocol, that the holdout labels never reach the agent process; if the pretrained language model's own training data already contained those labels and leaked them into its proposed code, the 'unseen' validation is contaminated and the 9-of-10 transfer count is inflated.","fun_headline_variants_meta":{"raw":{"variants":["Held-out audit passes 9 of 10 AI research decisions","AI research audit: 9 of 10 choices survive unseen data","Unseen data confirms 9 of 10 agent-selected research moves","Intervention audit: 9 of 10 AI decisions transfer, 1 rejected"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1214,"prompt_tokens":760,"completion_tokens":454,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":375}},"tokens_in":504,"tokens_out":454,"duration_ms":5365,"temperature":1.0,"reasoning_tokens":375,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:00:45.871424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same seven campaigns against a freshly collected materials-property holdout that was published after the agent's training cutoff, or use an agent whose pretraining provably excludes the benchmark labels; if the 9-of-10 transfer count collapses to chance on the replacement holdout, the reported transfer is leakage—if it persists, the audit is real.","supporting_citations":[],"review_version":1}