{"id":"d76a7621-dd83-4aa9-b9ee-3a2342c6ee2e","arxiv_id":"2608.09855","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Autonomous research should be designed as feedback-directed search: cheap epistemic progress signals guide experiments, and protected validation, not the search score, certifies discoveries.","lead":"This paper argues that autonomous research agents should be built like software fuzzers, using cheap progress signals to guide experiments and reserving validation for protected final checks. It defines three testable predictions that could decide whether feedback architecture, not just generation, is what makes auto-research actually work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central premise—that a cheap, dense, robust progress signal exists—is openly acknowledged as unverified, and each proposed proxy appears to violate at least one of C1-C3; until a concrete signal passes all three in a controlled pilot, the fuzzing frame remains conditional.","rationale":"The reader's weakest assumption identifies exactly the load-bearing premise: a cheap, dense, robust epistemic progress signal must exist for the feedback-directed loop to start. My analysis sharpens this into a concrete risk: each candidate signal the paper lists appears to violate at least one of C1-C3, so the existence assumption is not merely untested but plausibly unsatisfiable for the proposed examples. The paper itself is well-structured and honest about this limitation, and its three predictions are the correct empirical targets. The verdict of CONDITIONAL is therefore appropriate; there is no internal inconsistency or unsupported factual claim that would require moving to REJECT or to ACCEPT. The recommendation remains UNCHANGED, with the concrete pilot test as the decisive next step.","tokens_in":12607,"tokens_out":2839,"duration_ms":34694,"concrete_test":"Run a pilot on MLE-bench or ALE-Bench with a fixed candidate pool per problem: compute candidate proxies (e.g., LLM self-consistency, novelty against prior work, verifier confidence, surprise against a fitted baseline) and measure their rank correlation with protected test outcomes. Then implement a feedback-directed search that uses the best proxy to choose the next intervention, and compare validated discoveries per unit cost against a generate-and-rank baseline with the same proposer and budget. If no proxy shows correlation above chance or the search does not beat the baseline under protected validation, predictions 1 and 2 fail and the fuzzing frame collapses to repeated sampling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The position rests on three controlled comparisons, but all three presuppose the existence of an intermediate signal satisfying C1-C3 from Section 3: cheap per candidate, correlated with genuine progress, and robust under optimization. The paper candidly labels this 'the open question we hand the field,' yet the examples named in Section 3 all look defective on inspection. Novelty against prior work is cheap but trivially gameable (a search can maximize it by exploring irrelevant corners), violating C3. Verifier confidence and cross-replicate disagreement are expensive or sparse, violating C1, and are known to be gameable under optimization (Section 6 cites reward overoptimization). Surprise against a model requires a model-rich likelihood, contradicting the paper's own cost constraint. If no proxy clears all three requirements simultaneously, Lesson One fails, and Lesson Two's feedback-directed search cannot be distinguished from repeated sampling with a context-aware proposer. The paper's Prediction 1 is the right test, but the paper does not run it, and the analogy supplies no existence proof. The concern is not that the paper is wrong; it is that the central claim is an unverified conditional whose antecedent may be false for every known proxy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that autonomous research agents should be architected not as generate-and-rank systems but as instrumented, feedback-directed searches in the style of greybox fuzzing. It proposes three separable claims: (1) each experiment should expose a cheap, dense epistemic progress signal before final validation (requirements C1-C3 in Section 3); (2) the search policy should use that signal to choose the next intervention, yielding more validated discoveries per unit cost than repeated sampling (Section 4); and (3) final validation must be protected from adaptive reuse of the progress signal, reducing false discoveries (Section 5). The paper is a position piece: it provides an analogy, a component-by-component mapping, and three falsifiable predictions, but it runs no new experiments. It candidly identifies the existence of a C1-C3-compliant progress signal as the central open question and hands it to the field in Section 8.","tokens_in":12853,"tokens_out":14015,"duration_ms":129401,"significance":"If the framework is adopted, it would give auto-research systems a concrete architectural blueprint and a diagnostic vocabulary for failure (guidance vs. search vs. validation). The three predictions are genuinely testable, and the Signal Prediction in Section 3 is explicitly falsifiable. The paper is unusually honest about its own load-bearing assumption, and it makes good use of existing results (the Angora ablation, the MLE-bench counterfactual, and the reward-overoptimization literature) to motivate the split between guidance and verdict. The historical discussion in Section 6 usefully connects current auto-research to earlier self-improving systems. The main weakness is that the antecedent of the central conditional—the existence of a cheap, dense, optimization-robust epistemic progress signal—is not only unverified but appears difficult to satisfy for the paper's own candidate proxies; the manuscript does not supply even a toy instantiation or a principled argument that such a signal must exist.","major_comments":[{"comment":"The central claim is conditional on the existence of an epistemic progress signal satisfying C1-C3, but the manuscript provides no evidence that such a signal exists, and the candidate proxies listed in Section 3 each appear to violate at least one of the requirements. Novelty against prior work is cheap but trivially gameable by exploring irrelevant corners (C3); verifier confidence and cross-replicate disagreement are either expensive or sparse (C1) or known to be gameable under optimization (C3), as the paper itself notes in Section 6; surprise against a model requires a model-rich likelihood, which conflicts with the paper's own cost constraint (C1). The paper candidly labels this \"the open question we hand the field\" (Section 8), but the entire architecture depends on a positive answer: if no signal can satisfy C1-C3, Lesson One fails and Lesson Two's feedback-directed search is indistinguishable from repeated sampling with a context-aware proposer. This is load-bearing, not a presentation issue. I recommend either (a) explicitly restating the thesis as a conditional position and softening the title and abstract accordingly, or (b) providing at least one concrete exemplar—for example, a retrospective or synthetic pilot in which a specific proxy passes C1-C3 on a declared research problem—or a principled argument that such a signal must exist whenever a research problem is declarable.","section":"Section 3 (C1-C3) and Section 8"},{"comment":"The Search prediction is not falsifiable as stated because the information available to the repeated-sampling baseline is unspecified. The text says the comparison holds \"with the proposer, task, protected validator, and total budget fixed,\" but it does not state whether the proposer in the repeated-sampling arm sees the history of experimental outcomes. If the proposer is a context-window LLM that can condition on prior experiments, the two arms may differ only in the ranking mechanism, not in feedback-directed generation; if the proposer is forbidden from seeing any outcomes, the baseline is artificially weak and the comparison does not isolate the value of using intermediate feedback to select the next intervention. The paper should specify the information sets of both arms (for example, the repeated-sampling proposer sees only the problem statement and the candidate pool) and state how the \"next intervention\" is distinguished from a \"completed sample\" in operational terms. This is necessary for Lesson Two to be tested fairly.","section":"Section 4 (Search prediction)"}],"minor_comments":[{"comment":"The term \"epistemic progress\" is used as a primitive; giving an informal definition (for example, change in the agent's posterior over a hypothesis space, or reduction in the set of live hypotheses) would make C2 and the Signal Prediction less vulnerable to circularity.","section":"Section 3"},{"comment":"The Angora ablation numbers (97.0% vs. 77.4% on xmlwf) are reported without a pointer to the specific table or figure in [11]; please provide the precise location and clarify whether the comparison is on branch-constraint solving or on downstream bug detection.","section":"Section 4"},{"comment":"References [1] and [2] are a community newsletter and an X post, respectively; for claims attributed to Tworek and to the Auto-Research Summit, the paper should cite peer-reviewed or otherwise archival sources if any exist, or mark these as informal communications.","section":"References [1] and [2]"},{"comment":"The four panels in Figure 3 are unlabelled; adding axis labels or short panel captions would help readers connect the schematic to the candidate proxies (surprise, novelty, verifier confidence, distance to target) discussed in the text.","section":"Figure 3"},{"comment":"The historical claim \"Each wave stalled where the fuzzing loop predicts\" is an interpretive summary; adding specific citations for each claimed stall (for example, the NAS overfitting claim beyond [53]) would strengthen the narrative.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a position piece with an untested central premise; if the journal's scope favors empirical contributions, this may be a poor fit. However, the falsifiable predictions, the explicit boundary statements in Section 7, and the honest labeling of the open question make it a legitimate candidate for an AI venue. The main risk is that the assertive title and the declarative conclusion overstate what is actually a conditional thesis. I do not see equation-level circularity; the predictions are not fitted quantities. The stress-test concern about the missing progress signal is real, but it is a fixable gap rather than a demonstration that the claim is false."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serious position piece, not a breakthrough. It argues that auto-research should be built as greybox fuzzing: a cheap, dense epistemic progress signal steers the search, while a protected validator decides what counts as a discovery. The fuzzing analogy isn't brand new—content fuzzing and AIGS have pieces of it—but the paper does something useful: it consolidates those pieces into a clean control loop and adds a clear separation between guidance and certification. Section 7 is especially good; it takes six plausible objections (BOED, novelty search, bitter lesson, etc.) and answers them without hand-waving.\n\nWhat's new is the packaging and the three predictions. The paper states them in a falsifiable way: signal predicts protected outcomes, feedback-directed search beats repeated sampling per unit cost, protected validation reduces false discoveries. That's a real contribution to how we'd test these systems.\n\nNow the soft spots. The central claim stands or falls on the existence of a cheap, dense, robust progress signal. The paper openly admits this is the open question, and the candidates it lists are all problematic. Novelty is trivial to game. Verifier confidence and cross-replicate disagreement are either expensive or known to overoptimize. Surprise requires a model that contradicts the cost constraint. So the antecedent might be false, and the paper doesn't give a concrete existence argument. That's the load-bearing gap. It's not a flaw in the logic—the paper is explicit that the condition is unverified—but it limits the practical weight of the position.\n\nA minor nit: the repeated-sampling baseline in Prediction 2 needs a precise definition. If the baseline is 'same proposer, no feedback,' fine, but the line between repeated sampling and feedback-directed search can blur when the proposer is itself context-aware.\n\nBottom line: it deserves a serious referee. The structure is clear, the boundaries are honest, and the predictions are testable. I don't share the stress-test's stronger claim that the analogy collapses; rather, it's a conditional with an unverified antecedent. That's exactly what a peer-review process should probe. I'd send it out.","headline":"A careful, honest position paper that maps auto-research to greybox fuzzing, with three testable predictions—but its load-bearing assumption, the existence of a cheap robust progress signal, remains genuinely unproven.","tokens_in":13369,"tokens_out":2576,"would_cite":true,"duration_ms":25542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that auto-research should be rearchitected as instrumented, feedback-directed search: a cheap epistemic progress signal steers the next experiment, and protected validation, not the optimized signal, certifies discovery.","keywords":["autonomous research agents","greybox fuzzing","generate-and-rank","epistemic progress signal","feedback-directed search","protected validation","sparse feedback","coverage-guided search"],"falsifier":"A controlled campaign on a fixed declared research problem, with the proposer, task, validator, and total budget fixed, in which candidate progress signals are tested for predictive validity against protected outcomes and feedback-directed search is compared with repeated sampling on validated discoveries per unit cost; if no signal predicts protected progress and feedback-directed search does not win, the paper's central claim is falsified.","tokens_in":12421,"feed_emoji":"🧪","tokens_out":7186,"duration_ms":66809,"temperature":0.7,"pith_summary":"Autonomous research agents can now generate experiments faster than researchers can validate them. The paper argues that the dominant generate-and-rank response, asking a stronger proposer for more candidates and scoring them with a learned judge afterward, misses the real problem of sparse feedback. Its central claim is that an auto-research agent should run the control loop of a greybox fuzzer, propose a candidate, execute it, read a cheap signal of epistemic progress, and let that signal decide what to try next, while a separate, protected validation step decides what counts as a discovery. If true, research systems would need to expose measurable progress on every experiment rather than only build better generators, and the field would gain a concrete protocol for testing whether any proposed progress signal, search policy, or validator actually works.","feed_headline":"Auto-research should fuzz experiments, not just rank them","feed_subtitle":"Cheap progress signals should steer research; protected validation alone should certify discoveries.","key_machinery":"The load-bearing mechanism is the greybox-fuzzing control loop transferred to research: propose, execute, observe, choose next. Its named pieces are the epistemic progress signal, a cheap dense reading that tracks whether an experiment rules out an explanation, locates a boundary, sharpens a prediction, or changes which experiment should run next, the coverage analogue; the search policy, which allocates budget and chooses the next intervention from that signal rather than only ranking completed runs; and the protected validator, a final oracle that certifies discoveries using evidence shielded from the signal's adaptive optimization. The argument's work is done by this separation of roles: guidance must be observable on every execution, and nothing on the guidance path may double as the verdict.","core_discovery":"The central claim is that an autonomous research agent is, in control-structure terms, a greybox fuzzer: within a declared research problem it proposes a candidate experiment, executes it, observes feedback, and uses that feedback to choose the next candidate. The generate-and-rank paradigm that currently dominates auto-research throws away the intermediate information that each execution provides and therefore suffers from sparse feedback. The paper argues for splitting the evaluate box: a cheap, dense epistemic progress signal, the analogue of coverage, steers the search, while final validation, the analogue of a crash or sanitizer oracle, protected from adaptive reuse of the signal, decides what counts as a discovery. The paper states three falsifiable predictions: candidate signals should predict protected outcomes and allocate a fixed validation budget better than an uninformative or terminal-only signal; feedback-directed search should beat repeated sampling in validated discoveries per unit cost with proposer, task, validator, and budget fixed; and protected validation should reduce false discoveries relative to reporting the optimized proxy.","pith_inferences":["The paper leaves implicit that the same steering and certifying split could make human research workflows more efficient: cheap summary statistics and discordant results allocate expensive replication, and replication supplies the verdict.","Read as a design recipe, the three predictions suggest a benchmark suite in which every proposed epistemic signal is scored on predictive validity, gaming resistance, and cost before deployment.","If scientific outputs become machine-executable artifacts, coverage-style instrumentation can be applied directly to them, making the proposed loop testable in practice.","The distinction also predicts that research agents that conflate guidance and verdict should show inflated apparent success followed by sharp reversals under protected evaluation."],"forward_implications":["Built this way, every experiment would be instrumented to expose partial progress before final validation is available.","Generate-and-rank pipelines would keep ranking only as a final selection rule, not as the search policy itself.","Evaluation would shift to cost-normalized, validator-confirmed discoveries in head-to-head comparisons against repeated sampling.","Progress signals that satisfy the paper's requirements could be reused across campaigns like coverage instrumentation is reused across programs.","An optimized proxy gain would no longer be reportable as a discovery; protected validation would become the only gate."],"supporting_citations":[{"why":"Defines greybox fuzzing and the coverage-guided control loop that the paper maps onto auto-research.","marker":"[31]"},{"why":"Shows how feedback from each execution shapes mutation and scheduling decisions, the model for using feedback to search.","marker":"[15]"},{"why":"Provides the ablation where gradient-guided search solves more branch constraints than random mutation, the template for the search prediction.","marker":"[11]"},{"why":"Documents log-linear gains and diminishing returns from repeated sampling, the empirical basis for claiming generate-and-rank plateaus.","marker":"[8]"},{"why":"Shows a research agent using dense ranking scores as guidance and physical experiments as the final verdict, the concrete guidance-versus-verdict example.","marker":"[20]"},{"why":"Reports the gap between validation-best and test-best agents, evidence for the value of protected validation.","marker":"[48]"},{"why":"Establishes that adaptive reuse of data breaks validity, the theoretical basis for requiring protected validation evidence.","marker":"[13]"},{"why":"Shows reward overoptimization under repeated optimization, supporting the claim that an optimized progress signal cannot certify a discovery.","marker":"[17]"}],"fun_headline_variants":["Fuzz your research: feedback beats ranking","Auto-research: coverage signals, not just ranking","The fuzzer mindset for AI-driven discovery","Feedback-directed search: fuzzing for science","Why research agents should fuzz, not just generate and rank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that some cheap, dense, optimization-robust reading of epistemic progress exists for every declared research problem; if no such signal can be found, the feedback loop has nothing to steer and the fuzzing analogy degenerates into repeated sampling with extra bookkeeping.","fun_headline_variants_meta":{"raw":{"variants":["Fuzz your research: feedback beats ranking","Auto-research: coverage signals, not just ranking","The fuzzer mindset for AI-driven discovery","Feedback-directed search: fuzzing for science","Why research agents should fuzz, not just generate and rank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1672,"prompt_tokens":963,"completion_tokens":709,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":635}},"tokens_in":579,"tokens_out":709,"duration_ms":6685,"temperature":1.0,"reasoning_tokens":635,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:25:52.334799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled campaign on a fixed declared research problem, with the proposer, task, validator, and total budget fixed, in which candidate progress signals are tested for predictive validity against protected outcomes and feedback-directed search is compared with repeated sampling on validated discoveries per unit cost; if no signal predicts protected progress and feedback-directed search does not win, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"The Art, Science, and Engineering of Fuzzing: A Survey","cited_arxiv_id":null,"evidence_quote":"Defines greybox fuzzing and the coverage-guided control loop that the paper maps onto auto-research."},{"cited_title":"AFL++: Combining Incremental Steps of Fuzzing Research","cited_arxiv_id":null,"evidence_quote":"Shows how feedback from each execution shapes mutation and scheduling decisions, the model for using feedback to search."},{"cited_title":"Accelerating Scientific Discov- ery with Co-Scientist","cited_arxiv_id":null,"evidence_quote":"Shows a research agent using dense ranking scores as guidance and physical experiments as the final verdict, the concrete guidance-versus-verdict example."},{"cited_title":"AI Research Agents for Ma- chine Learning: Search, Exploration, and General- ization in MLE-bench","cited_arxiv_id":null,"evidence_quote":"Reports the gap between validation-best and test-best agents, evidence for the value of protected validation."},{"cited_title":"The reusable holdout: Preserv- ing validity in adaptive data analysis","cited_arxiv_id":null,"evidence_quote":"Establishes that adaptive reuse of data breaks validity, the theoretical basis for requiring protected validation evidence."},{"cited_title":"Scaling Laws for Reward Model Overoptimization","cited_arxiv_id":null,"evidence_quote":"Shows reward overoptimization under repeated optimization, supporting the claim that an optimized progress signal cannot certify a discovery."}],"review_version":1}