Pith. sign in

REVIEW 2 major objections 5 minor 61 references

Agentic Auto-Research is Fuzz Testing

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that auto-research should be rearchitected as instrumented, feedback-directed search: a cheap epistemic progress signal steers the next experiment, and protected validation, not the optimized signal, certifies discovery.

desk verdict A careful, honest position paper that maps auto-research to greybox fuzzing, with three testable predictions—but its load-bearing assumption, the existence of a cheap robust progress signal, remains genuinely unproven. read the letter →

arxiv 2608.09855 v1 pith:TYRU34N7 submitted 2026-08-10 cs.AI cs.CL

Yifeng He , Jicheng Wang , Yinzhe Zhao , Jiachen Liu , Hao Chen This is my paper · ORCID
classification cs.AIcs.CL
keywords autonomousresearchagentsgreyboxfuzzinggenerate-and-rankepistemicprogresssignalfeedback-directedsearchprotectedvalidationsparsefeedbackcoverage-guided
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autonomous research agents can now generate experiments faster than researchers can validate them. The paper argues that the dominant generate-and-rank response, asking a stronger proposer for more candidates and scoring them with a learned judge afterward, misses the real problem of sparse feedback. Its central claim is that an auto-research agent should run the control loop of a greybox fuzzer, propose a candidate, execute it, read a cheap signal of epistemic progress, and let that signal decide what to try next, while a separate, protected validation step decides what counts as a discovery. If true, research systems would need to expose measurable progress on every experiment rather than only build better generators, and the field would gain a concrete protocol for testing whether any proposed progress signal, search policy, or validator actually works.

What carries the argument

The load-bearing mechanism is the greybox-fuzzing control loop transferred to research: propose, execute, observe, choose next. Its named pieces are the epistemic progress signal, a cheap dense reading that tracks whether an experiment rules out an explanation, locates a boundary, sharpens a prediction, or changes which experiment should run next, the coverage analogue; the search policy, which allocates budget and chooses the next intervention from that signal rather than only ranking completed runs; and the protected validator, a final oracle that certifies discoveries using evidence shielded from the signal's adaptive optimization. The argument's work is done by this separation of roles: guidance must be observable on every execution, and nothing on the guidance path may double as the verdict.

What would settle it

A controlled campaign on a fixed declared research problem, with the proposer, task, validator, and total budget fixed, in which candidate progress signals are tested for predictive validity against protected outcomes and feedback-directed search is compared with repeated sampling on validated discoveries per unit cost; if no signal predicts protected progress and feedback-directed search does not win, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that an autonomous research agent is, in control-structure terms, a greybox fuzzer: within a declared research problem it proposes a candidate experiment, executes it, observes feedback, and uses that feedback to choose the next candidate. The generate-and-rank paradigm that currently dominates auto-research throws away the intermediate information that each execution provides and therefore suffers from sparse feedback. The paper argues for splitting the evaluate box: a cheap, dense epistemic progress signal, the analogue of coverage, steers the search, while final validation, the analogue of a crash or sanitizer oracle, protected from adaptive reuse of the signal, decides what counts as a discovery. The paper states three falsifiable predictions: candidate signals should predict protected outcomes and allocate a fixed validation budget better than an uninformative or terminal-only signal; feedback-directed search should beat repeated sampling in validated discoveries per unit cost with proposer, task, validator, and budget fixed; and protected validation should reduce false discoveries relative to reporting the optimized proxy.

Load-bearing premise

The load-bearing premise is that some cheap, dense, optimization-robust reading of epistemic progress exists for every declared research problem; if no such signal can be found, the feedback loop has nothing to steer and the fuzzing analogy degenerates into repeated sampling with extra bookkeeping.

Editorial extensions

If this is right

  • Built this way, every experiment would be instrumented to expose partial progress before final validation is available.
  • Generate-and-rank pipelines would keep ranking only as a final selection rule, not as the search policy itself.
  • Evaluation would shift to cost-normalized, validator-confirmed discoveries in head-to-head comparisons against repeated sampling.
  • Progress signals that satisfy the paper's requirements could be reused across campaigns like coverage instrumentation is reused across programs.
  • An optimized proxy gain would no longer be reportable as a discovery; protected validation would become the only gate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same steering and certifying split could make human research workflows more efficient: cheap summary statistics and discordant results allocate expensive replication, and replication supplies the verdict.
  • Read as a design recipe, the three predictions suggest a benchmark suite in which every proposed epistemic signal is scored on predictive validity, gaming resistance, and cost before deployment.
  • If scientific outputs become machine-executable artifacts, coverage-style instrumentation can be applied directly to them, making the proposed loop testable in practice.
  • The distinction also predicts that research agents that conflate guidance and verdict should show inflated apparent success followed by sharp reversals under protected evaluation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that autonomous research agents should be architected not as generate-and-rank systems but as instrumented, feedback-directed searches in the style of greybox fuzzing. It proposes three separable claims: (1) each experiment should expose a cheap, dense epistemic progress signal before final validation (requirements C1-C3 in Section 3); (2) the search policy should use that signal to choose the next intervention, yielding more validated discoveries per unit cost than repeated sampling (Section 4); and (3) final validation must be protected from adaptive reuse of the progress signal, reducing false discoveries (Section 5). The paper is a position piece: it provides an analogy, a component-by-component mapping, and three falsifiable predictions, but it runs no new experiments. It candidly identifies the existence of a C1-C3-compliant progress signal as the central open question and hands it to the field in Section 8.

Significance. If the framework is adopted, it would give auto-research systems a concrete architectural blueprint and a diagnostic vocabulary for failure (guidance vs. search vs. validation). The three predictions are genuinely testable, and the Signal Prediction in Section 3 is explicitly falsifiable. The paper is unusually honest about its own load-bearing assumption, and it makes good use of existing results (the Angora ablation, the MLE-bench counterfactual, and the reward-overoptimization literature) to motivate the split between guidance and verdict. The historical discussion in Section 6 usefully connects current auto-research to earlier self-improving systems. The main weakness is that the antecedent of the central conditional—the existence of a cheap, dense, optimization-robust epistemic progress signal—is not only unverified but appears difficult to satisfy for the paper's own candidate proxies; the manuscript does not supply even a toy instantiation or a principled argument that such a signal must exist.

major comments (2)
  1. [Section 3 (C1-C3) and Section 8] The central claim is conditional on the existence of an epistemic progress signal satisfying C1-C3, but the manuscript provides no evidence that such a signal exists, and the candidate proxies listed in Section 3 each appear to violate at least one of the requirements. Novelty against prior work is cheap but trivially gameable by exploring irrelevant corners (C3); verifier confidence and cross-replicate disagreement are either expensive or sparse (C1) or known to be gameable under optimization (C3), as the paper itself notes in Section 6; surprise against a model requires a model-rich likelihood, which conflicts with the paper's own cost constraint (C1). The paper candidly labels this "the open question we hand the field" (Section 8), but the entire architecture depends on a positive answer: if no signal can satisfy C1-C3, Lesson One fails and Lesson Two's feedback-directed search is indistinguishable from repeated sampling with a context-aware proposer. This is load-bearing, not a presentation issue. I recommend either (a) explicitly restating the thesis as a conditional position and softening the title and abstract accordingly, or (b) providing at least one concrete exemplar—for example, a retrospective or synthetic pilot in which a specific proxy passes C1-C3 on a declared research problem—or a principled argument that such a signal must exist whenever a research problem is declarable.
  2. [Section 4 (Search prediction)] The Search prediction is not falsifiable as stated because the information available to the repeated-sampling baseline is unspecified. The text says the comparison holds "with the proposer, task, protected validator, and total budget fixed," but it does not state whether the proposer in the repeated-sampling arm sees the history of experimental outcomes. If the proposer is a context-window LLM that can condition on prior experiments, the two arms may differ only in the ranking mechanism, not in feedback-directed generation; if the proposer is forbidden from seeing any outcomes, the baseline is artificially weak and the comparison does not isolate the value of using intermediate feedback to select the next intervention. The paper should specify the information sets of both arms (for example, the repeated-sampling proposer sees only the problem statement and the candidate pool) and state how the "next intervention" is distinguished from a "completed sample" in operational terms. This is necessary for Lesson Two to be tested fairly.
minor comments (5)
  1. [Section 3] The term "epistemic progress" is used as a primitive; giving an informal definition (for example, change in the agent's posterior over a hypothesis space, or reduction in the set of live hypotheses) would make C2 and the Signal Prediction less vulnerable to circularity.
  2. [Section 4] The Angora ablation numbers (97.0% vs. 77.4% on xmlwf) are reported without a pointer to the specific table or figure in [11]; please provide the precise location and clarify whether the comparison is on branch-constraint solving or on downstream bug detection.
  3. [References [1] and [2]] References [1] and [2] are a community newsletter and an X post, respectively; for claims attributed to Tworek and to the Auto-Research Summit, the paper should cite peer-reviewed or otherwise archival sources if any exist, or mark these as informal communications.
  4. [Figure 3] The four panels in Figure 3 are unlabelled; adding axis labels or short panel captions would help readers connect the schematic to the candidate proxies (surprise, novelty, verifier confidence, distance to target) discussed in the text.
  5. [Section 6] The historical claim "Each wave stalled where the fuzzing loop predicts" is an interpretive summary; adding specific citations for each claimed stall (for example, the NAS overfitting claim beyond [53]) would strengthen the narrative.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the three predictions are empirical, externally testable, and not fitted from the paper's own assumptions; a few self-citations support the analogy but are not load-bearing.

full rationale

The paper's central claim is explicitly staked on three controlled comparisons, not on a derivation from its own framework: 'The position stands or falls on the three controlled comparisons stated above: candidate signals must predict protected outcomes, feedback-directed search must beat repeated sampling per validated discovery, and protected validation must reduce false discoveries relative to the optimized proxy.' These are forward-looking empirical tests, and the paper does not run them or fit any parameter that is then relabeled as a prediction. The progress-signal requirements C1-C3 are criteria for evaluating candidate proxies, and Section 8 openly labels the existence of such a signal 'the open question we hand the field,' so the framework does not smuggle in its own antecedent. The self-citations ([21] Content Fuzzing, [58] LLAMA-FUZZ, and [27] agent-native artifacts) provide background or supporting examples for the fuzzing analogy, but the analogy is argued structurally from the fuzzing literature (AFL, AFL++, Angora, directed greybox fuzzing) and external results (Brown et al., MLE-bench, Dwork et al., Gao et al.), so the central claim does not reduce to those self-citations. No equation-level reduction, uniqueness argument, or fitted-input-as-prediction pattern appears. The mild score reflects only the presence of author self-citations in support of the analogy's generality; none is load-bearing.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

No numbers are fitted. The paper rests on presuppositions: the transferability of the fuzzing control loop, the existence of usable progress signals, and the possibility of protected validation. The epistemic progress signal is a proposed construct with no independent evidence yet; the paper's own proposed experiments are the handle to test it.

assumptions (4)
  • domain assumption The greybox fuzzing control loop is a valid model for search within a declared research problem.
    Section 2 maps each fuzzing component (seeds, mutation, coverage, oracle, scheduling) to a research component; Section 7 bounds the analogy to declared problems with explicit progress representations.
  • ad hoc to paper A cheap, dense, optimization-robust epistemic progress signal exists for research.
    Section 3 requirements C1-C3 define the desired properties but list only candidate proxies; the paper treats the signal's existence as its central open question.
  • domain assumption Final validation can be protected from adaptive reuse of the search guidance signal.
    Section 5 relies on protected validation (hidden test, replication, assay, independent review) to certify discoveries; the MLE-bench analysis illustrates the need, but the paper does not demonstrate such protection works for open-ended research.
  • domain assumption Research artifacts can be machine-executable, exposing a surface for progress signals.
    Section 6, referencing Liu et al. [27], states the proposals presuppose artifact-native outputs that agents can parse and run; this surface is not yet standard.
invented entities (1)
  • Epistemic progress signal
    purpose: A cheap, dense measurement that exposes partial scientific progress before final validation, used to steer the next experiment.
    Introduced in Section 3 as the research analogue of coverage, with candidate proxies (surprise, novelty, verifier confidence, distance to target) but no measured handle yet. The paper's three predictions are the intended test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agentic Auto-Research is Fuzz Testing." pith.science (2026). https://pith.science/paper/TYRU34N7

@misc{pith2026260809855,
  author       = {Pith},
  title        = {Pith review of: Agentic Auto-Research is Fuzz Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYRU34N7}},
  note         = {Machine review of arXiv:2608.09855}
}
read the original abstract

Autonomous research agents can generate experiments faster than researchers can validate them. Researchers have responded by scaling the proposer and ranking more samples with a learned judge or human reviewers. We argue that this *generate-and-rank* paradigm misses the problem of sparse feedback. Within a declared research problem, an agent follows the control loop of a greybox fuzzer: it proposes a candidate, executes it, observes feedback, and chooses what to try next. A fuzzer rarely finds a bug, but coverage makes partial progress observable on every execution. Fuzzers then use that signal to mutate inputs and allocate effort, rather than only to rank completed runs. Auto-research needs the same two capabilities. First, each experiment should expose a cheap, dense signal of epistemic progress before final scientific validation is available. Second, that signal should determine the next intervention so that the agent searches rather than repeatedly samples. Because the optimized progress signal is guidance rather than a verdict, final validation must still decide what counts as a discovery using evidence protected from adaptive reuse. We propose controlled tests of whether candidate signals predict validated progress, whether feedback-directed search yields more validated discoveries per unit cost than repeated sampling, and whether protected validation reduces false discoveries. Feedback architecture, not only generation, is a central bottleneck in auto-research.

Figures

Figures reproduced from arXiv: 2608.09855 by the authors.

Figure 1
Figure 1. Auto-research as feedback-directed search. Initial directions enter a state pool. A scheduler selects a state, the proposer generates candidates, and experiments expose intermediate signals. Comparison retains promising results as new states and closes the loop. A protected validator separately certifies discoveries without steering the search. do not establish a finding. Auto-research therefore needs an intermediat… view at source ↗
Figure 2
Figure 2. Splitting the evaluate box. (a) The common research cycle scores each experiment with a single evaluation that both steers the search and certifies the result. (b) Fuzzing separates the two roles: a cheap progress signal steers the next proposal, while protected validation certifies discoveries and never feeds the search. two loops component by component. The correspondence concerns control structure, not literal eq… view at source ↗
Figure 3
Figure 3. Intermediate signals make partial research progress ob￾servable before final validation. Each instrument reveals a different kind of structure in the search landscape, even though the distant objective remains unresolved. in the search landscape while the distant finding remains uncertified. Coverage gives fuzzers one answer to the resulting de￾sign question: what intermediate metric can expose useful progress befor… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A scheduler concentrates a fixed budget on the directions whose progress readings improve, and prunes the rest early. Be￾cause the supply does not grow, pruning D3 and D5 frees flow for D2. Directed greybox fuzzers schedule energy the same way, by distance to a target …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 39 canonical work pages

  1. [1]

    AGI House community newslet- ter, July 23, 2026

    AGI House.Jerry Tworek & Core Automation: Auto-research is Moving from Benchmarks to Real- World Loops. AGI House community newslet- ter, July 23, 2026. 2026.URL: https : / / agihousecommunity . beehiiv . com / p / jerry - tworek - core - automation - auto - research - is - moving - from - 7 Agentic Auto-Research is Fuzz Testing benchmarks- to- real- worl...

    work page 2026

  2. [2]

    X article, July 31, 2026

    AGI House.More Inference, More Experiments: What We Learned Hosting the Auto-Research Summit. X article, July 31, 2026. 2026.URL: https : / / x . com / agihouse _ org / status / 2083323121713733661 (visited on 08/02/2026)

    work page 2026

  3. [3]

    The Oracle Problem in Software Testing: A Survey

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. “The Oracle Problem in Software Testing: A Survey”. In:IEEE Transactions on Software Engi- neering41.5 (2015), pp. 507–525.DOI: 10.1109/ TSE.2014.2372785

    arXiv 2015

  4. [4]

    Directed Greybox Fuzzing

    M. Böhme, V .-T. Pham, M.-D. Nguyen, and A. Roy- choudhury. “Directed Greybox Fuzzing”. In:Pro- ceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). 2017, pp. 2329–2344.DOI: 10.1145/3133956. 3134020

    doi:10.1145/3133956 2017

  5. [5]

    Coverage-based Greybox Fuzzing as Markov Chain

    M. Böhme, V . -T. Pham, and A. Roychoudhury. “Coverage-based Greybox Fuzzing as Markov Chain”. In:Proceedings of the 2016 ACM SIGSAC Confer- ence on Computer and Communications Security (CCS). 2016, pp. 1032–1043.DOI: 10 . 1145 / 2976749.2978428

    arXiv 2016

  6. [6]

    On the Re- liability of Coverage-Based Fuzzer Benchmarking

    M. Böhme, L. Szekeres, and J. Metzman. “On the Re- liability of Coverage-Based Fuzzer Benchmarking”. In:Proceedings of the 44th International Conference on Software Engineering (ICSE). 2022, pp. 1621– 1633.DOI:10.1145/3510003.3510230

    arXiv 2022

  7. [7]

    Autonomous chemical research with large language models

    D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. “Autonomous chemical research with large language models”. In:Nature624.7992 (2023), pp. 570–578. DOI:10.1038/s41586-023-06792-0

    doi:10.1038/s41586-023-06792-0 2023

  8. [8]

    Large Language Mon- keys: Scaling Inference Compute with Repeated Sam- pling

    B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V . Le, C. Ré, and A. Mirhoseini. “Large Language Mon- keys: Scaling Inference Compute with Repeated Sam- pling”. In:arXiv preprint arXiv:2407.21787(2024). URL: https : / / arxiv . org / abs / 2407 . 21787

    arXiv 2024

Show all 61 references
  1. [9]

    Exploration by Random Network Distillation

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov. “Exploration by Random Network Distillation”. In:In- ternational Conference on Learning Representations (ICLR). 2019.URL: https : / / openreview . net/forum?id=H1lJJnR5Ym

    2019

  2. [10]

    MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

    H. Chen et al. “MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research”. In:Ad- vances in Neural Information Processing Systems 38: Datasets and Benchmarks Track. 2025.URL: https : / / openreview . net / forum ? id = JX9DE6colf

    2025

  3. [11]

    Angora: Efficient Fuzzing by Principled Search

    P. Chen and H. Chen. “Angora: Efficient Fuzzing by Principled Search”. In:2018 IEEE Symposium on Security and Privacy (SP). IEEE, 2018, pp. 711–725. DOI:10.1109/SP.2018.00046

    2018

  4. [12]

    An Unsolvable Problem of Elementary Number Theory

    A. Church. “An Unsolvable Problem of Elementary Number Theory”. In:American Journal of Mathe- matics58.2 (1936), pp. 345–363

    1936

  5. [13]

    The reusable holdout: Preserv- ing validity in adaptive data analysis

    C. Dwork, V . Feldman, M. Hardt, T. Pitassi, O. Rein- gold, and A. Roth. “The reusable holdout: Preserv- ing validity in adaptive data analysis”. In:Science 349.6248 (2015), pp. 636–638.DOI: 10 . 1126 / science.aaa9375

    2015

  6. [14]

    Transforming Science with Large Language Models: A Survey on AI-Assisted Scien- tific Discovery, Experimentation, Content Generation, and Evaluation

    S. Eger et al. “Transforming Science with Large Language Models: A Survey on AI-Assisted Scien- tific Discovery, Experimentation, Content Generation, and Evaluation”. In:arXiv preprint arXiv:2502.05151 (2025).URL: https : / / arxiv . org / abs / 2502.05151

    2025

  7. [15]

    AFL++: Combining Incremental Steps of Fuzzing Research

    A. Fioraldi, D. Maier, H. Eißfeldt, and M. Heuse. “AFL++: Combining Incremental Steps of Fuzzing Research”. In:14th USENIX Workshop on Offen- sive Technologies (WOOT 20). USENIX Association, 2020

    2020

  8. [16]

    A Learning Machine: Part I

    R. M. Friedberg. “A Learning Machine: Part I”. In:IBM Journal of Research and Development2.1 (1958), pp. 2–13

    1958

  9. [17]

    Scaling Laws for Reward Model Overoptimization

    L. Gao, J. Schulman, and J. Hilton. “Scaling Laws for Reward Model Overoptimization”. In:Proceedings of the 40th International Conference on Machine Learning (ICML). V ol. 202. Proceedings of Machine Learning Research. PMLR, 2023, pp. 10835–10866. URL: https://proceedings.mlr....

    2023

  10. [18]

    Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I

    K. Gödel. “Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I”. In:Monatshefte für Mathematik und Physik38 (1931), pp. 173–198

    1931

  11. [19]

    Speculations Concerning the First Ul- traintelligent Machine

    I. J. Good. “Speculations Concerning the First Ul- traintelligent Machine”. In:Advances in Computers. V ol. 6. Academic Press, 1966, pp. 31–88

    1966

  12. [20]

    Accelerating Scientific Discov- ery with Co-Scientist

    J. Gottweis et al. “Accelerating Scientific Discov- ery with Co-Scientist”. In:Nature655.8122 (2026), pp. 487–496.DOI: 10 . 1038 / s41586 - 026 - 10644-y

    2026

  13. [21]

    Content Fuzzing for Escaping Information Cocoons on Digital Social Me- dia

    Y . He, Z. Tang, and H. Chen. “Content Fuzzing for Escaping Information Cocoons on Digital Social Me- dia”. In:Findings of the Association for Computa- tional Linguistics: ACL 2026. San Diego, Califor- nia, USA: Association for Computational Linguis- tics, 2026, pp. 11253–1127...

    2026

  14. [22]

    MLA- gentBench: Evaluating Language Agents on Ma- chine Learning Experimentation

    Q. Huang, J. V ora, P. Liang, and J. Leskovec. “MLA- gentBench: Evaluating Language Agents on Ma- chine Learning Experimentation”. In:Proceedings of the 41st International Conference on Machine Learning (ICML). V ol. 235. Proceedings of Machine Learning Research. PMLR, 2024, p...

    2024

  15. [23]

    ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineer- ing

    Y . Imajuku, K. Horie, Y . Iwata, K. Aoki, N. Taka- hashi, and T. Akiba. “ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineer- ing”. In:Advances in Neural Information Processing Systems 38: Datasets and Benchmarks Track. 2025. URL: https : / / openrevie...

    2025

  16. [24]

    Evaluating Fuzz Testing

    G. Klees, A. Ruef, B. Cooper, S. Wei, and M. Hicks. “Evaluating Fuzz Testing”. In:Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS). 2018, pp. 2123– 2138.DOI:10.1145/3243734.3243804

    2018

  17. [25]

    Abandoning Ob- jectives: Evolution through the Search for Novelty Alone

    J. Lehman and K. O. Stanley. “Abandoning Ob- jectives: Evolution through the Search for Novelty Alone”. In:Evolutionary Computation19.2 (2011), pp. 189–223.DOI:10.1162/EVCO_a_00025

    2011 doi

  18. [26]

    On a Measure of the Information Provided by an Experiment

    D. V . Lindley. “On a Measure of the Information Provided by an Experiment”. In:The Annals of Math- ematical Statistics27.4 (1956), pp. 986–1005.DOI: 10.1214/aoms/1177728069

    1956

  19. [27]

    Liu et al.The Last Human-Written Paper: Agent- Native Research Artifacts

    J. Liu et al.The Last Human-Written Paper: Agent- Native Research Artifacts. 2026. arXiv: 2604 . 24658 [cs.LG].URL: https://arxiv.org/ abs/2604.24658

    2026 arXiv

  20. [28]

    AIGS: Generating Science from AI- Powered Automated Falsification

    Z. Liu, K. Liu, Y . Zhu, X. Lei, Z. Yang, Z. Zhang, P. Li, and Y . Liu. “AIGS: Generating Science from AI- Powered Automated Falsification”. In:arXiv preprint arXiv:2411.11910(2024).URL: https://arxiv. org/abs/2411.11910

    2024 arXiv

  21. [29]

    The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha. “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery”. In:arXiv preprint arXiv:2408.06292(2024).URL: https://arxiv. org/abs/2408.06292

    2024 arXiv

  22. [30]

    Prompt Fuzzing for Fuzz Driver Generation

    Y . Lyu, Y . Xie, P. Chen, and H. Chen. “Prompt Fuzzing for Fuzz Driver Generation”. In:Proceed- ings of the 2024 ACM SIGSAC Conference on Com- puter and Communications Security (CCS). Salt Lake City, UT, USA: ACM, 2024, pp. 3793–3807.DOI: 10.1145/3658644.3670396

    2024

  23. [31]

    The Art, Science, and Engineering of Fuzzing: A Survey

    V . J. M. Manès, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo. “The Art, Science, and Engineering of Fuzzing: A Survey”. In:IEEE Transactions on Software Engineering47.11 (2021), pp. 2312–2331.DOI: 10 . 1109 / TSE . 2019 . 2946563

    2021

  24. [32]

    Illuminating Search Spaces by Mapping Elites

    J.-B. Mouret and J. Clune. “Illuminating Search Spaces by Mapping Elites”. In:arXiv preprint arXiv:1504.04909(2015).URL: https://arxiv. org/abs/1504.04909

    2015 arXiv

  25. [33]

    AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery

    A. Novikov et al. “AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery”. In:arXiv preprint arXiv:2506.13131(2025).URL: https : //arxiv.org/abs/2506.13131

    2025 arXiv

  26. [34]

    The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

    A. Pan, K. Bhatia, and J. Steinhardt. “The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models”. In:International Conference on Learning Representations (ICLR). 2022.URL: https : / / openreview . net / forum ? id = JYtwGwIL7ye

    2022

  27. [35]

    LLM Evaluators Recognize and Favor Their Own Generations

    A. Panickssery, S. R. Bowman, and S. Feng. “LLM Evaluators Recognize and Favor Their Own Generations”. In:Advances in Neural Infor- mation Processing Systems 37 (NeurIPS). 2024. URL: https : / / proceedings . neurips . cc / paper _ files / paper / 2024 / hash / 7f1f0218e45f541...

    2024

  28. [36]

    Curiosity-driven Exploration by Self-supervised Pre- diction

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. “Curiosity-driven Exploration by Self-supervised Pre- diction”. In:Proceedings of the 34th International Conference on Machine Learning (ICML). PMLR, 2017, pp. 2778–2787.URL: https : / / arxiv . org/abs/1705.05363

    2017 arXiv

  29. [37]

    Modern Bayesian Experimental De- sign

    T. Rainforth, A. Foster, D. R. Ivanova, and F. Bick- ford Smith. “Modern Bayesian Experimental De- sign”. In:Statistical Science39.1 (2024), pp. 100– 114.DOI:10.1214/23-STS915

    2024 doi

  30. [38]

    Towards Scientific Dis- covery with Generative AI: Progress, Opportunities, and Challenges

    C. K. Reddy and P. Shojaee. “Towards Scientific Dis- covery with Generative AI: Progress, Opportunities, and Challenges”. In:Proceedings of the AAAI Con- ference on Artificial Intelligence (AAAI). V ol. 39. 27. 2025, pp. 28601–28609.DOI: 10 . 1609 / aaai . v39i27.35084

    2025

  31. [39]

    Classes of Recursively Enumerable Sets and Their Decision Problems

    H. G. Rice. “Classes of Recursively Enumerable Sets and Their Decision Problems”. In:Transactions of the American Mathematical Society74.2 (1953), pp. 358–366

    1953

  32. [40]

    Mathematical Discoveries from Program Search with Large Language Models

    B. Romera-Paredes et al. “Mathematical Discoveries from Program Search with Large Language Models”. In:Nature625.7995 (2024), pp. 468–475.DOI: 10. 1038/s41586-023-06924-6

    2024

  33. [41]

    Agent Laboratory: Using LLM Agents as Research Assistants

    S. Schmidgall et al. “Agent Laboratory: Using LLM Agents as Research Assistants”. In:Findings of the Association for Computational Linguistics: EMNLP

  34. [42]

    Ultimate Cognition à la Gödel

    J. Schmidhuber. “Ultimate Cognition à la Gödel”. In: Cognitive Computation1.2 (2009), pp. 177–193

    2009

  35. [43]

    Can LLMs Generate Novel Research Ideas? A Large-Scale Hu- man Study with 100+ NLP Researchers

    C. Si, D. Yang, and T. Hashimoto. “Can LLMs Generate Novel Research Ideas? A Large-Scale Hu- man Study with 100+ NLP Researchers”. In:Inter- national Conference on Learning Representations (ICLR). 2025.URL: https : / / openreview . net/forum?id=M23dTGWCZy

    2025

  36. [44]

    Defining and Characterizing Reward Gaming

    J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. “Defining and Characterizing Reward Gaming”. In:Advances in Neural Infor- mation Processing Systems 35 (NeurIPS). 2022. URL: https : / / proceedings . neurips . cc / paper _ files / paper / 2022 / hash / 3d719fee332c...

    2022

  37. [45]

    Scaling LLM Test-Time Compute Optimally Can Be More Effec- tive than Scaling Parameters for Reasoning

    C. Snell, J. Lee, K. Xu, and A. Kumar. “Scaling LLM Test-Time Compute Optimally Can Be More Effec- tive than Scaling Parameters for Reasoning”. In:In- ternational Conference on Learning Representations (ICLR). 2025.URL: https : / / openreview . net/forum?id=4FWAwZtd2n

    2025

  38. [46]

    R. S. Sutton.The Bitter Lesson. Blog post, incompleteideas.net. 2019.URL: http : / / www . incompleteideas . net / IncIdeas / BitterLesson.html(visited on 07/19/2026)

    2019

  39. [47]

    The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies

    K. Swanson, W. Wu, N. L. Bulaong, J. E. Pak, and J. Zou. “The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies”. In:Nature646.8085 (2025), pp. 716–723.DOI: 10 . 1038 / s41586 - 025-09442-9

    2025

  40. [48]

    AI Research Agents for Ma- chine Learning: Search, Exploration, and General- ization in MLE-bench

    E. Toledo et al. “AI Research Agents for Ma- chine Learning: Search, Exploration, and General- ization in MLE-bench”. In:Advances in Neural In- formation Processing Systems (NeurIPS). 2025.URL: https : / / openreview . net / forum ? id = RwfrdKSgCE

    2025

  41. [49]

    On Computable Numbers, with an Application to the Entscheidungsproblem

    A. M. Turing. “On Computable Numbers, with an Application to the Entscheidungsproblem”. In:Pro- ceedings of the London Mathematical Societys2-42.1 (1936), pp. 230–265

    1936

  42. [50]

    Large Language Models are not Fair Evaluators

    P. Wang et al. “Large Language Models are not Fair Evaluators”. In:Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). 2024, pp. 9440– 9450.URL: https : / / aclanthology . org / 2024.acl-long.511/

    2024

  43. [51]

    RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts

    H. Wijk et al. “RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts”. In:Proceedings of the 42nd International Conference on Machine Learning. V ol. 267. Proceedings of Machine Learning Research. PMLR, 2025, pp. 66772–66832.URL: htt...

    2025

  44. [52]

    The AI Scientist- v2: Workshop-Level Automated Scientific Discov- ery via Agentic Tree Search

    Y . Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha. “The AI Scientist- v2: Workshop-Level Automated Scientific Discov- ery via Agentic Tree Search”. In:arXiv preprint arXiv:2504.08066(2025).URL: https://arxiv. org/abs/2504.08066

    2025 arXiv

  45. [53]

    NAS Evaluation Is Frustratingly Hard

    A. Yang, P. M. Esperança, and F. M. Carlucci. “NAS Evaluation Is Frustratingly Hard”. In:International Conference on Learning Representations (ICLR). 2020.URL: https://arxiv.org/abs/1912. 12522

    2020

  46. [54]

    Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review

    R. Ye et al. “Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review”. In:arXiv preprint arXiv:2412.01708 (2024).URL: https : / / arxiv . org / abs / 2412.01708

    2024 arXiv

  47. [55]

    Dolphin: Moving Towards Closed- loop Auto-research through Thinking, Practice, and Feedback

    J. Yuan et al. “Dolphin: Moving Towards Closed- loop Auto-research through Thinking, Practice, and Feedback”. In:Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Associa- tion for Computational ...

    2025 doi

  48. [56]

    Position: LLMs Can’t Jump

    T. Zahavy. “Position: LLMs Can’t Jump”. In:Pro- ceedings of the 43rd International Conference on Machine Learning (ICML). 2026.URL: https:// openreview.net/forum?id=klU4737opt

    2026

  49. [57]

    Zalewski.American Fuzzy Lop (AFL)

    M. Zalewski.American Fuzzy Lop (AFL). 2013.URL: https : / / lcamtuf . coredump . cx / afl/ (visited on 07/19/2026)

    2013

  50. [58]

    LLAMA- FUZZ: Large Language Model Enhanced Greybox Fuzzing

    H. Zhang, Y . Rong, Y . He, and H. Chen. “LLAMA- FUZZ: Large Language Model Enhanced Greybox Fuzzing”. In:Proceedings of the 7th ACM/IEEE Inter- national Conference on Automation of Software Test (AST). Rio de Janeiro, Brazil: ACM, 2026, pp. 34–45. DOI:10.1145/3793654.3793756

    2026

  51. [59]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    L. Zheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”. In:Advances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track. 2023. URL: https : / / proceedings . neurips . cc / paper _ files / paper / 2023 / hash / 91f18a1287b398d...

    2023

  52. [60]

    Neural Architecture Search with Reinforcement Learning

    B. Zoph and Q. V . Le. “Neural Architecture Search with Reinforcement Learning”. In:International Con- ference on Learning Representations (ICLR). 2017. URL: https : / / arxiv . org / abs / 1611 . 01578. 11

    2017

  53. [2025]

    5977–6043.DOI: 10

    Suzhou, China: Association for Computa- tional Linguistics, 2025, pp. 5977–6043.DOI: 10. 9 Agentic Auto-Research is Fuzz Testing 18653 / v1 / 2025 . findings - emnlp . 320. URL: https://aclanthology.org/2025. findings-emnlp.320/

    2025

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.