{"id":"b908e93a-980b-4d2d-ac5c-44df92392c23","arxiv_id":"2505.16938","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A closed-loop LLM-agent framework that auto-generates research ideas and code, reported to improve baseline performance on all 12 tasks it was tested on.","lead":"InternAgent is a multi-agent system that takes a research task and baseline code, automatically generates research ideas, converts them into detailed methods, writes and debugs code, and runs experiments. The paper reports performance gains over baselines on 12 AI and science tasks, with the largest gains on reaction yield prediction and enhancer activity prediction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are best-of-N selections over 10 ideas plus adaptive reruns; without a baseline-equivalent search control, the gains cannot be attributed to generated ideas rather than search budget.","rationale":"InternAgent is described in sufficient detail, the code is open-sourced, and the framework plausibly automates a closed research loop; I do not see a fatal flaw that makes the central claim false. The load-bearing weakness is in the comparison protocol rather than in the system design. Because the headline results are maxima over 10 generated ideas (each subject to adaptive reruns and debug retries), and because the reported 'average' explicitly excludes non-improving runs (Sec. 3.2), the numbers in Tables 1 and 2 are selected extremes. The baseline runs are not given an equivalent best-of-N budget, and most tasks lack error bars, so the extra gains cannot be cleanly attributed to 'innovative ideas' rather than to search over a large space of code modifications. The paper's own ablation in Table 8 confirms that the adaptive rerun loop contributes materially, which makes the attribution problem concrete rather than hypothetical. The reader's weakest-assumption analysis identified the same issue: the comparison assumes strong baselines and that the agent's budget is not the actual cause. I therefore agree with the CONDITIONAL verdict; the concern does not overturn the result but conditions it on supplying baseline-equivalent search controls, full per-idea distributions instead of censored averages, and error bars. The open-sourced baselines and code are real assets and should make these controls straightforward to run.","tokens_in":31602,"tokens_out":6114,"duration_ms":45433,"concrete_test":"Run a no-agent control on three representative tasks (AutoRYP, Auto2DCls, AutoSenCls): apply 10 simple stochastic perturbations to each baseline (random seeds, learning-rate/weight-decay variations, and small architecture modifications), with the same total run budget, debug retries, max-over-runs reporting, and censored average as in Tables 1 and 2. If this control achieves gains comparable to InternAgent's reported deltas (say, within 50% of them), then the claim that generated ideas drive the improvements is unsupported. A simpler complementary readout is to report the maximum over 10 random seeds of the unmodified baseline; if best-of-10 baseline closes most of the gap, the same conclusion follows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that InternAgent's generated ideas, not its search budget, drive the reported improvements, and the experimental protocol cannot support that attribution. Sec. 3.1.3 sets a large budget: 15 ideas, each evolved up to 4 generations, top 5 selected, up to 4 debug attempts, and up to 5 adaptive runs. Sec. 3.2 then reports 'Max Performance' over 10 tested ideas per task and defines 'Average Performance' as the average only over experiments with performance gains. Every headline number in Tables 1 and 2 is therefore a selected extreme, and the 'average' column is a censored statistic that discards failures. AutoVLM, for instance, has 1 improving idea out of 5 successful runs (Table 4), so its 'average' of 67.6 is just the single success. Table 8 shows the adaptive-evolution rerun loop itself raises max accuracy from 81.6 to 83.3 on Auto2DCls and improves the success rate on AutoRYP from 2/5 to 4/6, so the retry mechanism is a large part of the reported gain. Table 7 further shows the AutoRYP baseline is 24.2 ± 4.2 over 5 seeds, and the quoted baseline 27.6 is itself the maximum of those runs. Because the baseline is not given the same best-of-N protocol, the comparison conflates idea quality with selection over retries and random seed noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces InternAgent, a closed-loop multi-agent framework that autonomously performs scientific research: it generates and evolves research ideas, converts them into detailed methodologies, implements them in code, runs experiments, and iterates. The authors report experiments across 12 diverse tasks (reaction yield prediction, molecular dynamics, power flow estimation, time series forecasting, transcription prediction, enhancer activity prediction, sentiment classification, image classification, 3D point cloud classification, semantic segmentation, autonomous driving, and VLM fine-tuning). The central claim is that InternAgent's generated ideas improve baseline code across all 12 tasks, with headline examples including AutoRYP R2 from 27.6 to 35.4, AutoEAP HK-PCC from 0.65 to 0.79, and Auto2DSeg mIoU from 78.8 to 81.0. The paper also compares against DOLPHIN and AI-Researcher and presents a human evaluation of idea novelty.","tokens_in":31859,"tokens_out":6222,"duration_ms":51491,"significance":"If the central claim were established with controlled comparisons, this would be an important demonstration of end-to-end LLM-agent research: the task coverage is unusually broad, the full pipeline from idea generation to experimental verification is explicit, and the authors have open-sourced both baselines and generated code, which is a real reproducibility asset. However, the stress-test concern lands squarely: the experimental protocol does not isolate the contribution of generated ideas from the large search and retry budget, and several headline numbers are selected extrema or censored averages. The paper contains no formal derivation and no fitted quantitative model, so the issue is not circularity but attribution: the evidence as presented cannot support the abstract's claim that InternAgent's ideas, rather than its search budget, are responsible for the reported gains.","major_comments":[{"comment":"The central attribution claim is not supported by the experimental protocol. InternAgent is given a large budget: 10 tested ideas per task, up to 4 debug attempts, and up to 5 adaptive reruns (Aider) or 3 (OpenHands), while the baselines and DOLPHIN are not given the same number of random restarts, hyperparameter trials, or debug iterations. Table 8 shows that the adaptive-evolution rerun loop alone changes Auto2DCls max accuracy from 81.6 to 83.3 and AutoRYP improved/successful from 2/5 to 4/6, so a substantial part of the reported gain is the retry/search mechanism rather than the content of the generated ideas. Without a baseline-equivalent search control (for example, best-of-N baseline runs, or InternAgent without adaptive evolution and with one run per idea), the abstract's claim that InternAgent 'generates innovative ideas to enhance baseline code' is not established.","section":"Sec. 3.1.3; Tables 1, 2, 8"},{"comment":"The 'Average Performance' reported in Tables 1 and 2 is defined as the average across experiments with performance gains, which is a censored statistic. Tables 3 and 4 show that many ideas fail to run or fail to improve: AutoVLM has only 1 improving idea out of 5 successful runs (1/5/10), so its reported average of 67.6 is simply the single success, and AutoPCDet has 2/5/10. Reporting only the censored average makes the 'versatility across 12 tasks' claim appear stronger than the data warrant. The paper should report the full distribution over all 10 ideas, including failed runs, using an uncensored mean or median, and should present per-idea results where feasible.","section":"Sec. 3.2; Tables 1-4"},{"comment":"The baseline number for AutoRYP is inconsistent across claims. Table 1 and the abstract use baseline R2 = 27.6, but Section 1 and Table 7 report the baseline as 24.2 ± 4.2 over five repeated runs, and 27.6 is the maximum of those runs. The headline '27.6 to 35.4' therefore compares a best-of-N baseline against a best-of-N-plus-retries agent, while the mean-to-mean comparison in Table 7 is 24.2 to 34.8. The paper must state which statistic is being compared and use the same protocol for both the baseline and the agent; otherwise the reported improvement is inflated by seed selection.","section":"Table 7 vs. Table 1 and Abstract"},{"comment":"The human evaluation of idea novelty does not currently provide credible evidence for the novelty claim. The paper does not report whether evaluators were blinded to the source of the ideas, whether they were independent of the project team (Appendix A lists an 'Idea Human Evaluation Specialist' group within the project team), or any inter-rater reliability statistic. Appendix B.1 promises per-idea scores but only provides scoring criteria, not the detailed scores. Without independent, blinded review and agreement measures, the large score gaps in Table 10 could reflect expectation bias, and the claim that InternAgent generates more novel ideas than AI-Scientist-V2 is not substantiated.","section":"Table 10 and Appendix B"}],"minor_comments":[{"comment":"Section 3.1.3 says the idea generation agent generates 15 ideas and selects the top 5 after evolution, while Tables 1-4 state that 10 ideas are tested per task; please clarify how the 10 tested ideas are produced and how they relate to the 15 initial ideas and the evolution tree.","section":"Sec. 3.1.3; Tables 1-4"},{"comment":"The abstract says 'precision advanced' for 2D semantic segmentation, but the metric defined in Sec. 3.1.2 and reported in Table 2 is mIoU, not precision; please use the correct metric name.","section":"Abstract; Sec. 3.1.2"},{"comment":"The cost tables report training time and per-idea generation and debugging costs, but they do not report the total number of runs or the cost of the adaptive-evolution reruns; the 'Total Cost' in Table 9 should be defined consistently with the per-run costs in Tables 5 and 6.","section":"Tables 5, 6, and 9"},{"comment":"In Table 9, AI-Scientist-V2 is listed without performance numbers, and the text states it struggled to write runnable code; please provide the observed success rates, number of executable runs, and any partial results so the comparison is transparent.","section":"Table 9"},{"comment":"The statement that human researchers typically require several months to achieve a similar level of performance improvement is unsupported; please either provide a citation or soften the claim.","section":"Sec. 1"}],"recommendation":"major_revision","confidential_remarks":"The DOLPHIN baseline comparison is from a same-group system, which is not disqualifying but should be disclosed explicitly with configuration details. The paper would be substantially stronger if the authors added a baseline-equivalent search control and reported uncensored performance distributions; without those, the central claim remains an engineering demonstration rather than a controlled scientific result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"InternAgent is a genuine step forward in the practical engineering of autonomous research agents. The three-module design — self-evolving idea generation with human feedback, idea-to-methodology construction, and adaptive experimental planning — is clearly described, and repo-level coding support is a real advance over single-file systems like DOLPHIN. Open-sourcing the baselines and generated code is commendable. The 12-task sweep, including per-task success/failure counts, gives a useful picture of where such systems work and fail today.\n\nBut the empirical claims as stated are not supported. The 'average performance' is defined as the average only over runs that improved, which discards failures. The AutoRYP baseline is reported as 27.6, the best of five seeds, while the mean is 24.2 ± 4.2. The main baseline comparison is against DOLPHIN, a same-group system, and the adaptive-evolution ablation shows that a large part of the gain comes from the retry loop rather than the generated ideas. The human evaluation of idea novelty was run by the authors themselves (Appendix A lists the evaluation specialists), with no inter-rater reliability. The efficiency claim against human months is unsupported.\n\nThe stress-test note is on target: with 10 ideas, up to 4 debug attempts and multiple adaptive reruns, the reported max performance is a selected extreme. To attribute the gains to the ideas, you need a control where the baseline gets the same retry budget. Without that, the defensible claim is 'a well-resourced agent can improve baselines,' not 'generates innovative ideas.'\n\nThat said, this is not a toy. Repo-level autonomous coding and debugging is a meaningful capability worth reporting. The paper deserves peer review precisely to force a fix of the evaluation protocol: repeated seeds, full-run statistics, a retry-matched baseline, and an independent human evaluation. I would not cite the current numbers in my own work, but I would cite the system as related work once the protocol is cleaned up.\n\nRecommendation: send to peer review, flagging the evaluation statistics as a major revision.","headline":"Real engineering, unreliable numbers: the framework is a solid systems contribution but the headline gains are best-of-N artifacts rather than demonstrated idea quality.","tokens_in":32510,"tokens_out":3872,"would_cite":false,"duration_ms":28127,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InternAgent closes the loop from hypothesis to verification, improving baseline performance on all 12 scientific and AI tasks.","keywords":["multi-agent systems","autonomous scientific research","large language models","hypothesis generation","idea-to-methodology","automated code generation and debugging","closed-loop research pipeline","AI for science"],"falsifier":"Run InternAgent on the same 12 tasks with the idea-generation and self-evolution stages replaced by a random or fixed set of no-op 'ideas' while keeping the same code-debug and rerun budget; if the no-op pipeline matches InternAgent's improvements, the closed-loop hypothesis generation is not what drives the results.","tokens_in":31393,"feed_emoji":"🤖","tokens_out":8183,"duration_ms":63371,"temperature":0.7,"pith_summary":"InternAgent is an LLM-based multi-agent system that claims to automate the entire research loop: survey literature, generate and evolve candidate ideas, turn an idea into a step-by-step methodology, implement it as code, debug it, run experiments, and feed results back for another round. The paper's central evidence is that across 12 tasks—from chemical reaction-yield prediction and enhancer-activity prediction to semantic segmentation and vision-language fine-tuning—InternAgent improved the given baseline's score on every task, with headline gains such as reaction-yield $R^2$ rising from 27.6 to 35.4 in 12 hours and enhancer-activity correlation rising from 0.65 to 0.79 in 4 hours. A sympathetic reader would care because, if this holds, it shifts the bottleneck of applied research from manual implementation and trial-and-error to articulating the task; the framework would be a reusable instrument for turning hypotheses into verified code. The paper also claims that human experts rate InternAgent-generated ideas higher in soundness, contribution, and overall quality than ideas from prior automated research systems, and that the monetary cost per idea is only a few dollars.","feed_headline":"Autonomous AI agent beats baselines on all 12 research tasks","feed_subtitle":"From reaction-yield prediction to semantic segmentation, InternAgent turns its own hypotheses into working code in hours.","key_machinery":"The central mechanism is the closed-loop research cycle, carried by four connected modules. First, self-evolving idea generation: a Survey Agent reads roughly 50 papers, an Idea Innovation Agent generates 15 initial ideas and repeatedly evolves each into 3 variants, an Assessment Agent scores ideas on coherence, credibility, verifiability, novelty, and alignment while preserving diversity, and human feedback can steer the process. Second, idea-to-methodology construction: a Method Development Agent expands an idea into a formal, step-by-step method description, including pseudocode and equations, before coding starts. Third, evolutionary experimental planning and execution: the coder, Aider for small files and OpenHands for repositories, implements the method, and an exception-guided debugger fixes failures, with up to four debug attempts. Fourth, adaptive evolution: the Orchestration Agent re-plans after each run, keeping only components that empirically improve performance. This sequence is what lets an idea become a verified experimental result rather than just a proposal.","core_discovery":"On its own terms, the paper's discovery is that a single closed-loop multi-agent architecture can carry a research project from a natural-language task description to working improved code with no human in the loop except optional feedback. The architecture's load-bearing move is to convert each high-level idea into a detailed methodology, with formal equations and step-by-step algorithm descriptions, before any code is written; the paper argues this is why the coder can actually implement the idea. Multi-round experimental planning then rolls the methodology's components into the baseline incrementally, adapting the plan after each run. The quantitative claim is that InternAgent outperforms the baseline on all 12 tasks and outperforms DOLPHIN on every task where DOLPHIN runs, with the best ideas also more stable across repeated runs in the few-shot yield-prediction study. In human evaluation, reviewers gave InternAgent's ideas higher soundness, contribution, and overall ratings than AI-Scientist-V2's ideas across the four tasks compared.","pith_inferences":["Editorial inference: The reported numbers do not separate idea quality from search budget; a budget-matched comparison against random ideas or a hyperparameter search would clarify how much of the gain is the 'scientist' versus the compute spent.","Editorial inference: The idea-to-methodology step, with formal algorithms and equations, could be reused as a benchmark itself, allowing tests of which parts of a generated method description are necessary for a coder to reproduce the intended change.","Editorial inference: The same closed loop could be pointed at open problems where the 'experiment' is simulation or data analysis rather than training runs, such as choosing priors, designing loss functions, or selecting ablation studies, as long as evaluation is scriptable."],"forward_implications":["If correct, an autonomous research pipeline can take a task description and baseline repository and return improved, working code within hours, at API costs of roughly $0.6 to $1.2 per idea and training costs of 0.1 to 192 A100 GPU-hours per task.","The claimed gains across chemistry, molecular dynamics, power flow, time series, genomics, NLP, 2D and 3D vision, and VLM fine-tuning indicate the method is domain-agnostic rather than tuned to one benchmark.","Because the framework handles repo-level codebases in Auto2DSeg, AutoPCDet, and AutoVLM, it extends automated research beyond single-file toy experiments.","Human experts rated the generated ideas higher on soundness, contribution, and overall quality than those from prior automated systems, suggesting the pipeline produces proposals experts find plausible, not just code that runs.","The adaptive-evolution ablation in Table 8 indicates that iterative plan-refinement after each run is responsible for part of the gain; without it, success rates and improvements drop."],"supporting_citations":[{"why":"Supplies the Suzuki-Miyaura reaction dataset (5,760 reactions) used for the headline reaction-yield prediction benchmark.","marker":"(Perera et al., 2018)"},{"why":"Supplies the MD17 molecular dynamics benchmark used to test InternAgent on energy and force prediction.","marker":"(Chmiela et al., 2017)"},{"why":"Supplies the UMI-STARR-seq enhancer activity data used for the AutoEAP task.","marker":"(Arnold et al., 2013)"},{"why":"DeepSTARR is the AutoEAP baseline whose 0.65 HK-PCC score InternAgent improves to 0.79.","marker":"(de Almeida et al., 2022)"},{"why":"DOLPHIN is the prior auto-research system that InternAgent must beat on single-file tasks.","marker":"(Yuan et al., 2025)"},{"why":"AI-Scientist-V2 is the comparator for generated-idea human evaluation and for performance on AutoRYP and Auto2DCls.","marker":"(Yamada et al., 2025)"},{"why":"GPT-4o powers the survey, review, idea-generation, assessment, and orchestration agents, carrying the framework's reasoning load.","marker":"(Hurst et al., 2024)"},{"why":"Aider is the coding assistant used for single-file implementations in the debugging loop.","marker":"(Gauthier & Contributors, 2023)"},{"why":"OpenHands is the coding platform for repo-level baselines such as Auto2DSeg, AutoPCDet, and AutoVLM, enabling those results.","marker":"(Wang et al., 2024b)"}],"fun_headline_variants":["AI agent automates research, beats baselines on 12 tasks","Closed-loop agent turns hypothesis into working code","Multi-agent system outperforms baselines in 12 science tasks","Autonomous agent improves results across 12 scientific tasks","From hypothesis to code: AI agent boosts 12 research tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the baselines are strong, honestly reproduced reference points and that the gains come from InternAgent's generated ideas and methodology rather than from its extra search and rerun budget, which includes 10 candidate ideas, idea evolution, up to 4 debug attempts, and adaptive re-running of experiments.","fun_headline_variants_meta":{"raw":{"variants":["AI agent automates research, beats baselines on 12 tasks","Closed-loop agent turns hypothesis into working code","Multi-agent system outperforms baselines in 12 science tasks","Autonomous agent improves results across 12 scientific tasks","From hypothesis to code: AI agent boosts 12 research tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1270,"prompt_tokens":969,"completion_tokens":301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":220}},"tokens_in":585,"tokens_out":301,"duration_ms":2191,"temperature":1.0,"reasoning_tokens":220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:52:01.536202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run InternAgent on the same 12 tasks with the idea-generation and self-evolution stages replaced by a random or fixed set of no-op 'ideas' while keeping the same code-debug and rerun budget; if the no-op pipeline matches InternAgent's improvements, the closed-loop hypothesis generation is not what drives the results.","supporting_citations":[{"cited_title":"Deepstarr predicts enhancer activity from dna sequence and enables the de novo design of synthetic enhancers","cited_arxiv_id":null,"evidence_quote":"DeepSTARR is the AutoEAP baseline whose 0.65 HK-PCC score InternAgent improves to 0.79."},{"cited_title":"A platform for automated nanomole-scale reaction screening and micromole-scale synthesis in flow","cited_arxiv_id":null,"evidence_quote":"Supplies the Suzuki-Miyaura reaction dataset (5,760 reactions) used for the headline reaction-yield prediction benchmark."},{"cited_title":"u tt, and Klaus-Robert M \\","cited_arxiv_id":null,"evidence_quote":"Supplies the MD17 molecular dynamics benchmark used to test InternAgent on energy and force prediction."},{"cited_title":"Genome-wide quantitative enhancer activity maps identified by starr-seq","cited_arxiv_id":null,"evidence_quote":"Supplies the UMI-STARR-seq enhancer activity data used for the AutoEAP task."},{"cited_title":"Aider: Ai pair programming in your terminal","cited_arxiv_id":null,"evidence_quote":"Aider is the coding assistant used for single-file implementations in the debugging loop."}],"review_version":1}