{"id":"0d11df71-ddc0-4690-9d69-2f8546012db8","arxiv_id":"2501.03916","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Dolphin closes the loop between idea generation, code implementation, and experimental feedback, improving on baselines and producing one 3D classification model comparable to a human-designed state-of-the-art.","lead":"Dolphin is an AI system that automatically generates research ideas, writes and debugs code to test them, and feeds experimental results back into the next round of idea generation. On three standard benchmarks and a small slice of MLE-bench, it improved on its baselines, and one 3D classification model it produced matched a human-designed state-of-the-art method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 93.9% ModelNet40 result is one selected run among 40 generated ideas with no seeds or error bars; the closed-loop and SoTA claims depend on proving this is not selection noise.","rationale":"The paper proposes a plausible closed-loop auto-research framework with concrete engineering components: task-attribute-guided paper ranking, traceback-guided local code structure debugging, and feedback memory. The ablations in Tables 4-5 give qualitative support for those components, and the public code is a real asset. The central claim, however, is stronger: that the loop continuously improves performance and that a generated method is comparable to SOTA. That claim is anchored in Table 1 and Table 7. I focused on the ModelNet40 result because it is the only 'SoTA-comparable' number in the abstract. The result is a selected maximum over 40 generated ideas, reported once, with no seed variance. Without repeated runs, we cannot distinguish a genuine architectural contribution from an outlier selected by multiple comparisons. The loop-improvement evidence in Table 6 is similarly underpowered: 21 successes total, no control arm, and no statistical test. The reader framed the weakest assumption as baseline fidelity; I agree that is a genuine concern, but it is not the most load-bearing. Even a perfectly faithful baseline leaves the selection-noise problem intact. My proposed test would quantify variance and control for the search process itself. I recommend keeping the reader's CONDITIONAL verdict: the framework deserves a chance, but the headline claims should not be accepted until the seed-controlled replication and placebo check are run.","tokens_in":21085,"tokens_out":7053,"duration_ms":73896,"concrete_test":"Run Dolphin's exact ModelNet40 pipeline 5 times with different seeds under the same 40-idea budget, and in parallel run a placebo condition where the 40 ideas are no-op or randomly perturbed code modifications selected by the same verification pipeline; report mean±std for baseline, per-loop improvement rate, and max-of-40 OA. If the placebo max-of-40 matches Dolphin's max within noise, or if Dolphin's max is not consistently above its own baseline across seeds, the central claims of continuous improvement and SoTA-comparable auto-research are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline 'comparable to SOTA' rests on a single number: Table 1 reports max improvement 93.9% OA on ModelNet40 vs baseline† 91.0, and Table 7 puts this 0.1% above GPSFormer. This number is the best of 40 ideas across two loops (Sec 4.2), with no repeated runs, no standard deviations, and no multiple-comparison correction. A best-of-40 selection under even modest per-run noise will systematically overestimate the reproducible gain; the +2.9 OA over baseline and the 0.1 OA gap to GPSFormer are both within plausible seed-to-seed variation that the paper never measures. The same issue undercuts the continuous-improvement claim: Table 6's loop rates are 2/7, 3/6, 4/8 successes, with no no-feedback control and no significance test, so the upward trend could be random fluctuation. The reader's baseline-fidelity concern is valid but not the deepest issue: even if the PointNet/WRN/BERT reimplementations exactly match their published protocols, the reported improvements and the SoTA comparison remain statistically unsupported. The paper needs seed-level replication and an appropriate control for the selection process.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dolphin, a closed-loop LLM-driven framework for automated scientific research that cycles through idea generation, experimental verification, and result feedback. On ModelNet40, CIFAR-100, SST-2, and selected MLE-bench tasks, the authors report that Dolphin generates ideas that improve over their implemented baselines, with the best ModelNet40 run (93.9% OA) slightly surpassing the published GPSFormer result (93.8% OA). The paper also introduces task-attribute-guided paper ranking and exception-traceback-guided debugging, and reports ablation studies for these components.","tokens_in":21321,"tokens_out":2976,"duration_ms":28637,"significance":"If the central claims hold, the work is a useful step toward closed-loop automated research: it provides one of the first frameworks where LLM-generated ideas are validated by actually running code on standard benchmarks, and the proposed debugging and paper-ranking components are concrete, plausible engineering contributions. The reported accuracy numbers are externally obtained from executed experiments, not merely from model self-reporting. However, the current evidence does not support the strength of the headline claims: the SoTA comparison rests on a single best-of-40 run without seeds or error bars, the closed-loop improvement trend is not statistically distinguishable from noise, and the novelty evaluation is performed by the same LLM family that generates the ideas. These issues are load-bearing for the abstract and the main experimental sections.","major_comments":[{"comment":"The statement that Dolphin's methods are 'comparable to the state-of-the-art' rests on a single best-of-40 run (93.9% OA on ModelNet40, Table 7), with no seed-level replication, no standard deviation, and no correction for the fact that 40 ideas were attempted. Since the maximum of many noisy candidates systematically overestimates the reproducible gain, the paper must report per-idea distributions across multiple seeds or a pre-registered holdout before the SoTA claim can be evaluated.","section":"Table 1, Table 7, Section 4.2"},{"comment":"The closed-loop 'continuous improvement' claim is based on improvement rates of 2/7, 3/6, and 4/8 over three loops, with no no-feedback control and no test of trend significance. As reported, the sequence is consistent with random fluctuation; a paired control using the same idea budget without feedback, together with an appropriate statistical test, is required to support the claim.","section":"Table 6, Section 4.3"},{"comment":"The novelty metric is gpt-4o's self-assessment of ideas generated by the same model, and the independence filter is a cosine threshold over summary embeddings. No human evaluation or external benchmark is provided to validate that the reported 19/20 novelty rate reflects genuine novelty. Please add a human-rated validation sample or an external novelty signal to break the circularity.","section":"Section 3.1, Table 3"},{"comment":"The 'Baseline†' is labeled as 'our implementation' and described as 'partially refers to the codebase', but the paper does not show that the reimplemented PointNet, WRN-28-10, and BERT-base match their canonical training protocols, hyperparameters, and published accuracies. If the reimplemented baselines are weaker than the original implementations, the reported average and max improvements, as well as the comparisons to GPSFormer, ResNeXt, and BERT-large, are inflated.","section":"Appendix B.1.2, Table 1"},{"comment":"Table 1 introduces two baselines for PointNet (89.2 and 91.0) without clearly explaining the relationship between 'Baseline' and 'Baseline†'. The text and the improvement numbers should state explicitly which baseline is used for the reported improvements, since the choice changes the magnitude and interpretation of the claims.","section":"Table 1, Section 4.2"}],"minor_comments":[{"comment":"The scheduler name is misspelled as 'CosineAnuealing'; it should be 'CosineAnnealing'.","section":"Appendix B.1.1"},{"comment":"The framework name is misspelled as 'DOPLINE' in this section; it should be 'DOLPHIN'.","section":"Appendix C.2"},{"comment":"The phrase 'We were supervised to observe' is unclear; likely 'surprised to observe' is intended.","section":"Section 1"},{"comment":"The header 'Number ideas' should read 'Number of ideas'.","section":"Table 1"},{"comment":"The keyword frequencies are not normalized by the number of papers in each retrieval condition, making the comparison across 'Naive' and 'Filter' hard to interpret; please report proportions or the underlying paper counts.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The statistical weakness is the main barrier: a best-of-40 comparison without seeds and a three-loop trend without controls cannot support the abstract's SoTA and continuous-improvement claims. The work is otherwise well within the scope of the journal and the engineering contributions are credible, so I would be willing to re-evaluate a revision that adds seed-level replication, a no-feedback control, and a human or otherwise external novelty validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Dolphin is a real closed-loop auto-research system, not just a wrapper around AI-Scientist. The two genuinely new pieces—task-attribute-guided paper ranking and traceback-guided local code structure for debugging—are concrete, well-motivated, and backed by ablations that show they help. The experimental verification is also real in a way many prior papers on LLM research agents are not: ideas get implemented and run on ModelNet40, CIFAR-100, SST-2, and three MLE-bench tasks, and the code and prompts are released. The Limitations section is honest about knowledge leakage and code-scale limits. That part deserves credit.\n\nThe soft spots are mainly about the strength of the headline claims, not about the existence of the framework. 'Continuously improve' is supported by two to three loops and small counts (2/7, 3/6, 4/8 successful ideas), with no no-feedback control and no significance test. The SoTA-comparable number—93.9% OA on ModelNet40, 0.1% above GPSFormer—is one selected run out of 40 generated ideas, with no seeds or error bars. Best-of-N selection under ordinary training noise will inflate the expected gain; +2.9 OA over their own baseline and the 0.1 gap to GPSFormer are both plausible seed-to-seed variation. The stress-test note is right that even perfect baseline fidelity would not fix this. The novelty evaluation is also self-referential: gpt-4o decides novelty of ideas gpt-4o generated. Baseline fidelity is a lesser but real question: Table 1 marks 'Baseline†' as their implementation, and Appendix B.1.2 says 'partially refers' to a codebase without a commit hash or comparison to published training outcomes.\n\nNone of this sinks the paper. The framework is sensible, the ablations are informative, and the direction is useful. What is missing is statistical discipline: repeated runs, multi-seed tables, a no-feedback control, and an independent or at least blinded novelty check. Also publish exact configs with a commit hash.\n\nWho it is for: researchers building LLM-driven research agents, and anyone thinking about what counts as reproducible evidence in auto-research. It deserves a serious referee. I would send it to review, with a clear request for seed-level replication and a no-feedback baseline before accepting the headline claims.","headline":"A genuinely closed-loop auto-research framework with real benchmark runs, but the headline SoTA claim rests on a single best-of-40 run with no seeds, so treat the numbers as promising, not established.","tokens_in":21873,"tokens_out":2332,"would_cite":true,"duration_ms":22186,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dolphin closes the research loop: LLM-generated ideas are implemented, run, and fed back, and one generated 3D classifier reaches 93.9% accuracy on ModelNet40.","keywords":["closed-loop auto-research","LLM-driven science","idea generation","experimental verification","results feedback","3D point cloud classification","ModelNet40","MLE-bench"],"falsifier":"Reproduce the three baselines using the original authors' code and training schedules on ModelNet40, CIFAR-100, and SST-2, and compare them with the paper's Baseline† numbers; then rerun the Dolphin-generated models under the same evaluation. If PointNet-CSR's 93.9% OA does not reproduce, or if the reimplemented baselines are measurably weaker than the originals, the central claim of closed-loop improvement is not supported.","tokens_in":20864,"feed_emoji":"🤖","tokens_out":8317,"duration_ms":70873,"temperature":0.7,"pith_summary":"Dolphin is a closed-loop framework in which a large language model generates research ideas, turns them into runnable code, executes the experiments, and feeds the measured results back into the next round of idea generation. The paper's central claim is that this cycle improves the input topic iteratively, and that it can produce methods comparable to human state of the art: on ModelNet40 the generated PointNet-CSR reaches 93.9% overall accuracy, slightly above GPSFormer's 93.8%. The framework also reports gains on CIFAR-100 and SST-2, and on three MLE-bench tasks it improves code templates from other pipelines. The point of the work is to replace novelty-only evaluation of AI-generated ideas with experimental verification and feedback, the loop that human researchers use.","feed_headline":"AI research loop hits 93.9 percent on ModelNet40","feed_subtitle":"An LLM-driven generate-test-feedback cycle invented PointNet-CSR, matching or beating human point-cloud methods.","key_machinery":"The load-bearing object is the three-stage research cycle: idea generation, experimental verification, and results feedback. Ideas are generated with reference to retrieved papers, which are filtered by an LLM-scored ranking that checks both topic relevance and task attributes such as model input and output; redundant ideas are removed by comparing summary embeddings against an idea bank, and novelty is screened by the LLM against the retrieved literature. Implementations are produced by modifying reference code, and failures are repaired by an exception-traceback-guided debugging step that extracts the function names, lines, and custom-code pieces from the traceback, asks the LLM to reconstruct the local code structure, and then plans the fix. After successful runs, results are classified as improvement, maintenance, or decline; declining or stagnant ideas are added to the idea bank to block similar proposals, while effective ideas are injected into the next idea-generation prompt. These two named mechanisms, task-attribute-guided paper ranking and exception-traceback-guided debugging, are what make the closed loop run at acceptable novelty and execution-success rates.","core_discovery":"The central claim is that an auto-research system can be closed: ideas are judged by whether they improve a reference implementation on a standard benchmark, and the outcome of each experiment changes what the next round of ideas is allowed to be. In two loops on ModelNet40, Dolphin generated 40 ideas, executed 21 that passed novelty and redundancy checks, and found 5 that improved the PointNet baseline; the best, PointNet-CSR, reaches 93.9% overall accuracy and 91.1% mean class accuracy, compared with 93.8% and 91.8% for GPSFormer under comparable settings (no pre-training, no voting). Across loops the average improvement over its own PointNet baseline is 1.0% OA; on CIFAR-100 the best idea adds 0.8% over WRN-28-10, and on SST-2 the best idea adds 1.5% over BERT-base. The paper also reports that 9 of 21 successfully executed ideas improved their baselines across three loops, with the improvement rate rising from 2/7 in loop 1 to 4/8 in loop 3.","pith_inferences":["A testable extension of the paper's own logic is to push the loop well past two iterations and measure when improvement plateaus or reverses; the feedback mechanism predicts continued gains until the idea space near the baseline is exhausted.","The 3D result suggests that a simple parameter-free contextual-semantic module can compete with learned graph convolutions; one could test whether that module transfers to other 3D backbones or to other modalities.","The paper's Limitations section concedes that only titles and abstracts are read and that project-level code is beyond current LLM coding ability; full-text grounding and hierarchical code-aware agents are natural next steps that the closed loop is compatible with.","Because the reported gains are computed against the paper's own reimplementations, a neutral reader should treat the improvement magnitudes as conditional on baseline fidelity until the baselines are matched against original published protocols."],"forward_implications":["Given a topic, a reference model, and a dataset, Dolphin can keep proposing validated ideas across loops, with the per-loop improvement rate rising from 2/7 to 4/8 in the 3D classification experiments.","A generated method can match or exceed a human-designed state of the art on a standard benchmark: PointNet-CSR scores 93.9% OA versus 93.8% for GPSFormer on ModelNet40 without pre-training or voting.","Task-attribute filtering lifts the share of novel ideas from 8/20 to 19/20, and traceback-guided debugging roughly doubles the execution-success rate from 33.3% to 50.0%.","On MLE-bench tasks, the loop improves code templates produced by other pipelines, e.g., raising an insult-detection score from 81.0 to 84.7 and a toxic-comment score from 94.7 to 97.2.","The per-idea cost is about $0.20 including retrieval, generation, and novelty checks, making the loop cheap enough to run many iterations."],"supporting_citations":[{"why":"PointNet is the reference baseline on ModelNet40 that Dolphin modifies and improves.","marker":"[Qi et al., 2017a]"},{"why":"GPSFormer is the human-designed state of the art that the Dolphin-generated PointNet-CSR is compared against on ModelNet40.","marker":"[Wang et al., 2024a]"},{"why":"ModelNet40 is the benchmark dataset where the strongest claim, 93.9% overall accuracy, is demonstrated.","marker":"[Wu et al., 2015]"},{"why":"CIFAR-100 is the benchmark for the 2D image classification experiments with WRN-28-10 as baseline.","marker":"[Krizhevsky et al., 2009]"},{"why":"SST-2 is the sentiment classification benchmark used to show the framework transfers across modalities.","marker":"[Socher et al., 2013]"},{"why":"BERT-base is the baseline model for the SST-2 experiments.","marker":"[Devlin et al., 2019]"},{"why":"Supplies the novelty-check and self-reflection practices Dolphin adapts, and represents the prior approach that lacked a feedback loop from experiments to ideas.","marker":"[Lu et al., 2024]"},{"why":"MLE-bench supplies the Kaggle-based tasks used to show integration with existing code-generation pipelines and version updates.","marker":"[Chan et al., 2024]"}],"fun_headline_variants":["Dolphin closed loop invents PointNet-CSR, lifts 3D accuracy to 93.9%","Auto-research agent Dolphin improves on PointNet, hits 93.9% accuracy","AI research loop: 21 tested ideas, 5 beat baseline, best hits 93.9%","Dolphin generates PointNet-CSR, matching state-of-the-art on ModelNet40"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the paper's reimplementations of PointNet, WRN-28-10, and BERT-base faithfully match the original methods; if those baselines are weaker or differently tuned, the reported improvements are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Dolphin closed loop invents PointNet-CSR, lifts 3D accuracy to 93.9%","Auto-research agent Dolphin improves on PointNet, hits 93.9% accuracy","AI research loop: 21 tested ideas, 5 beat baseline, best hits 93.9%","Dolphin generates PointNet-CSR, matching state-of-the-art on ModelNet40"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001565,"raw_usage":{"total_tokens":6270,"prompt_tokens":988,"completion_tokens":5282,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":5179}},"tokens_in":604,"tokens_out":5282,"duration_ms":38066,"temperature":1.0,"reasoning_tokens":5179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:43:33.890243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the three baselines using the original authors' code and training schedules on ModelNet40, CIFAR-100, and SST-2, and compare them with the paper's Baseline† numbers; then rerun the Dolphin-generated models under the same evaluation. If PointNet-CSR's 93.9% OA does not reproduce, or if the reimplemented baselines are measurably weaker than the originals, the central claim of closed-loop improvement is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CIFAR-100 is the benchmark for the 2D image classification experiments with WRN-28-10 as baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BERT-base is the baseline model for the SST-2 experiments."}],"review_version":1}