Artificial Intelligence · arXiv:2610.20431 v3 · submitted 14 Oct 2026
SwarmRSI: Recursive Self-Improvement through Evolving Scientific Agent Swarm
The paper says
“Recursive self-improvement (RSI) in scientific research aims to improve agents' research capabilities through their own experimental experience. Scientific agent swarms can combine evidence and implementations from different research directions to guide their agents' subsequent experiments.”
more of the abstract
However, poor experimental results can leave unclear whether to revise a hypothesis or its implementation, while overall program scores can hide components useful to other agents. To address these challenges, we introduce SwarmRSI, a framework that evolves scientific agent swarms for recursive self-improvement. We devise Feedback-Guided Swarm Mutation to refine hypotheses and their implementations using experimental feedback from other agents. We also propose Complementary Swarm Recombination to reuse retained hypotheses and complementary program components across agents, allowing partial results to support new research. Recursive Swarm Evolution integrates both mechanisms into a feedback loop, using new experimental evidence to guide subsequent research. Experiments on OpenProblems, NatureBench and MLE-Bench Lite demonstrate SwarmRSI's state-of-the-art (SOTA) performance. Further evaluations show that accumulated research experience improves performance on held-out tasks. Biologists also reproduce its programs for follow-up biological analyses.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 15 Oct 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Solid integration of known agentic-science ideas with careful ablations; the cross-method SOTA margin is real but not yet cleanly attributed to the swarm mechanisms. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at §5.1, §B.4, Table 9
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The headline NatureBench margin over the matched-backbone baselines is not yet attributable to the swarm-reuse mechanisms, because the reruns are matched on wall-clock time, protocol, and hardware (§B.4) but not on search effort. Scored-evaluation counts are reported only for the two illustrated tasks in Table 9, where SWARMRSI uses 18 and 19 scored evaluations against Arbor's 3 and 2, and on 00639 AIDE reaches g=0.0552 with 8 evaluations against SWARMRSI's 0.0644 with 19.
more of objection 1
The token means in Figure 6 show that SWARMRSI does not win by spending more LLM tokens, which partially addresses the compute concern, but tokens do not bound the number of executed search attempts. Please report per-method scored-evaluation and LLM-call counts for the full 90-task comparisons, and, where feasible, add a control that caps SWARMRSI's scored evaluations per task at the baseline's count. The ablations (Tables 4 and 6) and the reuse-triggered new-best rates (Table 11) show that the mechanisms are not inert, so this is a request for attribution rather than a claim that they are ineffective.
The paper says, at §B.7, §5.1, Table 2
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The claim that the NatureBench gains over Arbor and AIDE are 'both significant' rests on a fragile margin. The 7.8pp Match-SOTA gain over Arbor corresponds to 7 discordant task pairs (7/0) with exact McNemar p=0.0156; because §B.7 discloses that no task crossed the g=0 threshold across seeds and the binary outcomes are seed-invariant, a single discordant pair moving to the other direction (6/1) gives p=0.125.
more of objection 2
The paper deserves credit for disclosing the discordant counts and the threshold invariance, but the prose in §5.1 overstates the robustness of a result that is one discordant pair away from non-significance and that is insensitive to the reported per-task g variation across seeds. Please hedge the significance claim or supplement it with a continuous-metric test on mean g over the 90 tasks.
The paper says, at §C.2, §F, Table 12
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.85):
“Cross-task ev- idence comes from six source tasks, 12 held-out tasks and one backbone, and generations are con- founded with the number of source tasks seen.”
Pith says: Objection 3 of 3 · major
The cross-task transfer experiment supports the claim that accumulated experience improves held-out performance, and the token reduction in Table 12 (12.17M to 10.94M tokens per task) shows that the gain is not bought with more held-out search budget. However, the §5.3 bullet 'Inherited research states transfer to new tasks' is mechanism-specific, and the authors correctly acknowledge (§F, sentence b569) that generations are confounded with the number of source tasks seen.
more of objection 3
What remains unresolved is the size of that confound: a control that runs the same six source tasks but discards or resets the exported state before the held-out search, or that varies the number of source tasks without inheritance, would separate graph-based state inheritance from mere accumulation of experience. Short of such a control, the conclusion should be scoped to 'experience accumulated over source tasks improves held-out performance', which is what the data directly establish.
4 minor objections
The paper says, at Figure 4
Pith says: minor objection 1 of 4
The caption says 'symmetric log axes', but the y-axis contains negative g values and spans zero; please specify the sign-preserving transformation used so that the reader can interpret the jumps near zero in panels (c) and (d).
The paper says, at Abstract, Table 3
Pith says: minor objection 2 of 4
The abstract sentence 'it wins medals on 81.8% of tasks' can be misread as state-of-the-art on MLE-Bench Lite, since the published GPT-5.5 Any-Medal rate of 86.4% exceeds it; the body (§5.1) correctly scopes the claim to hypothesis and program evolution agents with the same backbone, and the abstract should carry the same scoping.
show 2 more
The paper says, at §5.2
Pith says: minor objection 3 of 4
The sentence 'As shown in Table 6, all variants lower mean g...' refers to an appendix table before the reader reaches the main ablation table; adding the CI table to the main text or a forward reference would place the interval evidence next to the claim it supports.
The paper says, at §C.1, Figure 4
Pith says: minor objection 4 of 4
The trajectory comparison mixes recording conventions across methods (Arbor records evaluator calls, AIDE records candidate nodes, SWARMRSI records attempts), as explained in §C.1; please restate this counting convention in the Figure 4 caption, since the failure-recovery and hypothesis-improvement percentages quoted in the main text depend on these counts.
What would settle it
Extend the Section C.1 trajectory instrumentation from the two illustrative tasks (Table 9) and one seed to all 90 NatureBench tasks for SWARMRSI, Arbor, and AIDE under the same 4-hour budget, logging per-task scored-evaluation counts and LLM calls; then recompute Table 2 at matched attempt counts, e.g., compare best-so-far g against scored-evaluation index across tasks, or cap SWARMRSI's per-task attempts at Arbor's mean and re-run the McNemar test. If the 7-discordant-task margin (p=0.0156) survives attempt-matching, the mechanism attribution holds; if it falls to non-significance, the headline Match-SOTA advantage is a budget-deployment effect rather than evidence for swarm recombination.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 15 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The useful news first: this is a competent, honest integration paper, not a breakthrough. The three mechanisms — feedback-guided mutation, recombination, and the shared-graph loop — are established ideas (AIDE, Arbor, genetic crossover), and the paper says so plainly. What it adds is a careful empirical package: 118 tasks across three benchmarks, two backbones, matched-backbone reruns, three seeds, paired bootstrap CIs on every ablation, McNemar tests, a 644-attempt human failure annotation with kappa 0.92, and two biological follow-ups where experts reproduced the programs. The limitations section is unusually candid — it flags the rebase matching heuristic (Section F), the cross-task confound (b569), and the missing assessments of literature quality and peer review (b570).
What holds up: the within-harness ablations. Every removed mechanism drops mean g with bootstrap CIs excluding zero, and the trajectory analyses (reuse-triggered updates at 28.4% new-best vs 21.9%; 62% of proposed hypotheses improving vs 17% for Arbor) show the mechanisms are not inert. The central claim — cross-agent reuse of partial results helps — is defensible.
Where I get cautious is the headline NatureBench margin, and the caution is about attribution, not about the mechanisms. The reruns are matched on wall-clock time, not on scored evaluations or LLM calls; the paper reports token means (Figure 6) but never per-method evaluation counts for the Tables 2–3 comparisons, and its own Table 9 shows SWARMRSI running 18–19 scored evaluations against Arbor's 2–3 on the illustrated tasks. Tokens bound spend, not search attempts, so the 7.8pp SOTA edge over Arbor is not cleanly assigned to the swarm mechanisms. Second, Section B.7 reports per-task g varying across seeds while no task crosses a threshold, so the binary rates are seed-invariant: the McNemar test on the 7/0 discordant split is one comparison, and two flipped pairs would end its significance. The paper reports the seed-invariance fact but doesn't flag it as a power limitation. Third, the cross-task transfer gain (mean g 12.01 to 21.89) is confounded with the number of source tasks seen, which the authors do acknowledge.
The right fix is straightforward: release code and per-task score tables, report per-method scored-evaluation and LLM-call counts, and sensitivity-check the thresholded rates against the continuous g values. None of this sinks the paper; it just means the "SOTA / recursive self-improvement" phrasing outruns the evidence.
For anyone working on LLM agents for scientific coding, this is worth a serious referee. I would send it to review with conditional acceptance and make artifacts plus evaluation counts the gating requests.
The strongest objection
The central claim is that Feedback-Guided Swarm Mutation, Complementary Swarm Recombination, and the shared graph cause the reported gains. For this to hold, SWARMRSI and the matched-backbone reruns must be comparable in effective search effort, but they are matched only on wall-clock time and hardware (Section B.4: 4 hours per NatureBench task). The paper reports token means (Figure 6) but never per-method scored-evaluation or LLM-call counts for the Tables 2–3 comparisons; tokens bound LLM spend, not search attempts. Its own trajectory data show the gap can be large: on the two illustrated NatureBench tasks, SWARMRSI records 18 and 19 scored evaluations against Arbor's 3 and 2 (Table 9), and on 00639 AIDE approaches SWARMRSI's best g with 8 scored evaluations versus 19 (0.0552 vs 0.0644). The headline result is fragile to such imbalance: the statistically tested margin over Arbor is 7.8pp Match-SOTA, i.e., 7 discordant tasks of 90 (McNemar p=0.0156), and Section B.7 states that per-task g varied across seeds (ranges 0.0123–0.0781) without any threshold crossing, so the binary rates carry zero seed variance; shifting two discordant pairs to 5/2 would raise p to about 0.45. The reuse-triggered update rates (Table 11: 28.4% vs 21.9% new-best rates; 62% of hypotheses improving vs 17% for Arbor) and the within-harness ablations (Table 6, all bootstrap CIs excluding zero) are genuine partial support, so the mechanisms are not inert; what is unresolved is whether the cross-method SOTA margin reflects mechanism value or search-effort deployment. The cross-task transfer gain (Table 12, mean g 12.01 to 21.89) is additionally confounded with the number of source tasks seen, as the authors acknowledge (Section F, b569), and so cannot independently carry the recursive-improvement claim.
SwarmRSI is unusually careful: the trajectory arithmetic checks out (00639's mean g rises from 0.0575466 to 0.0644094 exactly as the per-instance changes imply; the reported McNemar p=0.015625 matches exact 7/0 discordance; Table 4 and Table 6 ablation drops are mutually consistent), the failure annotation uses two PhD annotators with Cohen's kappa 0.92, and biologist reproduction of two programs is independent, falsifiable support that should be credited. My single load-bearing concern is the causal attribution of the headline NatureBench result to the swarm mechanisms, because the matched-budget comparisons fix wall-clock time and hardware but leave scored-evaluation counts uncontrolled, and the paper's own Table 9 shows attempt-count ratios of 6–9x between methods under identical budgets. The fragility comes from the metric: with seed-invariant binary outcomes (Section B.7), the entire statistical case against Arbor is 7 discordant tasks, so a modest attempt imbalance could plausibly remove the margin. The reader's three candidate weak points overlap here: the search-effort attribution is the load-bearing one, the threshold coarseness is why the margin is thin, and the transfer confound (b569) is acknowledged by the authors and secondary because it bears only on the cross-task RSI narrative, not the in-benchmark comparisons. My recommended verdict is UNCHANGED: the reader's CONDITIONAL already names this condition, and the authors can resolve it with instrumentation they demonstrably possess, but until the attempt-matched analysis is reported the precise reading of 'SOTA' remains conditional.
The referee's report
If the results hold, this is a genuine advance in RSI for scientific agents: the decomposition of revision into hypothesis mutation versus program repair, the explicit reuse of partial results and retained hypotheses, and the cross-task inheritance results are substantive and go beyond single-agent refinement. The evaluation is well executed in several respects: results are graded by official benchmark scorers; the stronger baselines are rerun on the same backbone; per-seed OpenProblems tables are provided; the ablations use the identical wrapper with bootstrap CIs; the McNemar tests report the discordant counts; and the human failure study has high inter-annotator agreement. The within-harness ablations (Tables 4 and 6) and the reuse-triggered new-best rates (Table 11) provide genuine, non-circular evidence that the proposed mechanisms are active. The main unresolved issue is attribution: the headline NatureBench margin over Arbor and AIDE is matched on wall-clock time but not on search effort, and the significance of the margin is fragile under the disclosed seed-invariant binary metric.
Pith's reading of the paper (a machine's summary, not the authors')
This paper claims that recursive self-improvement in scientific research can be achieved by evolving the states agents inherit, namely hypotheses, programs, experimental evidence, and unresolved needs, rather than by improving any single agent's code or policy. When an experiment fails, the swarm's feedback decides whether the hypothesis or its implementation was at fault; when a program shows no overall gain, its components can still be recombined into another agent's work. On 118 tasks across OpenProblems, NatureBench, and MLE-Bench Lite, the design reports state-of-the-art results with two backbones, most notably a 44.4% Match-SOTA rate on all 90 NatureBench tasks, ahead of hypothesis- and program-evolution baselines run on the same model. A fair reader would care because the claim is that failed experiments and partial programs become reusable assets for later research, which is the core mechanism that recursive self-improvement requires.
What carries the argument. The central object is the inheritable research state, $s_t=(H_t,P_t,E_t,U_t)$, containing hypotheses, programs, experimental evidence, and unresolved needs, assembled from a shared graph $G_t$ that every agent reads and writes. The argument runs through three operators. Feedback-Guided Swarm Mutation distinguishes an H-mutation, which revises the hypothesis because per-instance results are mixed, from a P-mutation, which keeps the hypothesis and repairs the program because execution evidence points to an implementation barrier. Complementary Swarm Recombination defines rebase, which keeps a hypothesis and realizes it on another agent's valid program base selected by need-mechanism label overlap, and crossover, in which per-instance donor programs ranked by their largest advantage $\Delta_t(d)$ supply components where they beat the parent. Recursive Swarm Evolution selects the next operation with the discounted-UCB index $S_f=\hat{\mu}_f+\beta\sqrt{\log(1+N_H+N_P)/N_f}+\gamma\psi_f$, routes every outcome, including failures, back into the graph, and gates hypothesis credit through source and output audits. These operators do the load-bearing work: they convert ambiguous failure and hidden partial success into edits of what the next agent inherits.
Core claim. The paper's central claim is that a scientific agent swarm improves itself by updating a shared research graph in which each record holds a tested hypothesis, its parent and resulting programs, the outcome, and execution evidence, and by letting later agents assemble their next actions from that graph. Three mechanisms carry the argument: Feedback-Guided Swarm Mutation, which reads cross-instance performance differences as a signal to revise the hypothesis and execution failures as a signal to repair the program; Complementary Swarm Recombination, which rebases a retained hypothesis onto another agent's valid program or crosses over per-instance program components; and Recursive Swarm Evolution, which uses a discounted-UCB index to allocate effort between hypothesis and program frontiers and returns every outcome to the graph. The empirical claim is that this beats matched-backbone baselines: an OpenProblems average of 74.2 against TusoAI's 67.1 (76.2 with Gemini 3.1 Pro Preview), a NatureBench Match-SOTA rate of 44.4% against 36.6% for Arbor and 27.8% for AIDE, and an 81.8% Any-Medal rate on MLE-Bench Lite. In one seed over all 90 NatureBench tasks, about 77% of final best programs descend from a rebase, crossover, or feedback-triggered update, inherited states raise mean $g$ on 12 held-out tasks from 12.01 to 21.89, and biologists reproduce two of its programs for follow-up analyses of intestinal and splenic immunity.
The premise it rests on. The results are attributed to the swarm's mutation, recombination, and shared-graph mechanisms; the load-bearing assumption is that the matched-backbone baselines differ from SwarmRSI in those mechanisms rather than in the number of search attempts or LLM calls made under the shared time budget.
If the paper is right (Pith's inferences, not the authors')
Failed experiments become recoverable assets: 35% of failed implementations were later refined into new best valid programs, compared with 12% for AIDE and 5% for Arbor.
Hypothesis revision and program repair are complementary; removing either lowers NatureBench Match-SOTA by 10.0 or 13.3 percentage points, so a swarm needs both revision modes.
Inherited research states transfer to new tasks: after three generations on source tasks, mean $g$ on 12 held-out NatureBench tasks rises from 12.01 to 21.89 and Match-SOTA from 41.7% to 86.1%.
Most final solutions build on reuse: in one seed over all 90 NatureBench tasks, about 77% of final best programs descend from a rebase, crossover, or feedback-triggered update.
The gains are not tied to one model: results hold with DeepSeek-V4-Pro and Gemini 3.1 Pro Preview, and token costs stay below those of TusoAI and Arbor.
Not drawn in the paper: if the transfer result survives separating generation count from the number of source tasks seen (the paper acknowledges these are confounded), a swarm could bootstrap a new research benchmark with almost no cold-start cost; the clean test is to fix the source-task count and vary only the inherited state's content.
The failure-attribution scheme, which reads cross-instance score patterns to decide whether the hypothesis or the implementation was at fault, is a general credit-assignment idea that should transfer to non-scientific coding agents whenever per-test feedback exists, such as software-engineering benchmark suites.
The paper notes that per-task $g$ varied across seeds while no task crossed the Match-SOTA or Surpass-SOTA threshold, leaving the thresholded rates seed-invariant; comparing the full continuous-score distributions across seeds, task by task, would set a sharper test for the mechanism claims.
The paper acknowledges that rebase's base selection is a need-mechanism label-overlap heuristic; a likely stronger variant would learn from the graph's recorded outcomes which program bases best realize a retained hypothesis, instead of relying on labels.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Recursive self-improvement (RSI) in scientific research aims to improve agents' research capabilities through their own experimental experience. Scientific agent swarms can combine evidence and implementations from different research directions to guide their agents' subsequent experiments. However, poor experimental results can leave unclear whether to revise a hypothesis or its implementation, while overall program scores can hide components useful to other agents. To address these challenges, we introduce SwarmRSI, a framework that evolves scientific agent swarms for recursive self-improvement. We devise Feedback-Guided Swarm Mutation to refine hypotheses and their implementations using experimental feedback from other agents. We also propose Complementary Swarm Recombination to reuse retained hypotheses and complementary program components across agents, allowing partial results to support new research. Recursive Swarm Evolution integrates both mechanisms into a feedback loop, using new experimental evidence to guide subsequent research. Experiments on OpenProblems, NatureBench and MLE-Bench Lite demonstrate SwarmRSI's state-of-the-art (SOTA) performance. Further evaluations show that accumulated research experience improves performance on held-out tasks. Biologists also reproduce its programs for follow-up biological analyses.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-15
The authors' main claim, as the classifier picked it out 0.90
“Experiments on OpenProblems, NatureBench and MLE-Bench Lite demonstrate SwarmRSI's state-of-the-art (SOTA) performance.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.17 · human participants 0.14 · controlled comparison 0.17 · randomized assignment 0.01 · states quantitative result 0.06 · compares with prior results 0.74 · mentions code or data availability 0.10
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Recursive self-improvement (RSI) in scientific research aims to improve agents' research capabilities through their own experimental experience. | background 1.00 | 0.05 | 0.04 | asserted 0.99 |
| Scientific agent swarms can combine evidence and implementations from different research directions to guide their agents' subsequent experiments. | background 1.00 | 0.09 | 0.03 | asserted 0.91 |
| However, poor experimental results can leave unclear whether to revise a hypothesis or its implementation, while overall program scores can hide components useful to other agents. | gap 0.86 | 0.08 | 0.02 | asserted 0.82 |
| To address these challenges, we introduce SwarmRSI, a framework that evolves scientific agent swarms for recursive self-improvement. | contribution 1.00 | 0.81 | 0.09 | asserted 0.97 |
| We devise Feedback-Guided Swarm Mutation to refine hypotheses and their implementations using experimental feedback from other agents. | method 0.68 | 0.92 | 0.05 | asserted 0.98 |
| We also propose Complementary Swarm Recombination to reuse retained hypotheses and complementary program components across agents, allowing partial results to support new research. | method 0.63 | 0.85 | 0.07 | asserted 0.97 |
| Recursive Swarm Evolution integrates both mechanisms into a feedback loop, using new experimental evidence to guide subsequent research. | method 0.91 | 0.75 | 0.10 | asserted 0.98 |
| Experiments on OpenProblems, NatureBench and MLE-Bench Lite demonstrate SwarmRSI's state-of-the-art (SOTA) performance. | result 1.00 | 0.94 | 0.49 | asserted 1.00 |
| Further evaluations show that accumulated research experience improves performance on held-out tasks. | result 0.97 | 0.92 | 0.20 | asserted 1.00 |
| Biologists also reproduce its programs for follow-up biological analyses. | result 0.97 | 0.57 | 0.05 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.62
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 7,395 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (9)
Figures are the authors', reproduced from the paper.
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment