Artificial Intelligence · arXiv:2610.21791 v2 · submitted 15 Oct 2026
RSIFlow: Adaptive Self-Improvement Workflows with an Evolving Meta-Agent
The paper says
“Large language model (LLM) agents have demonstrated strong capabilities in complex tasks, including code generation, tool use, and multi-hop question answering.”
more of the abstract
Existing self-improvement methods enhance agent performance through model optimization, harness evolution, and executable artifact refinement, while recursive self-improvement (RSI) further extends optimization to the improvement mechanisms themselves. However, adapting self-improvement workflows to evolving performance bottlenecks while improving workflow orchestration through execution feedback remains a key challenge. We propose \textbfRSIFlow, a \textbfRecursive \textbfSelf-\textbfImprovement framework that integrates adaptive work\textbfflow orchestration with meta-agent capability evolution. RSIFlow diagnoses performance bottlenecks from task-agent trajectories and environmental feedback to dynamically select component-specific improvement workflows. The meta-agent then distills workflow outcomes into component-specific skills and cross-component principles, evolving its own harness to improve subsequent workflow orchestration. Experiments across six benchmarks and two backbone models show that RSIFlow achieves the SOTA average performance on all benchmarks. Ablation studies further demonstrate the contributions of adaptive workflow orchestration and meta-harness evolution. Code is available at https://github.com/1HUAN1/RSI-Flow.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Careful evaluation hygiene and a genuinely new third intervention route, but the SOTA headline is hostage to the proprietary meta-backbone and collapses on HumanEval+ when the meta model is swapped. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at §4.4, Fig. 4, App. C.5
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The headline claims — 'achieves the SOTA average performance on all benchmarks' (Abstract) and the 6.06-point overall margin over SIA (Sec. 4.2) — are established only for the GPT-5.6-Sol meta-backbone, whose latent capabilities are never measured. Fig. 4 and App.
more of objection 1
C.5 show that, holding the Qwen3-4B task agent, the evolution rounds, budget, and reporting set fixed, varying only the meta-backbone moves HumanEval+ from 68.29 (GPT-5.6-Sol) to 53.60 (Claude-Sonnet-5) to 49.51 (DeepSeek-V4.1-Flash), an 18.78-point spread that exceeds both the 6.06-point SOTA margin and the 5.74% meta-evolution ablation gain (Sec. 4.3). With DeepSeek-V4.1-Flash the final HumanEval+ (49.51) falls below SIA (53.66) and HarnessForge (54.27), and with Claude-Sonnet-5 it approximately ties SIA (53.60 vs. 53.66). The authors acknowledge this sensitivity in Sec. 4.4 and in the Limitations, and App. A.3 explicitly declines to assess semantic paraphrases or pretraining contamination, but the sensitivity analysis covers only HumanEval+, so the claim of SOTA on all six benchmarks is unverified under any other meta-backbone, and the Abstract and Sec. 4.2 statements are not conditioned on the meta-backbone. This is an attribution gap rather than an internal inconsistency: the within-meta-backbone ablations in Tables 2 and 3 are properly controlled and support the mechanism's existence. The concrete test needed is to run the full 744-task reporting set with at least one additional meta-backbone (ideally open-weight) and multiple evolution seeds; if the SOTA edge does not persist, the claims must be explicitly conditioned on the meta-backbone.
The paper says, at §4.2, Table 1
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The main results rest on single evolution runs with no reported variance. The pipeline contains several stochastic components — the meta-agent's diagnosis and orchestration decisions, LoRA training, and skill retrieval — and the per-round trajectories in Table 9 show step-dependent reporting gains (R3 to R4 +1.65, R4 to R5 +4.10), while the skill records in App.
more of objection 2
F report substantial regressions on accepted updates (e.g., +34/−27 at R3, +23/−22 at R4). Because the headline margins over the strongest baseline are 6.06 and 3.86 points (Sec. 4.2), run-to-run variation of only a few points could alter the ranking in Table 1. Please report mean and standard deviation over at least three seeds of the evolution loop for the main table, or at minimum for the ablation table (Table 2), and state the decoding protocol used for the meta-agent's decision calls.
The paper says, at §4.4, Fig. 3, Apps. C.1–C.2
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.94):
“Equal per-task ex- ecution budgets likewise do not imply equal total evolution cost.”
Pith says: Objection 3 of 3 · major
The 'favorable performance–budget trade-off' claim (Sec. 4.4 and the Conclusion) is based on a rollout-count budget that excludes Meta-Agent API token consumption, as acknowledged in App. C.1 ('Meta calls... are excluded') and App. A.3 ('Equal per-task execution budgets... do not imply equal total evolution cost').
more of objection 3
The excluded cost is material: App. C.5 reports cumulative GPT-5.6-Sol meta tokens of 252.6M over five rounds, with round 5 alone consuming 160.8M, which is comparable to the 257.1M rollout tokens recorded for the entire SFT baseline (App. C.1). Under a total-cost accounting, the empirical Pareto-frontier conclusion in Fig. 3 is under-supported. Please include meta-agent costs in the budget definition, even approximately, or restate the claim as specific to task-agent rollout counts.
6 minor objections
The paper says, at Table 1
Pith says: minor objection 1 of 6
In the Qwen3-8B block, the HarnessForge row reads '38.1559.0745.83'; it should be '38.15 59.07 45.83', and the current formatting makes the ACEBench-Qwen3-8B result (HarnessForge 59.07 vs. RSIFlow 52.62) easy to misread.
The paper says, at Throughout
Pith says: minor objection 2 of 6
The system name is rendered inconsistently as RSIFlow, RSIFLOW, and RSIFLow across the abstract, main text, Table 1, and figures; please standardize the capitalization.
show 4 more
The paper says, at Fig. 5, Table 9
Pith says: minor objection 3 of 6
The case-study 'Ave score' appears to be the same six-benchmark Overall used in Table 1 (R5's 51.21 equals Table 1's Overall 51.21), but in that case R1's 32.23 is below the Vanilla three-domain average of 37.49 implied by Table 2, so the first accepted harness update — which gained on the training batch (106 to 148 successes out of 600 in App.
more of objection 3
F) — decreased reporting-set performance. Please clarify whether the case-study run is the Table 1 run and add a sentence discussing this early-round transfer gap.
The paper says, at App. A.5
Pith says: minor objection 4 of 6
The statement that the evaluation temperature 'does not specify decoding for all main experiments' is ambiguous; please specify the decoding settings (temperature, top-p, max tokens) for the main-table evaluations and for the GPT-5.6-Sol meta-agent calls, and pin the exact API model version/date used.
The paper says, at Eq. (20), App. D.1
Pith says: minor objection 5 of 6
The retrieval scoring design introduces fixed choices (error-token weight 3, context cap 24,000 characters) with no robustness check; because retrieval is described as advisory, a brief sensitivity statement would be sufficient.
The paper says, at App. A.1
Pith says: minor objection 6 of 6
The text describes the batches as 'task-disjoint' but then notes that different training tasks may share an environment; please qualify the phrase as task-identifier-disjoint so that the round-separation guarantee is not overstated.
What would settle it
Run the full Table 1 matrix (Qwen3-4B and Qwen3-8B task agents, five rounds, at most two attempts per round, fixed 744-task reporting set) twice more, substituting DeepSeek-V4.1-Flash and an open-weight model (e.g., Qwen3-32B) for GPT-5.6-Sol, and recompute Overall and the margin over SIA. If any non-flagship meta-backbone yields an Overall margin over SIA smaller than the Table 2 meta-evolution ablation gain (5.74%) or a negative margin, the headline advantage is dominated by meta-model choice rather than the described orchestration and skill-ledger mechanisms.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Plain take: the evaluation protocol is the strongest thing here, and the mechanism — routing among Model, Harness, and a genuinely new Artifacts route, plus an evidence-maintained skill ledger — is plausibly real. The headline SOTA claim, though, is hostage to the proprietary meta-backbone, and the paper's own control shows it. Fig. 4 holds the task agent, rounds, budget, and reporting set fixed and varies only the meta model: HumanEval+ lands at 68.29 with GPT-5.6-Sol, 53.60 with Claude-Sonnet-5, 49.51 with DeepSeek-V4.1-Flash. The DeepSeek result is below both SIA (53.66) and HarnessForge (54.27) on the flagship benchmark, and Claude ends up level with SIA (53.60 vs 53.66). The overall 4B margin over SIA is 6.06 points, about a third of the backbone-driven spread on a single benchmark. The authors acknowledge the sensitivity (Sec. 4.4, Limitations) and explicitly decline to check semantic-paraphrase overlap or pretraining contamination (App. A.3). That is an attribution gap, not an internal inconsistency: the abstract's 'SOTA average performance' is what needs conditioning, while the mechanism claim survives on the within-meta-backbone ablations, which are properly controlled. Credit where it's earned. The third Artifacts route is new relative to SIA's two-way harness/weight choice, and the skill library's Add/Supplement/Revise/Merge operations with provenance, applicability conditions, and contradicting evidence are a real step beyond Voyager-style skill accumulation. The evaluation hygiene is the best part of the paper: fixed 744-task reporting set, hashed overlap checks, no reporting-evaluator feedback anywhere in the loop, gain-based acceptance restricted to training batches. Table 3, the common-initialization meta comparison, is the strongest single piece of evidence for meta-harness evolution. Soft spots, in order of weight. First, single runs everywhere: Table 1 has no seeds or error bars, and a 6-point win needs more than one trajectory. Second, the meta-backbone attribution gap above. Third, the budget-performance Pareto analysis counts rollout tokens only; meta-agent tokens are reported separately in Fig. 4/9, so the headline efficiency claim partly hides the meta cost. Minor: the hand-set retrieval weight and the unpinned code repo are fixable, and the soft self-referentiality of the loop is mitigated by the reporting separation. This deserves a serious referee. The right outcome is heavy revision toward conditioning the SOTA claim on the meta-backbone and adding variance reporting, not a desk reject. I'd cite it for the Artifacts route and the evaluation protocol, and I'd bring it to a reading group.
The strongest objection
The central claim (Abstract, Sec. 4.2) is that the reported SOTA results demonstrate a genuine mechanism: bottleneck-aware workflow orchestration coupled with an evidence-maintained skill ledger. That attribution fails on the paper's own control. Fig. 4 and App. C.5 vary only the meta-backbone while holding the Qwen3-4B task agent, evolution rounds, budget, and reporting set fixed; HumanEval+ moves from 68.29 (GPT-5.6-Sol) to 53.60 (Claude-Sonnet-5) to 49.51 (DeepSeek-V4.1-Flash), an 18.78-point spread driven by a component whose latent capability is never measured. Two consequences follow. First, the headline Overall margin over the strongest baseline (6.06 points, 51.21 vs 45.15) is smaller than the backbone-driven spread on a single benchmark, so the SOTA claim is not separable from the specific proprietary model. Second, with DeepSeek-V4.1-Flash the final HumanEval+ (49.51) falls below both SIA (53.66) and HarnessForge (54.27), and with Claude-Sonnet-5 it merely ties SIA (53.60 vs 53.66); the advertised HumanEval+ advantage collapses outside the flagship meta-backbone. The authors acknowledge meta-backbone sensitivity (Sec. 4.4, Limitations) and explicitly decline to assess semantic-paraphrase overlap or pretraining contamination (App. A.3), but the main results control for neither. This is an attribution gap rather than an internal inconsistency: the within-meta-backbone ablations (Tables 2 and 3) are properly controlled and support the mechanism's existence. What remains unresolved is whether the mechanism, rather than GPT-5.6-Sol's latent diagnosis and harness-writing skill, explains the headline margin, and whether any benefit survives an ordinary meta-backbone.
The paper's strongest evidence is genuinely supporting: the common-initialization comparison (Table 3) holds the task agent and meta-backbone fixed and still shows Meta4 beating Meta0 by 20.56% on Code under the same HARNESS route, and the Table 2 ablations isolate adaptive orchestration (+4.13 to 8.81% over fixed workflows) and meta-harness evolution (5.74% loss when disabled). These within-paper controls are coherent, and the held-out 744-task protocol (fixed manifest, hashed overlap checks, no reporting-set feedback in the loop, App. A.3) is unusually careful. The load-bearing weakness is attribution of the headline SOTA statement. Fig. 4 is a within-paper experiment whose manipulation, meta-backbone choice, moves HumanEval+ by 18.78 points, exceeding both the mechanism-attributed ablation gain (5.74%) and the entire Overall lead over SIA (6.06 points at 4B). The same figure shows the flagship HumanEval+ advantage over SIA collapses with either alternative backbone (49.51 and 53.60 vs 53.66). Because Table 1 is reported only for GPT-5.6-Sol, a proprietary model whose training data and benchmark familiarity are unverifiable, and because the main configuration is a single run with decoding details left unspecified in App. A.5, the data cannot distinguish (i) a working mechanism that requires a frontier orchestrator, (ii) headline gains dominated by the meta-model's latent capability, or (iii) inflation from benchmark contamination. The authors' own limitations text concedes the first-order version of this. That prevents ACCEPT but does not warrant REJECT: the mechanism's existence is supported by the controlled ablations, and the required experiment (a second meta-backbone across the full matrix, plus seeds) is directly specified by the paper's own sensitivity design. The reader's weakest assumption identified the same issue; I sharpen it with the DeepSeek/Claude-versus-SIA comparisons and the margin-versus-ablation comparison. The verdict stays CONDITIONAL.
The referee's report
The paper's procedural design is a genuine strength: a fixed 744-task reporting set, hashed training/reporting overlap checks, no reporting-evaluator feedback inside the evolution loop, and a gain-based acceptance rule restricted to training batches (App. A.3) make the results unusually robust against standard leakage criticisms. The within-meta-backbone ablations (Tables 2 and 3), in particular the Meta0-versus-Meta4 comparison where the same HARNESS component yields a 20.56-point Code gap on a common task initialization, give real evidence that the skill-ledger/meta-harness mechanism matters beyond component selection, and Algorithm 1's inclusion of failed attempts in the experience record addresses the softer self-reference concern. The detailed appendices (algorithms, prompt templates, skill records) and the released code support reproducibility. If the results hold, RSIFlow is a meaningful advance for recursive self-improvement, demonstrating an evidence-maintained skill library coupled with bottleneck-aware orchestration across code, tool use, and search-QA.
Pith's reading of the paper (a machine's summary, not the authors')
This paper claims that the way a self-improving agent decides what to improve should itself improve. RSIFlow pairs a task agent with a meta-agent that diagnoses the task agent's current bottleneck from execution trajectories and feedback, then picks one of three workflows (retrain the model, repair the harness, or revise the output) and banks every round's outcome, including failures, into an evidence-maintained skill ledger that steers later rounds. Reported results: top average scores on all six benchmarks, leading in 11 of 12 benchmark-backbone settings, with ablations attributing the margin to both the adaptive selection and the evolving meta-harness. If the attribution holds, orchestration is a learnable capability rather than a fixed recipe, which would matter for any deployment that wants an agent to keep improving on its own.
What carries the argument. The load-bearing object is the meta-harness, the execution interface of the meta-agent, written as H_t^Meta = (Action, Skill, Planning). Because the meta-agent's base model parameters phi stay fixed, all meta-level evolution happens by editing this harness: its action interfaces, retrieval calls, instructions, and prompts are rewritten to consume an evidence-maintained library of skills and principles. Each skill is a structured record k = (Omega_k, a_k, rho_k, Gamma_k, r-hat_k, E_k), listing applicability conditions, update scope, conditional guidance, preservation requirements, expected behavioral response, and supporting or contradicting evidence, maintained through four operations (Add, Supplement, Revise, Merge). This design turns workflow orchestration from a fixed policy into an accumulating capability: the same loop that updates the task agent also improves the mechanism that decides the next update.
Core claim. The central claim is that dynamic workflow adaptation and meta-agent capability evolution are coupled problems and that solving them together is what makes recursive self-improvement work. Over five evolution rounds, a meta-agent with frozen model weights reads the task agent's trajectories and environmental feedback, diagnoses the current bottleneck, selects one of three component-specific workflows (Model update via SFT or RL, Harness repair, or Artifacts revision), and evaluates candidates on the same task batch, accepting the first valid candidate with a positive paired gain. Every attempt is distilled into component-specific skills and cross-component principles, stored as structured records with applicability conditions, preservation constraints, expected behavior, and supporting or contradicting evidence; the meta-agent's own harness is then rewritten to invoke this library, closing the loop. The paper reports state-of-the-art average performance on all six benchmarks, leading in 11 of 12 benchmark-backbone settings, with a five-round trajectory on Qwen3-4B that follows Harness, Harness, Model, Model, Artifacts and raises the average score from 32.23 percent to 51.21 percent.
The premise it rests on. The central claim stands or falls on attributing the score gains to the orchestration-and-ledger mechanism rather than to the fixed proprietary API model that runs the meta-agent — an attribution the paper does not control for, since its own backbone sensitivity shows HumanEval+ outcomes varying from 49.51 percent to 68.29 percent and evolution gains from 5.09 percent to 24.41 percent depending only on which meta model is used.
If the paper is right (Pith's inferences, not the authors')
No single update route is globally best: in the ablations each fixed workflow (Harness-only, Model-only, Artifacts-only) wins in one domain, while the adaptive policy beats each of them by 4.13 to 8.81 percentage points.
Meta-agent evolution contributes independently of orchestration: switching off skill distillation and meta-harness revision lowers the three-domain average by 5.74 percentage points.
Budget efficiency improves with informed selection: RSIFlow sits on or near the empirical Pareto frontier on all six benchmarks, achieving the highest observed score per rollout budget.
The loop closes without retraining the meta model: all meta-level gains come from editing the meta-harness components rather than updating its frozen parameters phi.
The pattern holds across backbone scales: Qwen3-4B and Qwen3-8B both reach the top average score, with the 4B backbone gaining 6.06 percentage points over the strongest evaluated baseline.
pith_inferences":[
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language model (LLM) agents have demonstrated strong capabilities in complex tasks, including code generation, tool use, and multi-hop question answering. Existing self-improvement methods enhance agent performance through model optimization, harness evolution, and executable artifact refinement, while recursive self-improvement (RSI) further extends optimization to the improvement mechanisms themselves. However, adapting self-improvement workflows to evolving performance bottlenecks while improving workflow orchestration through execution feedback remains a key challenge. We propose \textbfRSIFlow, a \textbfRecursive \textbfSelf-\textbfImprovement framework that integrates adaptive work\textbfflow orchestration with meta-agent capability evolution. RSIFlow diagnoses performance bottlenecks from task-agent trajectories and environmental feedback to dynamically select component-specific improvement workflows. The meta-agent then distills workflow outcomes into component-specific skills and cross-component principles, evolving its own harness to improve subsequent workflow orchestration. Experiments across six benchmarks and two backbone models show that RSIFlow achieves the SOTA average performance on all benchmarks. Ablation studies further demonstrate the contributions of adaptive workflow orchestration and meta-harness evolution. Code is available at https://github.com/1HUAN1/RSI-Flow.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 1.00
“Experiments across six benchmarks and two backbone models show that RSIFlow achieves the SOTA average performance on all benchmarks.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.09 · human participants 0.04 · controlled comparison 0.35 · randomized assignment 0.02 · states quantitative result 0.52 · compares with prior results 0.87 · mentions code or data availability 0.99
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language model (LLM) agents have demonstrated strong capabilities in complex tasks, including code generation, tool use, and multi-hop question answering. | background 1.00 | 0.05 | 0.01 | asserted 1.00 |
| Existing self-improvement methods enhance agent performance through model optimization, harness evolution, and executable artifact refinement, while recursive self-improvement (RSI) further extends optimization to the improvement mechanisms themselves. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| However, adapting self-improvement workflows to evolving performance bottlenecks while improving workflow orchestration through execution feedback remains a key challenge. | gap 1.00 | 0.05 | 0.03 | asserted 1.00 |
| We propose \textbf{RSIFlow}, a \textbf{R}ecursive \textbf{S}elf-\textbf{I}mprovement framework that integrates adaptive work\textbf{flow} orchestration with meta-agent capability evolution. | contribution 0.99 | 0.88 | 0.09 | asserted 0.99 |
| RSIFlow diagnoses performance bottlenecks from task-agent trajectories and environmental feedback to dynamically select component-specific improvement workflows. | method 0.96 | 0.76 | 0.05 | asserted 1.00 |
| The meta-agent then distills workflow outcomes into component-specific skills and cross-component principles, evolving its own harness to improve subsequent workflow orchestration. | method 0.97 | 0.76 | 0.08 | asserted 1.00 |
| Experiments across six benchmarks and two backbone models show that RSIFlow achieves the SOTA average performance on all benchmarks. | result 1.00 | 0.96 | 0.76 | asserted 1.00 |
| Ablation studies further demonstrate the contributions of adaptive workflow orchestration and meta-harness evolution. | result 0.92 | 0.95 | 0.07 | asserted 1.00 |
| Code is available at https://github.com/1HUAN1/RSI-Flow. | contribution 0.70 | 0.48 | 0.02 | asserted 0.94 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 6,907 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Figures are the authors', reproduced from the paper.
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment