{"id":"c0aa93b5-ac21-4b60-8caa-4028df6ed6d2","arxiv_id":"2608.10333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A verifier-gated, multi-cycle adaptation loop improves a small agent model from execution traces, raising Qwen2.5-Coder-1.5B from 28.7% to 49.7% pass on held-out HumanEval+MBPP.","lead":"MERA is a protocol that uses traces from a live AI agent to repeatedly improve its small model, then routes work to the improved model behind a verification fallback. In code-generation benchmarks, four cycles raised a 1.5B model from 28.7% to 49.7% pass while keeping quality near the large-model level at about 61% of the cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 28.7→49.7 direct-pass gain may conflate model fine-tuning with the SkillBook prompt: Table 1's 'Direct SLM' note disables routing and fallback but not SkillBook, so base and trained rows are not matched.","rationale":"The reader's conditional verdict rests mainly on verifier coverage and replay fidelity, which are real but downstream. The most load-bearing issue for the paper's central claim is that its headline evidence does not isolate the model update. If the SkillBook prefix is present in final direct-SLM evaluation and absent for the base model, the 21-point gain could be substantially due to few-shot exemplars and formatting instructions rather than to the four-cycle update. The paper's own description makes this likely: the adapter is 'trained and queried with the SkillBook procedure prepended' (§3.3). The phrase 'direct-SLM evaluation' only disables routing and fallback (§4.1), not the SkillBook. This is not a question of consensus; it is an internal evaluation-matching gap. A simple two-row ablation would settle it, and the TAU-2 result does not rescue the code-gen claim because it deliberately removes SkillBook and is underpowered. I therefore keep the verdict conditional: the protocol is plausible and the TAU-2 ablation gives some evidence of model-level gain, but the headline code-generation number should be re-reported with and without the SkillBook prefix before the 'small-model capability' claim is accepted.","tokens_in":10567,"tokens_out":10459,"duration_ms":93279,"concrete_test":"Reproduce Table 1 with two extra rows on the same 582 held-out tasks and three seeds: (1) final SFT+GRPO adapter evaluated without the SkillBook prefix, or with an empty generic prefix; (2) base SLM evaluated with the final-cycle SkillBook prefix. If condition (1) falls toward 28.7% and condition (2) rises toward 49.7%, the reported gain is largely prompt-level skill support rather than model evolution. If (1) stays near 49.7% and (2) stays near 28.7%, the concern is refuted. Either result settles whether the direct-pass claim isolates the fine-tuned model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MERA improves the small model itself, with SkillBook as supporting deployment machinery. Table 1's headline readout, 'Direct SLM pass disables routing and fallback,' does not say SkillBook is disabled. §3.3 states the adapter is trained and queried with the SkillBook procedure prepended to its prompt, and §3.4 says each SkillBook entry includes static task-format instructions, bounded successful exemplars, and pitfalls. Therefore the 49.7% 'Multi-cycle SFT+GRPO' row likely evaluates the adapted adapter plus a few-shot skill prompt, while the 28.7% base row is the bare model. The comparison thus conflates prompt-level skill adaptation with weight-level model evolution. TAU-2 disables SkillBook and shows a pure adapter improvement, but it uses a different model and benchmark and is underpowered (p=0.17), so it does not establish the code-generation headline. The paper's own §7 limitations discuss verifier coverage and replay fidelity but not this evaluation mismatch; the admission gate also may have been evaluated on held-out replay, but the SkillBook confound alone is enough to make the headline 'small-model capability' claim unproven.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MERA, a multi-cycle adaptation protocol for deployed LLM agents. The method treats a single model invocation as the unit of adaptation: online traces are replayed to construct execution-verified teacher demonstrations, a SkillBook of reusable prompt templates is updated, a small-model LoRA adapter is fine-tuned with SFT or SFT+GRPO, and an input-only router is trained on executable labels; a joint replay gate admits only runtime states that preserve end-task quality with verifier-backed fallback. The main empirical claims are that four cycles raise Qwen2.5-Coder-1.5B direct pass from 28.7% to 44.2% (SFT) and 49.7% (SFT+GRPO) on 582 held-out HumanEval+MBPP tasks, that with verifier fallback the deployed policy retains 88.3% pass at 60.8% of always-Luna cost, and that a fine-tuned Qwen3.5-2B improves from 14/35 to 18/35 on TAU-2. The paper also gives a finance break-even planning calculation and an explicit limitations section.","tokens_in":10685,"tokens_out":5966,"duration_ms":50472,"significance":"If the small-model capability claim is established, the paper makes a useful systems contribution: it shifts the routing literature from fixed-student cost savings to verifier-backed multi-cycle model improvement, and it is careful to separate direct model quality from cascade quality. Strengths of the present version include the use of executable, benchmark-provided tests rather than self-generated tests; a held-out split with disjoint task identifiers; three training seeds with cached and matched teacher outputs; explicit reporting of the underpowered TAU-2 comparison; and public code and data. However, the central evidence for capability improvement is currently undermined by the SkillBook confound in the headline table, so the significance claim is conditional on the matched-control fix.","major_comments":[{"comment":"The headline direct-pass gain (Table 1: 28.7% → 44.2% / 49.7%) is not a matched comparison of the base model and the fine-tuned model. §3.3 states that 'The LLM adapter is trained and queried with the SkillBook procedure prepended to its prompt,' and §3.4 describes SkillBook entries that prepend static task-format instructions, bounded successful exemplars, and pitfalls to a new task. Table 1's note says only that Direct SLM pass disables routing and fallback; it does not say that SkillBook is disabled. Consequently the trained rows appear to measure the adapter plus a few-shot skill prompt, while the Base SLM row measures the bare model, so the 21-point claimed lift conflates weight-level model evolution with prompt-level skill adaptation. The paper should report a base row with the same SkillBook procedure prepended, and ideally an adapter-only row with SkillBook removed, under the same 1:10 cost accounting; alternatively it should explicitly state that SkillBook was disabled in the Direct-SLM evaluation and confirm this in the protocol description.","section":"§3.3, §3.4, Table 1"},{"comment":"The TAU-2 result cannot substitute for the missing matched code-generation control. Table 3 is a clean adapter-only comparison (SkillBook, routing, and fallback disabled), but it is explicitly underpowered: 14/35 → 18/35 with one-sided McNemar p=0.171875, a statement the text correctly repeats. With only 35 tasks and 10 discordant pairs, the four-task improvement is fully consistent with chance. The conclusion that 'MERA improves small-model capability rather than only routing around a fixed student' therefore rests entirely on the HumanEval+MBPP direct-pass numbers, which are subject to the SkillBook confound in the previous comment. Once the code-generation comparison is made matched, or an adapter-only row is added, the paper should state precisely which evidence supports the central claim.","section":"§4.4, Table 3"},{"comment":"The joint replay admission rule is applied to the held-out replay set, which creates a mild selection-on-evaluation risk for the cost-quality claims. §3.5 says an update is promoted only if replay preserves quality while reducing cost or fallback risk, and §6.2 reports 'the aggregate ratio over the held-out replay set.' The paper states that threshold calibration uses a disjoint shard and that policy results are reported on held-out task identifiers, but it does not state that the admission decisions themselves (which cycle to report, which artifacts to promote) were made without consulting the reported held-out set. If final-cycle selection used the same 582 tasks, the 88.3% pass / 60.8% cost operating point could be optimistically selected. Please specify that admission decisions were fixed on a separate validation split or describe the selection protocol.","section":"§3.5, §6.2"}],"minor_comments":[{"comment":"The 60.8% cost figure and the 88.3% pass point depend on the assumed small:large cost ratio of 1:10; please add a brief sensitivity check for this ratio or state more prominently that the headline cost number is conditional on a single assumed ratio.","section":"§4.1, Table 2"},{"comment":"The column header 'Train rows' is unclear; please rename it to 'Training set size' or similar, and state whether the 'Deploy cost ($/hr)' column uses the same serving setup as Table 2.","section":"Table 4"},{"comment":"The claim that matched GRPO exceeds SFT by 5.5–6.7 points rests on 95% paired-t intervals computed from three seeds; with n=3 the paired-t procedure is highly sensitive to distributional assumptions, so please report per-seed values or use a nonparametric summary.","section":"§4.2"},{"comment":"The phrase 'non-thinking templates' is undefined; please define what templates are used and why they matter for the TAU-2 comparison.","section":"§6.4"},{"comment":"The statement that RouteLLM-/FrugalGPT-style rows use 'matched adaptations rather than official checkpoints' should be expanded; as written it is unclear what was adapted and whether the comparison is apples-to-apples with the published methods.","section":"Table 2 caption"},{"comment":"The PDF text contains typesetting artifacts in the author list (for example, 'Y uhang Y ao'), which should be fixed in the final version.","section":"Title page"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually candid about its limitations, and the code/data release is a strength. My recommendation rests on the matched-control issue in the headline table; if the authors can provide a base-plus-SkillBook control and clarify the admission selection protocol, the paper can likely be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the invoke-level Skill→LLM→Router adaptation loop with verifier-gated replay admission is new relative to the cited routing and agent-optimization work, and the paper is unusually honest about its own weak spots: the weak router, the underpowered TAU-2 comparison, the replay-fidelity and verifier-coverage limits. Second, that honesty does not fully protect the headline “small-model capability” claim, because the main readout likely compares a bare base model against a fine-tuned model that also gets the SkillBook prompt. \n\nThe real contribution is the protocol: treating the invocation, not the task, as the unit of adaptation; scheduling SkillBook before adapter training and router fitting after; and admitting changes only through joint replay with verifier-backed fallback. That is a sensible systems design, and the paper separates direct-SLM pass from deployed cascade pass correctly. On the code-generation evidence, the matched SFT/SFT+GRPO split with cached teacher outputs and three seeds is solid work, and the GRPO-over-SFT gain being 5.5–6.7 points with CIs excluding zero is credible. The TAU-2 section is also handled well for a small-N agentic transfer check: seven wins, three losses, 25 ties, p=0.172 reported explicitly as feasibility, not as established transfer. I would not hold that against them. \n\nThe soft spot that matters is the SkillBook confound. Table 1’s “Direct SLM” note says routing and fallback are disabled, but Section 3.3 makes it clear the adapter is trained and queried with the SkillBook procedure prepended, and Section 3.4 says each SkillBook entry includes task-format instructions, successful exemplars, and pitfalls. That means the 49.7% row is probably evaluating the adapter plus a few-shot skill prompt, while the 28.7% base row is the bare model. The paper should have disabled SkillBook in the direct-SLM readout or used matched prompts for the base model. The one place that avoids this confound is TAU-2, which disables SkillBook, but it is underpowered. So the two workhorse results do not jointly establish the central claim: the strong code-gen result has the SkillBook confound, and the clean adapter result is underpowered. The finance table is illustrative and hand-set, so I would not put weight on it either way. \n\nThe paper deserves a serious referee: the protocol is novel enough, the code-gen measurement is clean in most respects, and the limitations section is unusually candid. The fix is straightforward—run a direct-SLM comparison with SkillBook disabled—and the paper would be much stronger for it. I am skeptical of the headline as written, but this is a fixable evaluation issue, not a broken method. I would engage with it and would bring it to a reading group, mainly for the admission-gate and dependency-scheduling design. Recommend: send to peer review with the SkillBook confound as the main requested revision.","headline":"A genuinely novel invoke-level adaptation loop with honest, mostly defensible code-generation evidence, but the headline 28.7→49.7 claim likely confounds weight evolution with the SkillBook prompt, so the central capability claim is not yet proven.","tokens_in":11361,"tokens_out":764,"would_cite":true,"duration_ms":8583,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retraining small coding agents beats routing: 28.7% to 49.7% pass","keywords":["MERA","model evolution","invocation-level adaptation","verifier-backed fallback","SkillBook","multi-cycle distillation","agent routing","GRPO"],"falsifier":"Run the admitted final-cycle state on a fresh task set whose ground truth includes hidden properties not covered by the provided executable tests. If verifier-passing student outputs fail those hidden properties at a materially higher rate after adaptation than before, the verifier has missed a semantic failure mode and the replay admission gate has overestimated the safety of the promoted state.","tokens_in":10233,"feed_emoji":"📈","tokens_out":8072,"duration_ms":62735,"temperature":0.7,"pith_summary":"MERA is an attempt to prove that the small model inside an agent system can be made more capable from its own execution traces, rather than merely being routed around. The paper proposes a closed loop in which failed invocations are replayed to produce execution-verified teacher demonstrations, recurring procedures are condensed into a SkillBook, and a small-model adapter is fine-tuned with SFT and optional GRPO; routing and verifier-backed fallback then decide which invocations can be served cheaply. The claim is that four such cycles raise a 1.5B coding model from 28.7% to 49.7% pass on held-out tasks, and that the deployed system preserves near-large-model quality at reduced cost. A sympathetic reader would care because the result would mean cost savings in agent deployment do not have to stop at the boundary of what the small model already knows.","feed_headline":"Retraining small coding agents beats routing: 28.7% to 49.7% pass","feed_subtitle":"Replay over verified traces plus verifier fallback holds 88.3% pass at 60.8% of full-large-model cost.","key_machinery":"The load-bearing mechanism is the verifier-backed joint replay gate operating on canonical step slices. Each invocation is canonicalized into a slice holding the prompt, local context, tool schema, output, verifier result, retry and fallback metadata, and skill assignment. Three tracks consume those slices: SkillBook aggregates stable prompt signatures into procedural prompt memory, the LLM adapter trains on hard examples with SFT or GRPO, and an input-only router learns cheap-versus-strong labels from executable outcomes. Updates run in Skill→LLM→Router order so the adapter sees the current SkillBook procedure and the router sees current-cycle student outcomes, and nothing is admitted unless replay shows the combined policy preserves verifier-passing quality. Routing is deliberately input-only and weak by design; safety comes from the verifier fallback path.","core_discovery":"On its own terms, the paper's central discovery is that treating a single model invocation, rather than a whole task, as the unit of adaptation turns ordinary runtime traces into reusable supervision. Failed invocations are replayed against a verifier to obtain teacher outputs; recurring local structure is promoted into SkillBook entries; and the student LoRA adapter is trained on hard slices, with the router fit last so its labels reflect the current-cycle student. Joint replay then admits a candidate runtime state only if end-to-end quality is preserved. The quantitative claim, averaged over three seeds on 582 held-out HumanEval+MBPP tasks, is that the base Qwen2.5-Coder-1.5B improves from 28.7% to 44.2% under SFT and to 49.7% with matched SFT+GRPO, and that with verifier-backed fallback the deployed policy keeps 88.3% pass at 60.8% of always-Luna cost. The paper is explicit that the learned pre-router is weak; most deployed quality preservation comes from verification and fallback, not from routing accuracy.","pith_inferences":["The admission gate's safety is bounded by verifier coverage, so the method would become stronger if verifiers extended beyond executable assertions to semantic or hidden-test checks; nothing in the loop prevents such an extension.","Because replay cannot capture the distribution shift caused by changing the runtime policy, the paper's own staging rule implies production deployments still need online canary validation; the replay gate is a necessary but not sufficient condition for safe promotion.","The Skill→LLM→Router schedule suggests cycle cadence and staleness are tunable: if traces are replayed with stale skills, the next cycle can absorb residual error, so one could test how much overlap across cycles is needed.","The same invocation-granularity loop might transfer to agent domains with cheap checkable verifiers, such as structured data extraction or API-call validation, though the paper only demonstrates code generation and one tool-use benchmark."],"forward_implications":["Cost savings from routing are no longer capped by the student's current ability, because the student itself improves each cycle from its own failed invocations.","Cheap execution can expand over time: the same verifier-gated replay that admits a skill or adapter also identifies new invocation slices that can be safely down-routed.","End-task quality can be held near the large-model level even when the learned router is inaccurate, as long as verification and fallback cover the failures.","On tool-use benchmarks such as TAU-2, an adapted 2B agent can reach the level of an unadapted 4B endpoint, suggesting adaptation can substitute for some model size.","Matched GRPO exceeds matched SFT by 5.5–6.7 points on the code-generation split, attributing part of the gain to reinforcement learning on top of distillation rather than to SFT alone."],"supporting_citations":[{"why":"Establishes the reasoning-and-acting agent trace structure that MERA treats as a stream of invocations for adaptation.","marker":"[Yao et al., 2023]"},{"why":"Supplies the FrugalGPT-style response cascade baseline that MERA compares against in the deployment table.","marker":"[Chen et al., 2024]"},{"why":"Supplies the RouteLLM-style learned routing baseline and the router paradigm MERA extends with verifier-backed fallback.","marker":"[Ong et al., 2024]"},{"why":"Shows step-by-step distillation into smaller models, the lineage of the paper's SFT teacher-demonstration track.","marker":"[Hsieh et al., 2023]"},{"why":"Provides iterative self-training from reflection, related hard-example adaptation that the MERA loop builds on.","marker":"[Yuan et al., 2025]"},{"why":"Supplies the skill knowledge base construction line that the SkillBook component instantiates.","marker":"[Wang et al., 2026]"},{"why":"Supports the interaction-trace RL training track analogous to MERA's optional GRPO update.","marker":"[Xi et al., 2026]"},{"why":"Supplies the TAU-2 tool-agent benchmark environment used for the controlled adapter comparison.","marker":"[Yao et al., 2025]"}],"fun_headline_variants":["Invocation-level adaptation lifts small coding agent from 28.7% to 49.7%","MERA turns failed calls into lessons: 1.5B model jumps to 49.7%","Per-call skill adaptation: 39.2% cost cut with 88.3% pass kept","Replay, distill, adapt: small coder improves by 21 points","Boost small models with SkillBook, not just routing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop hinges on the verifier catching every failure mode that matters and on replay of logged invocation slices predicting online behavior; if either fails, down-routing or skill promotion can be admitted while being unsafe in deployment.","fun_headline_variants_meta":{"raw":{"variants":["Invocation-level adaptation lifts small coding agent from 28.7% to 49.7%","MERA turns failed calls into lessons: 1.5B model jumps to 49.7%","Per-call skill adaptation: 39.2% cost cut with 88.3% pass kept","Replay, distill, adapt: small coder improves by 21 points","Boost small models with SkillBook, not just routing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000905,"raw_usage":{"total_tokens":3971,"prompt_tokens":1101,"completion_tokens":2870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":717,"completion_tokens_details":{"reasoning_tokens":2757}},"tokens_in":717,"tokens_out":2870,"duration_ms":16937,"temperature":1.0,"reasoning_tokens":2757,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:22:06.784064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the admitted final-cycle state on a fresh task set whose ground truth includes hidden properties not covered by the provided executable tests. If verifier-passing student outputs fail those hidden properties at a materially higher rate after adaptation than before, the verifier has missed a semantic failure mode and the replay admission gate has overestimated the safety of the promoted state.","supporting_citations":[{"cited_title":"React: Synergizing reasoning and acting in language models","cited_arxiv_id":null,"evidence_quote":"Establishes the reasoning-and-acting agent trace structure that MERA treats as a stream of invocations for adaptation."}],"review_version":1}