{"id":"b5794a5b-cc61-4448-9d80-0aa0a95616df","arxiv_id":"2605.03195","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Terminus-4B, a Qwen3-4B post-trained with SFT and rubric-based GRPO, matches or beats frontier models as a terminal-execution subagent while cutting main-agent tokens ~30% on SWE-Bench-style tasks.","lead":"A 4B-parameter model trained for terminal execution can stand in for frontier LLMs as a coding-agent subagent, cutting main-agent tokens by up to ~30% without hurting issue-resolution rates. That makes specialized small models a practical way to cut the cost of agentic coding.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Central claim of replacing frontier subagents rests on untested transfer from pre-provisioned SWE-Bench Docker/GitHub-issue regimes to real agentic terminal use.","rationale":"The reader correctly isolates the evaluation-scope assumption as the weakest link supporting the “replace frontier” claim; the multi-main-agent, No-Terminal, and LLM-judge ablations inside the paper are internally consistent and show clear gains over Vanilla-4B, so the result is real within its regime. No stronger internal contradiction (e.g., resolve-rate collapse, reward hacking that breaks end-to-end metrics, or prompt-only artifacts) appears in the tables or training plots. Because the paper already labels the gap and the reader already assigns CONDITIONAL, no verdict shift is warranted; the concrete external check above would simply confirm or falsify the remaining assumption.","tokens_in":20875,"tokens_out":551,"duration_ms":23232,"concrete_test":"Run the identical main-agent + Terminus-4B configuration (and the Opus/Sonnet subagent baselines) on Terminal-Bench or a held-out set of 50–100 real-world agent trajectories that include PowerShell/CMD, missing dependencies, and non-build tasks; recompute resolve rate, main-agent token delta, and Sub→Terminal rate. If any of those three metrics moves >10% relative to the SWE-Bench numbers (or falls below the frontier-subagent baseline), the transfer assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest empirical claim (resolve rates preserved on SWE-Bench Pro / internal C# while main-agent frontier tokens drop ~13–32% and Sub→Terminal distrust falls relative to Vanilla-4B) is measured exclusively inside Docker containers with pre-installed dependencies, Unix/Bash shells, and tasks mined from GitHub issues that are already build/test-heavy (Tables 3–6, §5.2, §6). Reference trajectories, SFT data, GRPO rollouts, and the 14-dimension rubric reward (§4.2) are all generated inside the same distribution. The paper’s own Limitations (§7) correctly flags that this does not cover messy real environments, non-Unix shells, deployment/infra tasks, or interactive debugging. If performance or the reliance metrics degrade outside that regime, the title-level claim that a 4B model can replace frontier LLMs at agentic execution does not hold; the benchmark results remain valid only as an existence proof inside a narrow, pre-sanitized setting.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces an Execution Subagent for coding agents and Terminus-4B, a Qwen3-4B model post-trained with SFT on telemetry trajectories plus GRPO with a rubric-based LLM-as-judge reward over structured Execution Plans. The central claim is that this 4B subagent can replace frontier models for agentic terminal execution: on SWE-Bench Pro and an internal SWE-Bench C# suite, main-agent (frontier) token use falls by up to ~30% versus a No-Subagent baseline with resolve rates essentially unchanged, while behavioral metrics (main-agent Terminal calls, Subagent→Terminal / Sub→Sub distrust, final-answer rate, and a 5-dimension LLM judge) improve over Vanilla-4B and approach or sometimes match Claude Sonnet/Opus and GPT-5.3-Codex as subagents. Supporting contributions include a decoupled subagent rollout framework (pass-through 4B main agent, fixed query, Dockerized repo state) and multi-dimensional reward design (execution quality, failure modes, final-answer quality).","tokens_in":21134,"tokens_out":1264,"duration_ms":24314,"significance":"If the reported results hold under standard scrutiny, the work is a concrete, practical demonstration that a narrowly scoped agentic skill—terminal execution and summarization—can be offloaded to a cheap SLM without harming end-to-end resolve rate on hard coding benchmarks. The decoupled rollout design and Execution-Plan intermediate representation are reusable for other subagent types and make RL training cost-effective. Strengths include systematic ablations (Vanilla / SFT / RL; multiple main agents; Subagent+Terminal vs Subagent-only), alignment between distrust metrics and the LLM-judge scores, and an honest Limitations section. This is useful systems evidence for the broader thesis that SLMs can carry repetitive agentic subtasks.","major_comments":[{"comment":"Tables 3–6 and §6.2 assert “no impact” on resolve rate and that Terminus-4B “matches or exceeds” frontier subagents, but no confidence intervals, bootstrap estimates, or multi-seed variance are reported. SWE-Bench C# has only 150 instances; swings of 1–3 points (e.g., Table 3: 30.0% No Subagent vs 31.5% Terminus vs 32.6% Sonnet; Table 4 Codex column) are within plausible sampling noise. Please add uncertainty estimates (or at least multiple evaluation seeds) and state statistical tests for the “no impact / matches” claims so that token savings are not over-interpreted relative to resolve-rate noise.","section":null},{"comment":"The abstract and §1 claim that Terminus-4B “often even exceeds” frontier subagent performance. On resolve rate this is sometimes true within noise, and on main-agent token reduction Terminus often beats Opus/Sonnet-as-subagent (Tables 3, 5). On the load-bearing reliance metrics, however, Terminus still trails frontier subagents: Subagent→Terminal is 0.14 vs 0.04–0.06 (Table 3) and 0.17–0.23 vs 0.06–0.13 (Table 5); only in the No-Terminal ablation does Sub→Sub match Opus (Table 6). Please qualify “exceeds” by metric (token cost vs. distrust vs. judge score) rather than as a blanket claim.","section":null},{"comment":"Half of the end-to-end evidence is the internal SWE-Bench C# suite (§5.2.1, Tables 4–6). Without a public instance list, harness, or release plan, those results are not independently verifiable. Either release the benchmark (or a documented subset) or move primary quantitative claims to SWE-Bench Pro and treat C# as supplementary, with enough protocol detail for replication of the collection pipeline.","section":null}],"minor_comments":[{"comment":"§4.2.3 / Eq. (2): α=0.5 and the hard penalties (−100 / −50) are free parameters; a short sensitivity note (or appendix) would strengthen the reward-design contribution.","section":null},{"comment":"Figure 6: y-axis “Mean Reward” scale and absolute units are hard to interpret relative to the 14 rubric dimensions (0–1 averages blended in Eq. 2). Clarify the reward scale after penalties and averaging.","section":null},{"comment":"Typos / wording: “suality” → “quality” (contributions list); “MOTIV ATING” spacing; “rrollouts” / “perfom” / “hte” / “subseqeunt” / “subagnet”; “Terminal-4B” once in §2.2 should be Terminus-4B; TerminalBench citation is “[?]”.","section":null},{"comment":"§5.2.2 lists “Claude Sonnet 4.6” as a main agent while Table 4/5 headers say “Claude Sonnet 4.5”—align version labels throughout.","section":null},{"comment":"§7 already flags Unix/Bash and Docker/pre-provisioned scope; a single sentence in the abstract or introduction that the ~30% claim is measured in that regime would better match the title’s generality.","section":null}],"recommendation":"minor_revision","confidential_remarks":"Solid empirical systems paper; suitable for a methods/systems venue. The skeptic concern about transfer beyond SWE-Bench Docker is real but already owned in §7 and does not invalidate the benchmark claims. Main fixable gaps are statistical uncertainty, tighter wording of “exceeds,” and reproducibility of the internal C# suite. I would not require new real-world deployments for acceptance if those three are addressed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean agent-systems paper with a usable result. They wire an Execution Subagent into a coding agent, post-train Qwen3-4B with SFT then GRPO against a 14-dimension rubric LLM judge, and show that Terminus-4B holds resolve rate on SWE-Bench Pro and their internal C# suite while cutting main-agent frontier tokens by roughly 13–32% and dropping main-agent terminal calls and Subagent→Terminal re-runs relative to Vanilla-4B. The No-Terminal ablation is the right stress test: when the main agent cannot compensate, Terminus still matches Opus-level Sub→Sub distrust and LLM-judge scores. That is the real contribution—not “subagents exist,” but a concrete, measured recipe for replacing a frontier terminal subagent with a 4B model inside the same loop.\n\nWhat is new is the combination: terminal execution as a delegated subagent with a strict <final_answer> contract, a decoupled rollout setup that freezes the main agent to a one-turn pass-through so GRPO only trains the subagent, and the Execution Plan intermediate so the judge scores plans rather than raw logs. Related work on Claude-style subagents, TerminalBench, and terminal RL is cited fairly; they are not inventing the category, they are specializing it and measuring cost.\n\nSoft spots are real but proportionate. Everything lives in pre-provisioned Docker with Unix/Bash and GitHub-issue build/test tasks; the Limitations section already says so. Reward and references come from frontier judges, so the policy is partly “imitate good frontier subagent plans,” though end-to-end resolve rate is external. No public model, code, or data; the C# bench and telemetry are internal; resolve rates lack uncertainty. Those are standard for this venue class, not load-bearing math errors. Free parameters (α, clip, turn limit) are stated; nothing looks cooked.\n\nThis is for people building coding agents who care about token cost and context hygiene. Worth a serious referee. I would engage: read the tables, try the recipe if artifacts appear, and treat the title claim as “works in the SWE-Bench Docker regime” until someone tests messy shells and infra tasks.","headline":"Solid systems result: a 4B terminal subagent can cut main-agent frontier tokens ~30% on SWE-Bench-style tasks without hurting resolve rate, with clean ablations; transfer beyond pre-provisioned Docker is the real open question.","tokens_in":21801,"tokens_out":579,"would_cite":true,"duration_ms":5894,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A 4B model trained for terminal execution can replace frontier LLMs as coding-agent subagents and cut main-agent tokens by ~30%.","keywords":["coding agents","subagents","terminal execution","small language models","SFT","GRPO","LLM-as-judge reward","SWE-Bench"],"falsifier":"Run the same main-agent configurations on a held-out suite of non-Unix, multi-shell, or infrastructure-heavy tasks outside the SWE-Bench Docker regime; if Terminus-4B’s resolve rate or main-agent token savings collapse relative to a frontier subagent, the replacement claim fails outside the benchmark setting.","tokens_in":21734,"feed_emoji":"⚙️","tokens_out":898,"duration_ms":7971,"temperature":0.7,"pith_summary":"Coding agents waste huge amounts of context on raw build and test logs. This paper argues that terminal work is a natural job for a specialized subagent, and that the subagent need not be a frontier model. The authors post-train a 4B Qwen model with supervised trajectories and reinforcement learning scored by a multi-dimensional rubric judge, producing Terminus-4B. On SWE-Bench Pro and an internal C# suite, plugging Terminus-4B into the execution subagent keeps resolve rates level with the no-subagent baseline while cutting the main agent’s token spend by up to about 30 percent, and often beats Sonnet, Opus, and GPT-5.3-Codex on how much the main agent actually trusts and reuses the subagent’s summaries. The practical claim is that narrow, expensive agent loops can be offloaded to small specialized models without sacrificing end-to-end success.","feed_headline":"4B model cuts coding-agent tokens ~30% as terminal subagent","feed_subtitle":"Terminus-4B matches frontier models on execution tasks without hurting resolve rates","key_machinery":"Execution Subagent plus Terminus-4B: a single-tool, turn-bounded agentic loop that runs terminal commands and returns structured <final_answer> summaries, trained by a decoupled rollout framework whose reward compares condensed Execution Plans against frontier reference trajectories along execution-quality, failure-mode, and final-answer dimensions.","core_discovery":"A 4B model post-trained with SFT plus GRPO under a rubric-based LLM-as-judge reward can serve as the Execution Subagent of a coding agent and match or exceed frontier models on that role, cutting main-agent token usage by up to ~30% with no loss of resolve rate on SWE-Bench Pro and SWE-Bench C#.","pith_inferences":["If the pattern generalizes, cost curves for long-horizon coding agents may be dominated by small specialized models rather than frontier calls for every tool loop.","Rubric-based LLM judges over condensed Execution Plans may be a reusable template for any agentic task that lacks a single binary success signal.","The evaluation gap the authors flag (Unix Docker SWE-Bench vs real multi-shell work) is the natural next stress test before production claims harden."],"forward_implications":["Coding agents can keep the same resolve rates while routing the bulk of terminal tokens through a 4B model instead of a frontier model.","Main agents can be instructed to prefer the Execution Subagent over direct Terminal, reducing context bloat from build and test logs.","The same SFT-then-rubric-RL recipe can be reused to train other narrow subagents (search, debug) without full main-agent rollouts.","Behavioral metrics such as Subagent→Terminal and final-answer rate become practical monitors of whether a small subagent is actually trusted."],"fun_headline_variants":["4B model matches frontier LLMs as terminal subagent, cuts tokens ~30%","Terminus-4B equals Claude GPT on agentic execution, saves ~30% tokens","Finetuned 4B replaces frontier models for coding-agent terminal tasks","Post-trained Qwen-4B cuts main-agent tokens ~30% as execution subagent","4B subagent matches frontier performance with no resolve-rate loss"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That success measured on SWE-Bench-style GitHub issues inside pre-provisioned Docker containers with Unix shells and frontier-judge rewards will hold for real messy agent use across other shells, platforms, and infrastructure tasks.","fun_headline_variants_meta":{"raw":{"variants":["4B model matches frontier LLMs as terminal subagent, cuts tokens ~30%","Terminus-4B equals Claude GPT on agentic execution, saves ~30% tokens","Finetuned 4B replaces frontier models for coding-agent terminal tasks","Post-trained Qwen-4B cuts main-agent tokens ~30% as execution subagent","4B subagent matches frontier performance with no resolve-rate loss"]},"model":"grok-4.5","effort":"low","cost_usd":0.004324,"raw_usage":{"total_tokens":1322,"prompt_tokens":854,"num_sources_used":0,"completion_tokens":92,"cost_in_usd_ticks":43240000,"prompt_tokens_details":{"text_tokens":854,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":376,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":854,"tokens_out":92,"duration_ms":3340,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T17:41:32.698780+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same main-agent configurations on a held-out suite of non-Unix, multi-shell, or infrastructure-heavy tasks outside the SWE-Bench Docker regime; if Terminus-4B’s resolve rate or main-agent token savings collapse relative to a frontier subagent, the replacement claim fails outside the benchmark setting.","supporting_citations":[],"review_version":3}