Pith. sign in

REVIEW 2 major objections 4 minor 38 references

Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Scout-first routing matches the strongest coding model at a fifth of the cost, and the handoff, not the router, carries the result.

desk verdict The engage-then-route architecture and the verification gate are genuinely new, but the headline cost claim is contradicted by Appendix N, and the handoff effect that carries the match is not statistically significant. read the letter →

arxiv 2608.04804 v1 pith:LVPSIN4B submitted 2026-08-05 cs.SE cs.AI

Ishaan Bhola , Adithyan Krishnan , Mukunda NS This is my paper
classification cs.SEcs.AI
keywords scroutingcost-awareLLMroutingSWE-benchProrepository-levelcoderepairlocalizationverify-then-striphandoffhidden-statefeaturescapped-budgetevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SuperScout tries to answer a deployment question: can a coding system keep a frontier model's solve rate while spending a small fraction of its cost? The paper's answer is to route after engaging with the repository, not from the issue text alone. A 7B searcher first scouts the codebase, produces a structured handoff whose reproduction claim is sandbox-verified and stripped if false, and its hidden states plus the task text then select one of four frontier fixers. On SWE-bench Pro's 266-task Python slice, the system solves 159 tasks versus 158 for the best solo fixer, at $0.230 per solve versus $1.274. The paper's own ablation attributes the result to the handoff rather than the router: always sending the cheapest fixer with the handoff ties the routed system, and the calibration evidence on 99 fresh tasks is directional only.

What carries the argument

The central machinery is the 'scrouting' pipeline: SuperScout-7B, a 7B searcher fine-tuned from Qwen2.5-Coder, explores the repository and emits a structured handoff; a verify-then-strip sandbox replays the handoff's reproduction claim and removes false claims; and a resume-based router, a per-fixer summary of solved and failed embedding centroids plus a base rate scored by a shared logistic regression, combines the task text and the searcher's pre-decode hidden state to pick one of four fixers in cheap-first order. The handoff is what the fixer consumes, the hidden state is what the router consumes, and the sandbox gate is what keeps either from being poisoned by the searcher's overclaimed reproductions. Routing walks the pool in cheap-first order, stopping at the first fixer whose predicted solve probability clears a caution threshold.

What would settle it

Run a fresh paired comparison of Kimi K2.5 solo versus Kimi K2.5 with the verified handoff on at least 200 tasks drawn from the same three repositories under the same capped-budget protocol; if the with-handoff advantage is absent or negative, the redistribution mechanism and the headline match at one-fifth cost collapse.

Watch

Extended reading notes

Core claim

The paper's central claim is that routing a repository-level coding task after a cheap search-and-verify pass can match the strongest available fixer at a fraction of its cost. On the full 266-task Python slice of SWE-bench Pro under the benchmark's official capped budget tier, the SuperScout pipeline solves 159 tasks versus 158 for the best solo fixer (Claude Opus 4.6), at $0.230 total cost per solve versus $1.274. The authors attribute this to the searcher's verified handoff, a structured note of implicated files, diagnostics, and a sandbox-replayed reproduction test with false claims stripped, rather than to the resume-based router: always sending the cheapest fixer with the handoff ties the routed system at 159 solves. A paired 99-task calibration study indicates the handoff redistributes rather than adds solving ability, lifting the three cheaper fixers while slightly hurting the strongest, with all per-fixer effects directional only; the searcher's hidden states improve cost routing on those labels, while the handoff's own text does not.

Load-bearing premise

The load-bearing premise is that the handoff genuinely lifts the cheaper fixers; the paired calibration on 99 fresh tasks found only directional gains (pooled +1.8 points, 95% CI [-1.0, +4.5], all per-fixer p>0.05), so if those lifts are noise the system's solve rate falls to Kimi K2.5's solo 149, which would not match Claude Opus 4.6's 158 under the paper's matched-rate criterion.

Editorial extensions

If this is right

  • A system that scouts the repository first can match the strongest model in a pool on SWE-bench Pro's Python census while spending roughly a fifth per solve ($0.230 versus $1.274).
  • The verified handoff, not the router, is what carries the matched rate: always sending the cheapest fixer with the handoff solves the same 159 tasks.
  • Adding a fixer to the pool costs only a resume built from 25-50 public outcomes, with no retraining of the searcher, embedder, or scorer.
  • The handoff appears to redistribute solving ability toward cheaper fixers and slightly away from the strongest one, a pattern that is directional at N=99.
  • Router features from the searcher's hidden states improve cost savings over task-text features, while feeding the handoff's own text to the router hurts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the handoff's redistribution pattern reproduces on other repositories, pairing a search-and-verify module with the cheapest strong open-weights fixer may become the default deployment, with model routing collapsed to a cost-allocation decision.
  • Inference: a pass-through arm that delivers unverified handoffs would test how much of the headline gain comes from the verify-then-strip gate; the paper leaves that as future work.
  • Inference: because the hidden-state routing features are tied to the exact searcher checkpoint, upgrading the searcher would require re-extracting states and refitting router heads, so the system's zero-retrain property applies to fixers only.
  • Inference: the solve-set nesting evidence across three benchmarks suggests accuracy routing has little headroom anywhere on current SWE evaluations, so cost routing is the only direction with room to grow.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces SuperScout, an engage-then-route pipeline for repository-level issue resolution. A 7B searcher (SuperScout-7B) first explores the repository and writes a structured handoff whose reproduction claims are sandbox-verified and stripped when false; a resume-based router then selects one of four frontier fixers using the issue text and the searcher's hidden states. On the full Python slice of SWE-bench Pro (266 tasks) under the official capped budget tier, the system is reported to solve 159 tasks at $0.230 per solve, matching solo Claude Opus 4.6 (158 solves, $1.274 per solve). A no-router ablation that always sends the handoff to the cheapest fixer ties the routed system, and the paper attributes the result to the handoff rather than to the routing decision. A paired calibration study (N=99) finds directional, non-significant handoff gains for the cheaper fixers and a loss for the strongest fixer, and reports that the searcher's hidden states improve cost routing while the handoff's text does not. The paper also contributes a zero-cost replay audit showing that solve sets are largely nested, motivating routing for cost rather than accuracy.

Significance. If the reported costs can be made internally consistent, the empirical finding is a useful existence proof: a cheap open-weights fixer plus a verified searcher handoff can match a frontier model on a hard benchmark at substantially lower cost. The paper is methodologically careful in several respects: the evaluation is a full census rather than a sample; protocol and cost-accounting choices are stated; the 105/266 byte-lossless JSON decode deviation is logged; the router is fit on fresh calibration labels rather than on the benchmark outcomes; and the authors openly disclose that the no-router ablation ties the routed system and that no handoff contrast reaches statistical significance. The release of model weights, router features, and code is a concrete reproducibility strength. The principal weakness is the cost-ledger inconsistency between Table IV and Appendix N, which directly undermines the headline cost ratio. In addition, the statistical fragility of the handoff lift means the 'match' claim is less robust than the abstract suggests.

major comments (2)
  1. [§6.3, Table IV, Appendix N (Table XVIII)] The all-in cost numbers in the main table are not reconcilable with the disclosed spend ledger. Table IV reports SuperScout at $0.137/task and $0.230/solve, which implies a total all-in system spend of about $36.4 over 266 tasks and 159 solves. Appendix N's evaluation-era ledger lists 'System arm' at $86.49, which under the same all-in convention implies $0.325/task and $0.544/solve. The no-router ablation row ($0.136/task, $0.227/solve) has the same problem, implying a total near $36.2 that cannot be matched to the $86.49 ledger line either. The text states that measured spend is disclosed in Appendix N, so the two sources should agree. As written, the abstract's central quantitative claim—'about a fifth of the total cost per solve'—is not supported; the ledger implies a ratio of about 0.43 rather than 0.18 against Claude Opus 4.6 solo. Please reconcile the ledger with Table IV, re-derive all cost per solve claims, and revise the abstract and conclusion accordingly.
  2. [§7.1, Table XIV, §8.2, §9] The headline solve-rate match is contingent on a handoff effect that is not statistically significant anywhere in the paper. Because the router sends 263 of 266 tasks to Kimi K2.5 (§7.2d), SuperScout's solve rate is effectively that of the no-router ablation, whose advantage over Kimi K2.5 solo (159 vs. 149) has p=0.245 on the 263-task subset (Table XIV). The paired calibration is also null: pooled +1.8 pp with 95% CI [-1.0, +4.5] and p=0.401 (§8.2, Appendix G). If the true handoff lift is zero, the system solves 149 rather than 159 and no longer matches Claude Opus 4.6's 158. The Limitations section acknowledges this, but the abstract and conclusion state the match without an uncertainty qualification. Please report a confidence interval for the system-level difference from the no-handoff Kimi baseline, and either soften the headline claims or justify explicitly why the census point estimate is sufficient despite the non-significant mechanism.
minor comments (4)
  1. [Appendix B, Table VII] The bracketing convention for solve counts at thresholds above the operating point is described only in the table note; please define it in the main text where the threshold sweep is discussed so that readers do not have to infer the convention from the note alone.
  2. [Figure 6] The histograms would be easier to read if the θ=0.30 threshold were marked directly on both panels and the bin width stated; the current caption describes the distributions but does not show how the gate is visualized.
  3. [§4.3, Table II] The feature list uses p for the base rate, which is easy to confuse with the predicted probability P(solve); consider renaming the base rate b in Table II and the surrounding text.
  4. [§5.3, Figure 5] The language-transfer comparison rests on very small cells for TypeScript (n=7 spontaneous handoffs) and C++ (n=8); the authors flag this, but Figure 5 would benefit from displaying the per-bar sample sizes directly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the headline arms are measured against an external benchmark with official scoring, the router is fit on fresh calibration labels, and the no-router ablation is a genuine control that undercuts the routing claim.

full rationale

The paper's derivation chain is not circular. The central comparison (SuperScout 159/266 at $0.230/solve versus Claude Opus 4.6 solo 158/266 at $1.274/solve) is a measured outcome on SWE-bench Pro's official harness, not an equation derived from a fitted parameter. The router's logistic head and threshold are calibrated on 99 fresh tasks drawn from 2026 repositories (Section 8.1), with the benchmark's 23 evaluation repositories blocklisted from the searcher's training data (Section 5.1), so the benchmark solve rates are not the labels that trained the router. The no-router ablation (always Kimi K2.5 with handoff, 159/266 at $0.227/solve) is an honest control; it shows the routing decision contributed nothing on this pool, which weakens the 'routing' claim but is the opposite of circularity. The paper explicitly flags that the handoff deltas are directional only (Section 8.2, pooled +1.8 pp, CI [-1.0, +4.5], p=0.401; Appendix I, p=0.245) and that the hidden-state head's gate effect is partly mechanical threshold shifting that was not separated by a matched shrinkage control (Section 7.2d). These are evidential weaknesses, not reductions to inputs. The one serious quantitative red flag is internal cost-accounting inconsistency: Section 6.3 defines Table IV's $0.230/solve as all-in, but Appendix N lists the 'System arm' at $86.49, implying about $0.544/solve; this is a correctness/consistency problem, not a circular step. No self-citation chain, imported uniqueness theorem, ansatz-by-citation, or renaming of a known result appears. Score 0 for circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five domain assumptions: the official harness is ground truth, list prices are stable, the calibration set represents the benchmark, the sandbox gate faithfully checks reproduction claims, and layer -4 hidden states carry task content. The fitted parameters are the routing threshold, the logistic head weights, and the searcher decoding temperature; all are disclosed and the threshold is robust across 0.25 to 0.30 in Appendix B. No invented entities are introduced beyond the trained components, which ship with weights.

free parameters (3)
  • Routing caution threshold theta = 0.30
    Recalibrated on the 99-task lab set using with-handoff outcomes via the matched-point rule (Section 6.2); Appendix B shows theta=0.25 gives the same 159 solves at $0.227, so the headline is insensitive across 0.25 to 0.30.
  • Router logistic head weights (task-text and hidden-state spaces) = not reported
    Fit on all calibration data per the Table II recipe; the C-vs-A cost saving holds in only three of five folds (Appendix F), so the fitted head is fragile.
  • Searcher decoding temperature = 0.9 (single draw)
    Chosen after the vault exam comparison in Section 5.2; it is a decoding hyperparameter, not fit to the SWE-bench Pro outcome, but it directly affects handoff quality and the headline result.
assumptions (5)
  • domain assumption The official SWE-bench Pro harness and capped budget tier are the ground-truth evaluator of correctness and cost.
    Section 6.1 states scoring uses the official harness and the gold-patch control passes 265 of 266 tasks; the single failure is an unwinnable dataset artifact kept in all denominators.
  • domain assumption Fixer API costs are list prices at measurement time and prices are treated as stable for the comparison.
    Section 9 states all costs are provider list prices recorded at measurement time, one fixer was served as a quantized build through a pinned provider, and cost conclusions are snapshots rather than stable constants.
  • domain assumption The 99-task fresh calibration set and the 266-task census are representative of each other and of deployment conditions.
    Section 8.1 uses 2026 repositories for fresh labels; Section 9 notes the benchmark spans three repositories, so effective diversity is closer to three codebases than to 266 independent draws.
  • domain assumption The verify-then-strip sandbox faithfully determines whether a reproduction claim genuinely fails against the unpatched repository.
    Section 4.2 describes the replay mechanism; Section 9 notes no pass-through arm was run, so the gate's contribution is inferred from the claim census rather than measured.
  • domain assumption The searcher's layer -4 pre-decode hidden state carries content informative for fixer success.
    Section 4.3 and Appendix F report hidden-state features at pooled AUC 0.600, fold-stable in four of five folds; Section 7.2d notes part of the gate effect is mechanical because the hidden-state head's support lies entirely above theta.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First." pith.science (2026). https://pith.science/paper/LVPSIN4B

@misc{pith2026260804804,
  author       = {Pith},
  title        = {Pith review of: Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LVPSIN4B}},
  note         = {Machine review of arXiv:2608.04804}
}
abstract

Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandbox-verified, with false claims stripped before delivery. The searcher's hidden states, together with the task text, then feed a resume-based router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark's official capped budget tier, SuperScout matches the best single model's solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result. A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at $N=99$ the per-fixer effects are directional only; the searcher's hidden states improve cost routing on the calibration labels while the handoff's own text does not. The searcher's compute adds less than half a cent of GPU time per task.

Figures

Figures reproduced from arXiv: 2608.04804 by the authors.

Figure 1
Figure 1. Cost per solve versus solve rate on SWE-bench Pro (Python-266). The dashed curve is the blind-mixing line (random cost-blind mix of Kimi K2.5 and Claude Opus 4.6). SuperScout matches Claude Opus 4.6 at about a fifth of its cost per solve, well above the line. The system point is all-in; solo points are fixer API only. The x-axis is logarithmic. SuperScout routes after scouting, a pattern we call scrouting. A 7B sear… view at source ↗
Figure 2
Figure 2. sketches the consequence: between any two models the accuracy prize is only a thin sliver, while the large shared region is where a cheaper model would have sufficed all along. Second, no learned router we tested exceeded the solve rate of always picking the strongest model; every observed gap fell within noise. Taken together, these results suggest a plain conclusion: routing for accuracy has little headroom on cur… view at source ↗
Figure 3
Figure 3. The SuperScout pipeline. SuperScout-7B explores the repository and emits a structured handoff, which a sandbox gate verifies before a résumé router selects one of four frontier fixers. The dashed path is the fallback: when no handoff is produced, the chosen fixer proceeds from the issue text alone. Adding a new fixer requires only a résumé, not retraining. TABLE II: The résumé router, end to end. Each fixer’s résumé… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Sampled decoding is what makes the searcher useful. Left: switching from greedy to temperature 0.9 raises the find rate from 0.110 to 0.306 (2.65×, n=426 of 450). Right: on a separate 100-task dial set, commitment rises 3.03× while per-handoff quality holds at 0.96×; o…
Figure 5
Figure 5. Figure 5: Localization transfers to languages SuperScout￾7B was never trained on. Per-language file-level F1 of spontaneous handoffs (searcher alone, single draw, SWE-bench Multilingual). Solid bars: three trained languages; hatched: six never-trained. Dashed lines mark pooled a…
Figure 7
Figure 7. Figure 7: Onboarding a fixer costs a résumé, not a training run. A short public outcome record averages into two centroids and a base rate, and the router reads that résumé at inference time. No component is retrained. support lies entirely above θ, part of the hidden-state head…
Figure 8
Figure 8. Figure 8: shows the paired comparison; exact rates and intervals appear in Appendix G. Confidence intervals here and in the appendix are 95% percentile intervals from a bootstrap resampling tasks [38]. The handoff lifted the three weaker fixers: Claude Opus 4.6 rose from 48.5% t…
Figure 9
Figure 9. Figure 9: Which router features buy cost savings. Held-out cost saving against an always-best-model anchor, all variants pinned to the same solve rate (.606). Adding SuperScout-7B’s hidden states (variant C, the routing blend) lifts savings from 30.5% to 34.3%. Variants with han…
Figure 10
Figure 10. Figure 10: One real handoff, exactly as the fixer received it. A spontaneous handoff from the SWE-bench Pro evaluation, reproduced verbatim after the verify-then-strip stage. This reproduction claim was genuine and forwarded intact; 174 of 249 claims were false and stripped. Lin…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 12 canonical work pages

  1. [1]

    SWE-bench: Can language mod- els resolve real-world GitHub issues?

    C. E. Jimenezet al., “SWE-bench: Can language mod- els resolve real-world GitHub issues?” inInternational Conference on Learning Representations (ICLR), 2024, arXiv:2310.06770

    arXiv 2024

  2. [2]

    RouteLLM: Learning to route LLMs with preference data,

    I. Onget al., “RouteLLM: Learning to route LLMs with preference data,” 2024, arXiv:2406.18665

    arXiv 2024

  3. [3]

    FrugalGPT: How to use large language models while reducing cost and improving performance,

    L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” 2023, arXiv:2305.05176

    arXiv 2023

  4. [4]

    SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks?

    X. Denget al., “SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks?” 2025, arXiv:2509.16941

    arXiv 2025

  5. [5]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yanget al., “SWE-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024, arXiv:2405.15793

    arXiv 2024

  6. [6]

    Agentless: Demystifying LLM-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,” 2024, arXiv:2407.01489

    arXiv 2024

  7. [7]

    SWE-smith: Scaling data for software engineering agents,

    J. Yanget al., “SWE-smith: Scaling data for software engineering agents,” 2025, arXiv:2504.21798

    arXiv 2025

  8. [8]

    Training software engineering agents and verifiers with SWE-Gym,

    J. Panet al., “Training software engineering agents and verifiers with SWE-Gym,” inInternational Conference on Machine Learning (ICML), 2025

    work page 2025

Show all 38 references
  1. [9]

    R2E-Gym: Procedural environments and hy- brid verifiers for scaling open-weights SWE agents,

    N. Jain, J. Singh, M. Shetty, L. Zheng, K. Sen, and I. Stoica, “R2E-Gym: Procedural environments and hy- brid verifiers for scaling open-weights SWE agents,” inConference on Language Modeling (COLM), 2025, arXiv:2504.07164

    2025 arXiv

  2. [10]

    SWE-Fixer: Training open-source LLMs for effective and efficient GitHub issue resolution,

    C. Xieet al., “SWE-Fixer: Training open-source LLMs for effective and efficient GitHub issue resolution,” 2025, arXiv:2501.05040

    2025 arXiv

  3. [11]

    LocAgent: Graph-guided LLM agents for code localization,

    Z. Chenet al., “LocAgent: Graph-guided LLM agents for code localization,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, 2025, pp. 8697–8727, arXiv:2...

    2025 arXiv

  4. [12]

    SweRank: Software issue localization with code ranking,

    R. G. Reddyet al., “SweRank: Software issue localization with code ranking,” 2025, arXiv:2505.07849

    2025 arXiv

  5. [13]

    Hybrid LLM: Cost-efficient and quality-aware query routing,

    D. Dinget al., “Hybrid LLM: Cost-efficient and quality-aware query routing,” inInternational Con- ference on Learning Representations (ICLR), 2024, arXiv:2404.14618

    2024 arXiv

  6. [14]

    Large language model routing with benchmark datasets,

    T. Shnitzeret al., “Large language model routing with benchmark datasets,” 2023, arXiv:2309.15789

    2023 arXiv

  7. [15]

    EmbedLLM: Learning compact represen- tations of large language models,

    R. Zhuang, T. Wu, Z. Wen, A. Li, J. Jiao, and K. Ram- chandran, “EmbedLLM: Learning compact represen- tations of large language models,” inInternational Conference on Learning Representations (ICLR), 2025, arXiv:2410.02223

    2025 arXiv

  8. [16]

    ICL-router: In-context learned model representations for LLM routing,

    C. Wanget al., “ICL-router: In-context learned model representations for LLM routing,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 39, 2026, pp. 33 413–33 421, arXiv:2510.09719

    2026

  9. [17]

    SWE-Router: Routing in multi-turn agen- tic software engineering tasks,

    S. Son, S. Yoon, J. Tang, S. Wang, L. Wolf, and I. Bogunovic, “SWE-Router: Routing in multi-turn agen- tic software engineering tasks,” 2026, presented at the 5th Deep Learning for Code Workshop, ICML 2026; arXiv:2607.00053

    2026 arXiv

  10. [18]

    CodeRescue: Budget-calibrated recovery routing for coding agents,

    Q. Heet al., “CodeRescue: Budget-calibrated recovery routing for coding agents,” 2026, arXiv:2607.19338

    2026 arXiv

  11. [19]

    TRACE-Router: Task-consistent and adaptive online routing for agentic AI,

    R. Raj, S. Kundu, S. Banerjee, D. Joshi, I. V ohra, and T. Krishna, “TRACE-Router: Task-consistent and adaptive online routing for agentic AI,” 2026, arXiv:2607.22465

    2026 arXiv

  12. [20]

    Toward training superintelligent soft- ware agents through self-play SWE-RL,

    Y . Weiet al., “Toward training superintelligent soft- ware agents through self-play SWE-RL,” inInterna- tional Conference on Machine Learning (ICML), 2026, arXiv:2512.18552

    2026 arXiv

  13. [21]

    Introducing SWE-bench Verified,

    OpenAI, “Introducing SWE-bench Verified,” https:// openai.com/index/introducing-swe-bench-verified/, 2024, accessed 2026-08-03

    2024

  14. [22]

    SWE-bench multilingual,

    J. Yanget al., “SWE-bench multilingual,” https://www. swebench.com/multilingual.html, 2025, dataset released alongside SWE-smith

    2025

  15. [23]

    Qwen2.5-Coder technical report,

    B. Huiet al., “Qwen2.5-Coder technical report,” 2024, arXiv:2409.12186

    2024 arXiv

  16. [24]

    Qwen3 Embedding: Advancing text embedding and reranking through foundation models,

    Y . Zhanget al., “Qwen3 Embedding: Advancing text embedding and reranking through foundation models,” 2025, arXiv:2506.05176

    2025 arXiv

  17. [25]

    gpt-oss-120b & gpt-oss-20b model card,

    OpenAI, “gpt-oss-120b & gpt-oss-20b model card,” 2025, arXiv:2508.10925

    2025 arXiv

  18. [26]

    Open-SWE-Traces: Advancing dual-mode mul- tilingual distillation for software engineering agents,

    W. U. Ahmad, N. Ludwig, S. Majumdar, and B. Gins- burg, “Open-SWE-Traces: Advancing dual-mode mul- tilingual distillation for software engineering agents,” 2026, arXiv:2606.16038; dataset: https://huggingface.co/ datasets/nvidia/Open-SWE-Traces

    2026

  19. [27]

    OpenHands Trajectories with Qwen3-Coder-480B-A35B-Instruct,

    M. Trofimova, A. Shevtsov, I. Badertdinov, K. Pyaev, S. Karasik, and A. Golubev, “OpenHands Trajectories with Qwen3-Coder-480B-A35B-Instruct,” Nebius blog / HuggingFace dataset, 2025, dataset: https://huggingface. co/datasets/nebius/SWE-rebench-openhands-trajectories

    2025

  20. [28]

    SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,

    I. Badertdinovet al., “SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,” inAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025

    2025

  21. [29]

    From SWE-ZERO to SWE-HERO: Execution-free to execution-based fine-tuning for software engineering agents,

    N. Ludwig, W. U. Ahmad, S. Majumdar, and B. Ginsburg, “From SWE-ZERO to SWE-HERO: Execution-free to execution-based fine-tuning for software engineering agents,” 2026, arXiv:2604.01496; dataset: https://huggingface.co/datasets/nvidia/ SWE-Hero-openhands-trajectories

    2026 arXiv

  22. [30]

    OpenHands: An open platform for AI software developers as generalist agents,

    X. Wanget al., “OpenHands: An open platform for AI software developers as generalist agents,” inInternational Conference on Learning Representations (ICLR), 2025, arXiv:2407.16741

    2025 arXiv

  23. [31]

    LoRA: Low-rank adaptation of large lan- guage models,

    E. J. Huet al., “LoRA: Low-rank adaptation of large lan- guage models,” inInternational Conference on Learning Representations (ICLR), 2022, arXiv:2106.09685

    2022 arXiv

  24. [32]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shaoet al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,” 2024, arXiv:2402.03300

    2024 arXiv

  25. [33]

    Introducing Claude Opus 4.6,

    Anthropic, “Introducing Claude Opus 4.6,” https://www. anthropic.com/news/claude-opus-4-6, 2026, accessed 2026-08-03

    2026

  26. [34]

    Introducing GPT-5.2,

    OpenAI, “Introducing GPT-5.2,” https://openai.com/index/ introducing-gpt-5-2/, 2025, accessed 2026-08-03

    2025

  27. [35]

    Kimi K2.5: Visual agentic intelligence,

    Kimi Team, “Kimi K2.5: Visual agentic intelligence,” 2026, arXiv:2602.02276

    2026 arXiv

  28. [36]

    Gemini 3 Flash model card,

    Google DeepMind, “Gemini 3 Flash model card,” https: //deepmind.google/models/model-cards/gemini-3-flash/, 2025, accessed 2026-08-03

    2025

  29. [37]

    Note on the sampling error of the dif- ference between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the dif- ference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  30. [38]

    Bootstrap methods: Another look at the jackknife,

    B. Efron, “Bootstrap methods: Another look at the jackknife,”The Annals of Statistics, vol. 7, no. 1, pp. 1–26, 1979. Appendix A The Prior Routing Audit The cost-routing premise of §3 rests on a zero-cost replay of published per-task outcome matrices. Table VI records the stru...

    1979

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.