Pith.

Artificial Intelligence · arXiv:2611.16485 v1 · submitted 12 Nov 2026 · 50 references

Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments

Nguyen Minh Thang

The paper says

“A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090.”

more of the abstract

Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Thorough, transparent, and well-executed evaluation kit for coding agents, mostly reporting null results with honest limitations. This line is wrong

2 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 50 references say.

The paper says, at §10, Table 29 and Table 30

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The screen set S (48 tasks) is described as 'not fresh' (Section 13): 32 of its 48 tasks had been run in earlier blocks. Because the screen selects candidates from several variants, and prior exposure of those tasks could have informed the variant design (even if the paper states that variants were built from general ideas, not per-task results), the possibility of information leakage into the variant selection should be discussed more explicitly.

more of objection 1

The paper acknowledges the non-freshness in the limitations, but given that the screens are the sole mechanism for candidate selection, the degree to which previous runs on those tasks could have influenced the builds (e.g., through debugging or tuning) should be quantified or argued. Otherwise, the screening procedure's theoretical protection against selection bias is weakened.

The paper says, at §9.1 (Table 18) and §9.9

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.71):

“It cannot show that a specific difference in the prompts has no effect.”

Pith says: Objection 2 of 2 · major

The case study's intervals (98.75% simultaneous intervals) leave room for drops of up to 5.1 points (arm H) and gains of up to 3.5 points (arm T). The paper correctly calls these 'null results, not evidence of no effect.' However, the abstract and introduction state the result as 'no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4).'

more of objection 2

While technically accurate, a reader could misinterpret this as evidence of equivalence. The distinction between 'not detected' and 'no effect' is carefully made elsewhere, but the framing in the abstract could be tightened to avoid over-interpretation by readers who skip the detailed discussion.

4 minor objections

The paper says, at §2.5

Pith says: minor objection 1 of 4

The term 'points' for percentage points is defined, but the first occurrence in the abstract uses it without definition. Consider adding a brief clarification in the abstract or early in Section 1.

The paper says, at Table 5 (Section 6)

Pith says: minor objection 2 of 4

The simulated pass rates for the 'no effect, exchangeable tasks' scenario show a family-wise false-claim rate of 0.23–1.90% for the small panel and 0.00–0.50% for the large panel. This is reported without an explicit justification of why the simulation uses a 0.025 threshold rather than the standard 0.05; the 2.5% level is mentioned later in Section 9.1.

more of objection 2

The connection would be clearer if the simulation description referenced the rule's requirement (p ≤ 0.025) and explained that the simulated size is the fraction of runs where p ≤ 0.025 under the null.

show 2 more

The paper says, at Fig. 2 and Table 1

Pith says: minor objection 3 of 4

The caption of Fig. 2 mentions 'disturbed measurement' for two cells, but the corresponding entry in Table 1 (8 streams, 8-192 context) is flagged with a short note. Consider adding a footnote to Fig. 2 that explains the KV-cache eviction and the unexplained cell, referencing the two sentences in the text.

The paper says, at Appendix B (Table 34)

Pith says: minor objection 4 of 4

The recomputation of the case study intervals with the released kit shows that the upper end of the H arm's interval moves from +0.38 to +0.25 percentage points depending on task order. The paper notes this and calls the intervals consistent.

more of objection 4

However, the test decision (rejection or not) does not change, but the interval width fluctuation of ~0.13 points could be larger than the precision implied in the main report (which gives 95% intervals to 0.01 points). For clarity, state that the reported intervals are based on the registered panel order and that small Monte Carlo variations (up to 0.13 points) are typical.

What would settle it

To further validate the workstation–scorer similarity, run a larger parity experiment: 100 tasks from the validated pool, 4 repeats each, on both the workstation (RTX 5090, 8 workers) and a 4×L4 notebook (serial). Compute the per-task difference in solve rates. If the 95% interval of the mean difference exceeds ±2 percentage points or if the cross-machine correlation of per-task solved counts is below 0.85, the assumption of interchangeability weakens. This test would quantify the residual environment effect that the 24-task check could not resolve.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 24 s · 45,690 tokens read, 2,717 written
  3. Editor wrote the verdict above 10 s · 46,350 tokens read, 896 written
  4. Referee minor revision: 2 major objections, 4 minor 26 s · 45,913 tokens read, 2,952 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper does several things well. It ships a working evaluation kit (drydock-eval) that is fast, safe, and checks fidelity to the hosted scorer at the token ID level. The empirical work is unusually careful: pre-registered protocols, frozen pass rules, clustering by repository, power simulations, and explicit reporting of null results. The noise measurements (run-to-run variation, screen lead as selection artefact) are genuinely useful for anyone building a similar pipeline. The authors clearly understand that a screen does not rank and that a hosted score is one noisy draw. They also release the full run-level data, which is rare and welcome.

The main vulnerability is the one the authors flag themselves: the workstation hardware (single RTX 5090, parallel jobs) is not the same as the hosted scorer (four L4 GPUs, serial). They check this on 24 tasks and find no gross difference, but cannot rule out small or task-specific effects. This does not undercut the kit's purpose (comparative evaluation within the same environment), but it limits what a local result says about an absolute hosted score. The paper is also very long—the extended version runs to 50+ pages with exhaustive tables—and the core findings are nearly all null. That is honest science, but it means the impact is mostly methodological: a model for how to do rigorous local evaluation, rather than a new discovery about agent performance.

The paper is clearly the work of a serious thinker. The design, the checks on the checks, the self-documented limitations, and the refusal to overclaim are all first-rate. Minor issues: some 31B arms are confounded with different blocks, one block's rows are missing, and the third candidate's p = 0.0405 is close but does not clear the prespecified bar—the treatment of that borderline case is fair.

Who should read this? Anyone running local agent evaluations, especially in competition settings, and anyone designing evaluation protocols with small effects and noisy scores. The noise estimates and the screen-vs-confirmation comparison are practical takeaways.

I would accept a peer review invitation for this paper. It is methodologically sound, data-rich, and transparent. The limitations are acknowledged. The contribution is real, even if most of the answers are "no difference detected."

The strongest objection

The paper is transparent about its limitations and does not overclaim. The most plausible vulnerability—the environmental difference between the workstation and the hosted scorer—is explicitly acknowledged (Section 13), checked on 24 tasks (Section 8.3), and noted as insufficient to rule out small task-specific effects. This concern does not undermine the central claim because the kit's primary use is comparative evaluation within the same environment, not absolute score prediction. All other potential weaknesses (low power, selection effects, depletion of the task pool) are documented and discussed with appropriate caution. No unaddressed error or omission supports a change of verdict.

The reader identified the same weakest assumption (workstation–scorer similarity) and still accepted the paper. The paper acknowledges this limitation and does not base its central claims on exact parity. The kit's value lies in internal consistency (high repeatability, conformance checks, paired statistics) and the null results are reported with honest intervals. All other points (low power, selection effects, repeated use of H) are explicitly discussed. Therefore no change in verdict is warranted.

The referee's report

If the null results hold—and the experimental design is rigorous—the paper makes three important contributions. First, it provides a validated local evaluation pipeline with known error rates and documented attack surfaces, which can serve as a template for competitions and agent development workflows. Second, it demonstrates that four documented prompt-rendering divergences (role, compaction, thought retention, template) do not materially affect the E4B agent's solve rate on a 231-task panel, an empirical finding that sharpens the community's understanding of which 'traps' matter for this model class. Third, it publishes a detailed noise budget: run-to-run variation, selection effects in screens, and the limited power of single hosted draws. The release of all 9,660 run rows enables others to plan powered comparisons. The significance is constrained by the specific model (E4B), a single agent bundle, and the use of Berth and development tasks, but within that scope the work is definitive.

Pith's reading of the paper (a machine's summary, not the authors')

This paper presents drydock-eval, a local evaluation kit for small coding agents (specifically the Gemma 4 E4B model) in the context of the Gemma 4 Developer Agent competition. The kit has three modules: Contain (a secure container runner that passed 200 scripted escape tests), Conform (eight checks that verify a local stack reproduces the hosted scorer's prompts down to token IDs), and Compare (paired statistics with repository-clustered tests and a frozen pass rule). Using this kit, the author conducted extensive experiments: serving measurements on four L4 GPUs, comparisons of six 31B agent configurations (solving 37–44 of 70 tasks with no pairwise difference detected), a pre-registered five-arm case study on 231 tasks measuring four prompt divergences (no solve-rate change detected, largest estimate -2.1 points), and three attempts to improve the E4B baseline through screen-and-confirm programmes (none confirmed). The overwhelming result is null: most changes produce no detectable improvement, and the noise structure—most tasks never solved, few tasks drive variability—makes it hard to see any signal.

What carries the argument. The central object is the drydock-eval kit, composed of three modules: Contan (container runner with no network, seccomp filter, read-only root, unprivileged user, eight hash matches before runs start), Conform (eight automated checks comparing local requests to the scorer path, including role, template rendering, thought handling, and compostion settings, with a mutation suite of 103 drifts and 18 controls verifying the checks), and Compare (paired tests with repositories as clusters, a wild cluster bootstrap-t with Rademacher weights, and a frozen pass rule requiring a gain of at least 3.45 points, one-sided p ≤ 0.025, at least three repositories gain, and no repository loses two task-equivalents). The machinery carries the argument by providing an instrument that—if the comparisons are faithful—allows decising whether a candidate agent is genuinely better than the baseline, and the experiments use it to show that most changes are not.

Core claim. The central claim is that the drydock-eval kit provides a fast (≈250 runs per hour on one RTX 5090), safe (network-less containers, 200 passed escape tests), and faithful (token-ID-level comparison against the scorer's configuration) local evaluation that can detect small differences between agent configurations, and that when applied, the largest measured effect of four prompt divergences on the E4B solve rate was -2.1 points (95% interval -4.4 to +0.4, not statistically significant after Holm correction. Three candidate improvements to the E4B agent led their screens but were not confirmed on new tasks (gains -0.35, -0.39, +4.10 points, none passing the pre-registered rule). The paper establishes that developing a small coding agent against a noisy hosted score requires a local evaluator with these properties, and that with such an evaluator most modifications show no detectable effect.

The premise it rests on. The similarity between the workstation (single RTX 5090, eight parallel jobs) and the hosted scorer (four L4 GPUs, serial jobs) is sufficient for results to generalize; the paper checks this on 24 tasks and finds no gross difference, but cannot rule out small or task-specific effects, and acknowledges that environment differences remain.

If the paper is right (Pith's inferences, not the authors')

The null results on E4B are striking: even large prompt changes (affecting 85–90% of requests) did not shift solve rate detectably. This suggests that for this model on these tasks, the solve rate is not determined by the exact phrasing of the prompt but by the model's inherent ability to fix issues—a ceiling effect that might hold for other small models too.

The fact that the runner's high-severity vectors came through shared host resources—and were found by independent verifiers—highlights that container-based sandboxes for agent evaluation need adversarial testing of the host interface, not just of the container boundary.

The three screen-and-confirm programmes show that a 'sure thing' in a small screen (lead of +6.2 points) can vanish on new tasks; the community might adopt a pre-registered two-stage evaluation as a standard to reduce false positives from selection.

The measured run-to-run noise on fixed tasks (SD ≈ 1.01 points for one arm) and the repository-level clustering (tasks within a repository succeed together) suggest that existing benchmarks reporting point estimates without confidence intervals may be misleading; every evaluation should report clustered intervals.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.39

“A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4).”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.84 · human participants 0.12 · controlled comparison 0.97 · randomized assignment 0.07 · states quantitative result 0.99 · compares with prior results 0.85 · mentions code or data availability 0.84

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. contribution 0.46 0.88 0.10 asserted 1.00
Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. method 0.91 0.89 0.08 asserted 1.00
AI agents built and attacked the kit; a human owner approved machine use and release. method 0.48 0.52 0.03 asserted 1.00
This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. contribution 1.00 0.92 0.09 asserted 1.00
On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. result 0.92 0.82 0.03 asserted 1.00
Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. result 1.00 0.91 0.16 asserted 1.00
A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). result 0.99 0.95 0.19 asserted 1.00
Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). result 1.00 0.94 0.10 asserted 1.00
The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. result 1.00 0.95 0.06 asserted 1.00
Prompts and agent bundles are withheld; every run-level result is public. limitation 0.68 0.73 0.05 asserted 1.00

What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,014 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (15)
Figure 1 of the paper
Figure 1. The kit: where each module sits. Main findings. Most of the results are null results, and we report them as such. • No attack on the runner escaped the container’s namespaces or privileges, but nine demon￾strated high-severity vectors came through what the host shared with it, in designs that passed their own escape tests. • Four prompt divergences from the scorer path changed the E4B solve rate by less than the experiment could detect; its simultaneous intervals leave room for a drop of up to 5.1 points and a gain of up to 3.5 points. • Six configurations of a 31B agent, from a…
Figure 2 of the paper
Figure 2. Decode throughput on four L4 GPUs with a context of 2,048 tokens. Left: tokens per second of one stream as the number of concurrent streams grows. Right: tokens per second summed over the streams. public Budget-Fit V5 configuration, the most-run public configuration in the hosted-score census of dariushafshar (Kaggle user) [7], and ID24 is a bundle of the public collection of Team Muhammed Idris [39]. 3 Serving measurements How fast the scorer’s server generates tokens decides how many model calls an agent can afford per task, and whether more than one agent can work on a task at…
Figure 3 of the paper
Figure 3. Canaries passed and high-severity vectors found, per runner version
Figure 4 of the paper
Figure 4. Six configurations of the 31B agent on the same 70 development tasks, one run per task. BF5 37 W1/W2 39 W3 38 SH1/SH3 41 SH5 41 SH6 44 of 70 solved by all six: 25 by some: 25 by none: 20 70 development tasks, ordered by the number of configurations that solved them (one run each)
Figure 5 of the paper
Figure 5. Which configuration solved which task. One column per development task, one row per configu￾ration; a filled cell is a solved run. 25 tasks were solved by all six configurations and 20 by none. The union of the six is 50 tasks, against 44 for the best single one. This gap is an upper bound on what a choice between configurations could gain, and it is inflated: a task that one configuration solves in half of its runs enters the union easily when six runs try it. With one run per task and configuration the two sources cannot be separated. The experiments with E4B below therefore use…
Figure 6 of the paper
Figure 6. Our hosted scores, one run each, against the range of ten public draws of the unchanged BF5 files [7]
Figure 7 of the paper
Figure 7. The unchanged E4B baseline on the workstation. Left: solved runs of each session on the same tasks, with 95% intervals. Right: the 128 tasks of panel H by the number of baseline runs that solved them, of eight runs in two sessions. run from the hosted share; it includes an eighth run, under runner version 2, whose rows were not kept. What this supports is limited. With 24 tasks and one run each, the 95% interval of 4 of 24 runs from 7% to 36%. The table rules out a gross difference between the machines, such as the workstation solving half or double the hosted share. It cannot show…
Figure 8 of the paper
Figure 8. Solved share of the E4B baseline by repository, over all its runs in the screen and confirmation blocks, with 95% intervals that treat runs as independent. Numbers of tasks and runs are in
Figure 9 of the paper
Figure 9. Time of solved and unsolved runs for the two models. E4B: agent phase of the Berth runs on the workstation. 31B: whole job of the development-task runs under emulated latency. E4B is quick, and quicker when it succeeds. On the workstation the agent phase of an E4B run took 75 s on average: 45 s when the task was solved and 84 s when it was not. The 31B runs on development tasks took 278 s, with 231 s and 341 s. The time limit ended 12.6% of the 31B runs and 0.3% of the E4B runs. For the 31B model the limit is a working constraint, and the arms of Section 7 differ mostly in how they…
Figure 10 of the paper
Figure 10. (A) Solve-rate difference, local minus Kaggle (pooled 8/48), for eight local runs of one 24-task list (95% Newcombe intervals, runs treated as independent); shaded: the pass rule’s ±3.45 points. (B) Change in solve rate against F on 231 tasks: estimates, 95% intervals and Holm-adjusted p. all include zero. The largest estimate is H’s −2.1 points (unadjusted p 0.11; −1.3 with time-limit rows as missing). In exploratory measures, H runs averaged 137 s against 78 s and ended unsolved at the bundle’s five-minute limit in 11.5% of runs against 1.8%; in C, 12.3% of runs ended unsolved on…
Figure 11 of the paper
Figure 11. Tasks whose number of solved repeats differs between an arm and F. Bars to the left of the centre line are tasks with fewer solved repeats in the arm. 23
Figure 12 of the paper
Figure 12. Change against F in each repository, per arm. Repositories are ordered by their number of tasks. 9.6 Sensitivity analyses Three sensitivity analyses were registered with the primary one (
Figure 13 of the paper
Figure 13. Agent time per run: each arm (blue) against F (grey). 9.9 What the case study does and does not show The four divergences are not small edits. Three of them change between 85% and 90% of all requests of a run, and one removes the model’s earlier reasoning from every request. Yet the simultaneous intervals exclude a loss of more than 5.1 points and a gain of more than 3.5 points for any of them. For E4B on these tasks, the solved share is insensitive to prompt details that we had expected to matter, and that public discussion treats as traps. Three limits apply. First, the result…
Figure 14 of the paper
Figure 14. Gain of each repository in the three applications of the pass rule (candidate minus baseline). Repositories are sorted by their total within each application. and baseline once on a frozen panel that shares no task with S, and apply the pass rule of Section 6. The whole sequence for one programme took 1,152 to 1,402 runs and three to five hours of work￾station time. 10.1 Pilots and screens
Figure 15 of the paper
Figure 15. Lead of each candidate in its screen on S against its gain in the confirmation run on other tasks, both in percentage points against the baseline of the same block. The horizontal line is the gain that the pass rule asks for. The noise of a screen. The baseline ran in all three screens, so each task of S has six baseline runs. The variation within tasks gives a standard deviation of 2.0 runs, or 2.1 points, for the total of one arm of 96 runs, and 2.9 points for the difference of two arms. S is a hard set for this model: 34 of its 48 tasks were solved in none of the six baseline…

Figures are the authors', reproduced from the paper.

References (50)
  1. [1] Alexandru Agache, Marc Brooker, Andreea Florescu, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa. Firecracker: Lightweight virtual- ization for serverless applications. In 17th USENIX Symposium on Networked Systems De- sign and Impleme... on Pith
  2. [2] Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineerin... on Pithdoi:10.52202/085713-0788
  3. [3] Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal... on PithPith's reviewarXiv
  4. [4] A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics , 90(3):414–427, on Pith
  5. [5] Ivan A. Canay, Andres Santos, and Azeem M. Shaikh. The wild bootstrap with a ”small” number of ”large” clusters. The Review of Economics and Statistics , 103(2):346–363, 2021. doi: 10.1162/rest_a_00887. URL https://doi.org/10.1162/rest_a_00887. Accessed 2026- 11-09 on Pithdoi:10.1162/rest_a_00887
  6. [6] damsolanke (GitHub user). gemma4-swe-kit. GitHub repository (Apache-2.0), 2026. URL https://github.com/damsolanke/gemma4-swe-kit. Accessed 2026-11-09 on Pith
  7. [7] dariushafshar (Kaggle user). Gemma 4 agent: Public config draws. Kaggle dataset (CC BY 4.0) and notebook ’Gemma 4: Most-Run Public Config A verages 10/58’, 2026. URL https: //www.kaggle.com/datasets/dariushafshar/gemma-4-agent-public-config-draws . Ac- cessed 2026-11-09 on Pith
  8. [8] Russell Davidson and Emmanuel Flachaire. The wild bootstrap, tamed at last. Journal of Econometrics , 146(1):162–169, 2008. doi: 10.1016/j.jeconom.2008.08.003. URL https: //doi.org/10.1016/j.jeconom.2008.08.003. Accessed 2026-11-09 on Pithdoi:10.1016/j.jeconom.2008.08.003
  9. [9] delta9000 (GitHub user). vLLM issue #38488, 2026. URL https://github.com/ vllm-project/vllm/issues/38488 on Pith
  10. [10] dimondev. fix: chat_template, keep tool-call reasoning across later user turns. Hugging Face pull request #44 on google/gemma-4-E2B-it, opened 2026-07-28 (open), 2026. URL https://huggingface.co/google/gemma-4-E2B-it/discussions/44 . Accessed 2026-11-09. 35 on Pith
  11. [11] Xing Gao, Zhongshu Gu, Zhengfa Li, Hani Jamjoom, and Cong Wang. Houdini’s escape: Breaking the resource rein of linux control groups. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS ’19) , pages 1073–1086, 2019. doi: 10.1145/331953... on Pithdoi:10.1145/3319535.3354227
  12. [12] Gemma Team. Gemma 4 technical report. arXiv:2607.02770, 2026 on PithPith's reviewarXiv
  13. [13] Georgi Gerganov. Chat template issue: extra newline character when rendering thinking? Hugging Face discussion #135 on google/gemma-4-31B-it, opened 2026-07-16, 2026. URL https://huggingface.co/google/gemma-4-31B-it/discussions/135 . Accessed 2026-11- 09 on Pith
  14. [14] Dmitrii Gluzdov. Gemma 4 task soundness and local results. Kag- gle dataset dmitriigluzdov/gemma-4-task-soundness-and-local-results (CC BY 4.0), 2026. URL https://www.kaggle.com/datasets/dmitriigluzdov/ gemma-4-task-soundness-and-local-results . Accessed 2026-11-09 on Pith
  15. [15] Google AI for Developers. Gemma 4 prompt formatting. Gemma documen- tation, accessed 2026-11-09, 2026. URL https://ai.google.dev/gemma/docs/core/ prompt-formatting-gemma4. Accessed 2026-11-09 on Pith
  16. [16] Google DeepMind. Google - the gemma 4 developer agent competition. Kaggle competition, on Pith
  17. [17] David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A. Smith, Hannaneh Hajishirzi, Kyle Lo, and Jesse Dodge. Signal and noise: A framework for reducing uncertainty in language model evaluation. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025)... on PithPith's reviewarXiv
  18. [18] hemantkumarbk (Kaggle user). Coding-agent thought retention: Reproduction data. Kaggle dataset, 2026. URL https://www.kaggle.com/datasets/hemantkumarbk/ thoughtkeep-reproduction-inputs on Pith
  19. [19] houchiunyo (Kaggle user). Gemma 4 agent resource: runs & checker. Kaggle dataset, 2026. URL https://www.kaggle.com/datasets/houchiunyo/gemma4-agent-silent-zeros on Pith
  20. [20] Hugging Face. Chat templates (transformers documentation). Online documentation, ac- cessed 2026-11-09, 2026. URL https://huggingface.co/docs/transformers/main/en/ chat_templating. Accessed 2026-11-09 on Pith
show 30 more
  1. [21] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations (ICLR 2024) , 2024. URL https:// arxiv.org/abs/23... on PithPith's reviewarXiv
  2. [22] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint, 2024. URL https://arxiv.org/abs/2407.01502. Accessed 2026-11-09. 36 on PithPith's reviewarXiv
  3. [23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principle... on Pithdoi:10.1145/3600006.3613165
  4. [24] James G. MacKinnon and Matthew D. Webb. Wild bootstrap inference for wildly different cluster sizes. Journal of Applied Econometrics , 32(2):233–254, 2017. doi: 10.1002/jae.2508. URL https://doi.org/10.1002/jae.2508. Accessed 2026-11-09 on Pithdoi:10.1002/jae.2508
  5. [25] Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz, and Harry Coppock. Quantifying frontier llm capabilities for container sandbox escape. In Proceedings of the 43rd International Conference on Machine Learning ... on Pith
  6. [26] marcofpereira (Kaggle user). Gemma 4 developer agent lab - instrumentos. Kaggle dataset marcofpereira/gemma4-lab-v1 (paper-track companion; MIT per its text), 2026. URL https: //www.kaggle.com/datasets/marcofpereira/gemma4-lab-v1. Accessed 2026-11-09 on Pith
  7. [27] Elan Markowitz, Bryan Perozzi, Benedek Rózemberczki, Glenn Cameron, Hadi Hemmati, Yuchen Li, Michael Galkin, Majid Farhadi, Ryan Holbrook, and Ashley Oldacre. Google - the gemma 4 developer agent paper track. Kaggle competition, 2026. URL https: //www.kaggle.com/competitions/g... on Pith
  8. [28] Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint, 2024. URL https://arxiv.org/abs/2411.00640. Accessed 2026-11-09 on PithPith's reviewarXiv
  9. [29] Mobassir. Too small to see: Resolution limits, leaky statements and re-sent reasoning when tun- ing a gemma 4 31b software-engineering agent. Gemma 4 Developer Agent Paper Track writeup and companion dataset ’SWE Agent Eval Validity: Gemma 4 Tasks & Results’ (Apache-2.0); auth... on Pith
  10. [30] moshegolderson (Kaggle user). Freshswe-py: Scorer-faithful tasks and three ways a coding leaderboard misleads. Gemma 4 Developer Agent Paper Track writeup; dataset ’FreshSWE- Py: post-cutoff SWE tasks for Gemma 4’, 2026. URL https://www.kaggle.com/datasets/ moshegolderson/fres... on Pith
  11. [31] OpenAI. Introducing swe-bench verified. OpenAI blog post (2024-08-13 per search index), 2024. URL https://openai.com/index/introducing-swe-bench-verified/ . Accessed 2026-11- 09 on Pith
  12. [32] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. In International Conference on Machine Learning (ICML 2025) , 2025. URL https://arxiv.org/abs/2412. 21139. Accessed 2026-11-09 on Pith
  13. [33] Jay Patel. Berth: Harness-compatible, post-cutoff python tasks for offline gemma 4 coding agents. Gemma 4 Developer Agent Paper Track, Kaggle; dataset jayp2005/berth-swe-tasks 37 (Apache-2.0), 2026. URL https://www.kaggle.com/datasets/jayp2005/berth-swe-tasks. Accessed 2026-11-09 on Pith
  14. [34] Rafiqul Rabin, Jesse Hostetler, Sean McGregor, Brett Weir, and Nick Judd. Sandboxeval: Towards securing test environment for untrusted code. arXiv preprint (preliminary version, working paper), 2025. URL https://arxiv.org/abs/2504.00018. Accessed 2026-11-09 on PithPith's reviewarXiv
  15. [35] rladeira. [litellm] role=’tool_responses’ for gemma 4 breaks tool calling on vertex ai model garden maas (400 invalid_argument) (issue #7977) and fix(models): use standard tool role for gemma 4 on vertex ai (pull request #7978). google/adk-python GitHub issue #7977 (opened 202... on Pith
  16. [36] ryanholbrook (Kaggle user). Getting started - Gemma 4 developer agent. Kaggle notebook, 2026. URL https://www.kaggle.com/code/ryanholbrook/ getting-started-gemma-4-developer-agent on Pith
  17. [37] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In International Conference on Learning Representations (ICLR 2024) , on Pith
  18. [38] Jayden Szeto, Daniel Guo, and Hayley Szeto. Scorer-faithful local evaluation for small coding agents. Gemma 4 Developer Agent Paper Track writeup; dataset ’Gemma 4 agent: scorer- faithful rig and run logs’ (Apache-2.0), 2026. URL https://www.kaggle.com/datasets/ jaydenszeto123... on Pith
  19. [39] Team Muhammed Idris. gemma4-agent-runs. Kaggle dataset (Apache-2.0), 2026. URL https: //www.kaggle.com/datasets/muhammedidris28/gemma4-agent-runs on Pith
  20. [40] The Agent Lightning (AGL) Team. No more retokenization drift: Returning token ids via the openai compatible api matters in agent rl. vLLM Blog, 2025-10-22, 2025. URL https: //vllm.ai/blog/2025-10-22-agent-lightning . Accessed 2026-11-09 on Pith
  21. [41] tyfeng1997 (Hugging Face user). gemma4-e4b-swe-sft-32k-competition-template. Hug- ging Face dataset, 2026. URL https://huggingface.co/datasets/tyfeng1997/ gemma4-e4b-swe-sft-32k-competition-template on Pith
  22. [42] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna... on PithPith's reviewarXiv
  23. [43] Matthew D. Webb. Reworking wild bootstrap-based inference for clustered errors. Canadian Journal of Economics/Revue canadienne d’économique , 56(3):839–858, 2023. doi: 10.1111/ caje.12661. URL https://doi.org/10.1111/caje.12661. Accessed 2026-11-09. 38 on Pithdoi:10.1111/caje.12661
  24. [44] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024) , pages 50528–50652... on PithPith's reviewarXiv
  25. [45] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025) ... on PithPith's reviewarXiv
  26. [46] Ethan G. Young, Pengfei Zhu, Tyler Caraza-Harter, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. The true cost of containing: A gvisor case study. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19) , 2019. URL https://www.usenix.org/ conference/hotc... on Pith
  27. [47] Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Ant... on PithPith's reviewarXiv
  28. [2008] doi: 10.1162/rest.90.3.414. URL https://doi.org/10.1162/rest.90.3.414. Accessed 2026-11-09 on Pithdoi:10.1162/rest.90.3.414
  29. [2024] URL https://arxiv.org/abs/2310.11324. Accessed 2026-11-09 on PithPith's reviewarXiv
  30. [2026] URL https://www.kaggle.com/competitions/gemma-4-developer-agent . Accessed 2026-11-09 on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment