Artificial Intelligence · arXiv:2611.16485 v1 · submitted 12 Nov 2026 · 50 references
Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments
The paper says
“A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090.”
more of the abstract
Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Thorough, transparent, and well-executed evaluation kit for coding agents, mostly reporting null results with honest limitations. This line is wrong
2 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 50 references say.
The paper says, at §10, Table 29 and Table 30
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The screen set S (48 tasks) is described as 'not fresh' (Section 13): 32 of its 48 tasks had been run in earlier blocks. Because the screen selects candidates from several variants, and prior exposure of those tasks could have informed the variant design (even if the paper states that variants were built from general ideas, not per-task results), the possibility of information leakage into the variant selection should be discussed more explicitly.
more of objection 1
The paper acknowledges the non-freshness in the limitations, but given that the screens are the sole mechanism for candidate selection, the degree to which previous runs on those tasks could have influenced the builds (e.g., through debugging or tuning) should be quantified or argued. Otherwise, the screening procedure's theoretical protection against selection bias is weakened.
The paper says, at §9.1 (Table 18) and §9.9
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.71):
“It cannot show that a specific difference in the prompts has no effect.”
Pith says: Objection 2 of 2 · major
The case study's intervals (98.75% simultaneous intervals) leave room for drops of up to 5.1 points (arm H) and gains of up to 3.5 points (arm T). The paper correctly calls these 'null results, not evidence of no effect.' However, the abstract and introduction state the result as 'no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4).'
more of objection 2
While technically accurate, a reader could misinterpret this as evidence of equivalence. The distinction between 'not detected' and 'no effect' is carefully made elsewhere, but the framing in the abstract could be tightened to avoid over-interpretation by readers who skip the detailed discussion.
4 minor objections
The paper says, at §2.5
Pith says: minor objection 1 of 4
The term 'points' for percentage points is defined, but the first occurrence in the abstract uses it without definition. Consider adding a brief clarification in the abstract or early in Section 1.
The paper says, at Table 5 (Section 6)
Pith says: minor objection 2 of 4
The simulated pass rates for the 'no effect, exchangeable tasks' scenario show a family-wise false-claim rate of 0.23–1.90% for the small panel and 0.00–0.50% for the large panel. This is reported without an explicit justification of why the simulation uses a 0.025 threshold rather than the standard 0.05; the 2.5% level is mentioned later in Section 9.1.
more of objection 2
The connection would be clearer if the simulation description referenced the rule's requirement (p ≤ 0.025) and explained that the simulated size is the fraction of runs where p ≤ 0.025 under the null.
show 2 more
The paper says, at Fig. 2 and Table 1
Pith says: minor objection 3 of 4
The caption of Fig. 2 mentions 'disturbed measurement' for two cells, but the corresponding entry in Table 1 (8 streams, 8-192 context) is flagged with a short note. Consider adding a footnote to Fig. 2 that explains the KV-cache eviction and the unexplained cell, referencing the two sentences in the text.
The paper says, at Appendix B (Table 34)
Pith says: minor objection 4 of 4
The recomputation of the case study intervals with the released kit shows that the upper end of the H arm's interval moves from +0.38 to +0.25 percentage points depending on task order. The paper notes this and calls the intervals consistent.
more of objection 4
However, the test decision (rejection or not) does not change, but the interval width fluctuation of ~0.13 points could be larger than the precision implied in the main report (which gives 95% intervals to 0.01 points). For clarity, state that the reported intervals are based on the registered panel order and that small Monte Carlo variations (up to 0.13 points) are typical.
What would settle it
To further validate the workstation–scorer similarity, run a larger parity experiment: 100 tasks from the validated pool, 4 repeats each, on both the workstation (RTX 5090, 8 workers) and a 4×L4 notebook (serial). Compute the per-task difference in solve rates. If the 95% interval of the mean difference exceeds ±2 percentage points or if the cross-machine correlation of per-task solved counts is below 0.85, the assumption of interchangeability weakens. This test would quantify the residual environment effect that the 24-task check could not resolve.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does several things well. It ships a working evaluation kit (drydock-eval) that is fast, safe, and checks fidelity to the hosted scorer at the token ID level. The empirical work is unusually careful: pre-registered protocols, frozen pass rules, clustering by repository, power simulations, and explicit reporting of null results. The noise measurements (run-to-run variation, screen lead as selection artefact) are genuinely useful for anyone building a similar pipeline. The authors clearly understand that a screen does not rank and that a hosted score is one noisy draw. They also release the full run-level data, which is rare and welcome.
The main vulnerability is the one the authors flag themselves: the workstation hardware (single RTX 5090, parallel jobs) is not the same as the hosted scorer (four L4 GPUs, serial). They check this on 24 tasks and find no gross difference, but cannot rule out small or task-specific effects. This does not undercut the kit's purpose (comparative evaluation within the same environment), but it limits what a local result says about an absolute hosted score. The paper is also very long—the extended version runs to 50+ pages with exhaustive tables—and the core findings are nearly all null. That is honest science, but it means the impact is mostly methodological: a model for how to do rigorous local evaluation, rather than a new discovery about agent performance.
The paper is clearly the work of a serious thinker. The design, the checks on the checks, the self-documented limitations, and the refusal to overclaim are all first-rate. Minor issues: some 31B arms are confounded with different blocks, one block's rows are missing, and the third candidate's p = 0.0405 is close but does not clear the prespecified bar—the treatment of that borderline case is fair.
Who should read this? Anyone running local agent evaluations, especially in competition settings, and anyone designing evaluation protocols with small effects and noisy scores. The noise estimates and the screen-vs-confirmation comparison are practical takeaways.
I would accept a peer review invitation for this paper. It is methodologically sound, data-rich, and transparent. The limitations are acknowledged. The contribution is real, even if most of the answers are "no difference detected."
The strongest objection
The paper is transparent about its limitations and does not overclaim. The most plausible vulnerability—the environmental difference between the workstation and the hosted scorer—is explicitly acknowledged (Section 13), checked on 24 tasks (Section 8.3), and noted as insufficient to rule out small task-specific effects. This concern does not undermine the central claim because the kit's primary use is comparative evaluation within the same environment, not absolute score prediction. All other potential weaknesses (low power, selection effects, depletion of the task pool) are documented and discussed with appropriate caution. No unaddressed error or omission supports a change of verdict.
The reader identified the same weakest assumption (workstation–scorer similarity) and still accepted the paper. The paper acknowledges this limitation and does not base its central claims on exact parity. The kit's value lies in internal consistency (high repeatability, conformance checks, paired statistics) and the null results are reported with honest intervals. All other points (low power, selection effects, repeated use of H) are explicitly discussed. Therefore no change in verdict is warranted.
The referee's report
If the null results hold—and the experimental design is rigorous—the paper makes three important contributions. First, it provides a validated local evaluation pipeline with known error rates and documented attack surfaces, which can serve as a template for competitions and agent development workflows. Second, it demonstrates that four documented prompt-rendering divergences (role, compaction, thought retention, template) do not materially affect the E4B agent's solve rate on a 231-task panel, an empirical finding that sharpens the community's understanding of which 'traps' matter for this model class. Third, it publishes a detailed noise budget: run-to-run variation, selection effects in screens, and the limited power of single hosted draws. The release of all 9,660 run rows enables others to plan powered comparisons. The significance is constrained by the specific model (E4B), a single agent bundle, and the use of Berth and development tasks, but within that scope the work is definitive.
Pith's reading of the paper (a machine's summary, not the authors')
This paper presents drydock-eval, a local evaluation kit for small coding agents (specifically the Gemma 4 E4B model) in the context of the Gemma 4 Developer Agent competition. The kit has three modules: Contain (a secure container runner that passed 200 scripted escape tests), Conform (eight checks that verify a local stack reproduces the hosted scorer's prompts down to token IDs), and Compare (paired statistics with repository-clustered tests and a frozen pass rule). Using this kit, the author conducted extensive experiments: serving measurements on four L4 GPUs, comparisons of six 31B agent configurations (solving 37–44 of 70 tasks with no pairwise difference detected), a pre-registered five-arm case study on 231 tasks measuring four prompt divergences (no solve-rate change detected, largest estimate -2.1 points), and three attempts to improve the E4B baseline through screen-and-confirm programmes (none confirmed). The overwhelming result is null: most changes produce no detectable improvement, and the noise structure—most tasks never solved, few tasks drive variability—makes it hard to see any signal.
What carries the argument. The central object is the drydock-eval kit, composed of three modules: Contan (container runner with no network, seccomp filter, read-only root, unprivileged user, eight hash matches before runs start), Conform (eight automated checks comparing local requests to the scorer path, including role, template rendering, thought handling, and compostion settings, with a mutation suite of 103 drifts and 18 controls verifying the checks), and Compare (paired tests with repositories as clusters, a wild cluster bootstrap-t with Rademacher weights, and a frozen pass rule requiring a gain of at least 3.45 points, one-sided p ≤ 0.025, at least three repositories gain, and no repository loses two task-equivalents). The machinery carries the argument by providing an instrument that—if the comparisons are faithful—allows decising whether a candidate agent is genuinely better than the baseline, and the experiments use it to show that most changes are not.
Core claim. The central claim is that the drydock-eval kit provides a fast (≈250 runs per hour on one RTX 5090), safe (network-less containers, 200 passed escape tests), and faithful (token-ID-level comparison against the scorer's configuration) local evaluation that can detect small differences between agent configurations, and that when applied, the largest measured effect of four prompt divergences on the E4B solve rate was -2.1 points (95% interval -4.4 to +0.4, not statistically significant after Holm correction. Three candidate improvements to the E4B agent led their screens but were not confirmed on new tasks (gains -0.35, -0.39, +4.10 points, none passing the pre-registered rule). The paper establishes that developing a small coding agent against a noisy hosted score requires a local evaluator with these properties, and that with such an evaluator most modifications show no detectable effect.
The premise it rests on. The similarity between the workstation (single RTX 5090, eight parallel jobs) and the hosted scorer (four L4 GPUs, serial jobs) is sufficient for results to generalize; the paper checks this on 24 tasks and finds no gross difference, but cannot rule out small or task-specific effects, and acknowledges that environment differences remain.
If the paper is right (Pith's inferences, not the authors')
The null results on E4B are striking: even large prompt changes (affecting 85–90% of requests) did not shift solve rate detectably. This suggests that for this model on these tasks, the solve rate is not determined by the exact phrasing of the prompt but by the model's inherent ability to fix issues—a ceiling effect that might hold for other small models too.
The fact that the runner's high-severity vectors came through shared host resources—and were found by independent verifiers—highlights that container-based sandboxes for agent evaluation need adversarial testing of the host interface, not just of the container boundary.
The three screen-and-confirm programmes show that a 'sure thing' in a small screen (lead of +6.2 points) can vanish on new tasks; the community might adopt a pre-registered two-stage evaluation as a standard to reduce false positives from selection.
The measured run-to-run noise on fixed tasks (SD ≈ 1.01 points for one arm) and the repository-level clustering (tasks within a repository succeed together) suggest that existing benchmarks reporting point estimates without confidence intervals may be misleading; every evaluation should report clustered intervals.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. AI agents built and attacked the kit; a human owner approved machine use and release. This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. Prompts and agent bundles are withheld; every run-level result is public.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 0.39
“A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4).”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.84 · human participants 0.12 · controlled comparison 0.97 · randomized assignment 0.07 · states quantitative result 0.99 · compares with prior results 0.85 · mentions code or data availability 0.84
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| A hosted score in the Gemma 4 Developer Agent competition is one noisy number, so development needs a local evaluator that is fast, safe for agent-written code, and matching the hosted scorer's prompts. drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090. | contribution 0.46 | 0.88 | 0.10 | asserted 1.00 |
| Contain runs each job in a fresh container without network and passed 200 scripted escape tests; conform provides eight checks that compare a local stack's requests with the scorer's configuration, down to rendered tokens; compare provides paired statistics that respect repository clusters. | method 0.91 | 0.89 | 0.08 | asserted 1.00 |
| AI agents built and attacked the kit; a human owner approved machine use and release. | method 0.48 | 0.52 | 0.03 | asserted 1.00 |
| This extended version repeats the short paper and adds the full record of the experiments made with and around the kit: 9,660 rows of agent runs, 2,494 grading controls and 72 serving measurements, all released. | contribution 1.00 | 0.92 | 0.09 | asserted 1.00 |
| On four L4 GPUs the 31B model decodes 38.1 tokens per second in one stream and E4B 105.8. | result 0.92 | 0.82 | 0.03 | asserted 1.00 |
| Six configurations of a 31B agent solved between 37 and 44 of 70 development tasks, and no pairwise difference was detected. | result 1.00 | 0.91 | 0.16 | asserted 1.00 |
| A pre-registered five-arm experiment (231 tasks, 4,620 jobs) measured four prompt divergences: no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4). | result 0.99 | 0.95 | 0.19 | asserted 1.00 |
| Three changes of the E4B agent led their screens by +1.0 to +6.2 points; each was then tested once on new tasks under a frozen pass rule, and none was confirmed (gains -0.35, -0.39 and +4.10 points). | result 1.00 | 0.94 | 0.10 | asserted 1.00 |
| The unchanged baseline, run again on the same 128 tasks, moved by 1.17 points, and 65% of these tasks were solved in none of eight runs. | result 1.00 | 0.95 | 0.06 | asserted 1.00 |
| Prompts and agent bundles are withheld; every run-level result is public. | limitation 0.68 | 0.73 | 0.05 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,014 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (15)
Figures are the authors', reproduced from the paper.
References (50)
- [1] Alexandru Agache, Marc Brooker, Andreea Florescu, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa. Firecracker: Lightweight virtual- ization for serverless applications. In 17th USENIX Symposium on Networked Systems De- sign and Impleme... on Pith
- [2] Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineerin... on Pithdoi:10.52202/085713-0788
- [3] Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal... on PithPith's reviewarXiv
- [4] A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics , 90(3):414–427, on Pith
- [5] Ivan A. Canay, Andres Santos, and Azeem M. Shaikh. The wild bootstrap with a ”small” number of ”large” clusters. The Review of Economics and Statistics , 103(2):346–363, 2021. doi: 10.1162/rest_a_00887. URL https://doi.org/10.1162/rest_a_00887. Accessed 2026- 11-09 on Pithdoi:10.1162/rest_a_00887
- [6] damsolanke (GitHub user). gemma4-swe-kit. GitHub repository (Apache-2.0), 2026. URL https://github.com/damsolanke/gemma4-swe-kit. Accessed 2026-11-09 on Pith
- [7] dariushafshar (Kaggle user). Gemma 4 agent: Public config draws. Kaggle dataset (CC BY 4.0) and notebook ’Gemma 4: Most-Run Public Config A verages 10/58’, 2026. URL https: //www.kaggle.com/datasets/dariushafshar/gemma-4-agent-public-config-draws . Ac- cessed 2026-11-09 on Pith
- [8] Russell Davidson and Emmanuel Flachaire. The wild bootstrap, tamed at last. Journal of Econometrics , 146(1):162–169, 2008. doi: 10.1016/j.jeconom.2008.08.003. URL https: //doi.org/10.1016/j.jeconom.2008.08.003. Accessed 2026-11-09 on Pithdoi:10.1016/j.jeconom.2008.08.003
- [9] delta9000 (GitHub user). vLLM issue #38488, 2026. URL https://github.com/ vllm-project/vllm/issues/38488 on Pith
- [10] dimondev. fix: chat_template, keep tool-call reasoning across later user turns. Hugging Face pull request #44 on google/gemma-4-E2B-it, opened 2026-07-28 (open), 2026. URL https://huggingface.co/google/gemma-4-E2B-it/discussions/44 . Accessed 2026-11-09. 35 on Pith
- [11] Xing Gao, Zhongshu Gu, Zhengfa Li, Hani Jamjoom, and Cong Wang. Houdini’s escape: Breaking the resource rein of linux control groups. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS ’19) , pages 1073–1086, 2019. doi: 10.1145/331953... on Pithdoi:10.1145/3319535.3354227
- [12] Gemma Team. Gemma 4 technical report. arXiv:2607.02770, 2026 on PithPith's reviewarXiv
- [13] Georgi Gerganov. Chat template issue: extra newline character when rendering thinking? Hugging Face discussion #135 on google/gemma-4-31B-it, opened 2026-07-16, 2026. URL https://huggingface.co/google/gemma-4-31B-it/discussions/135 . Accessed 2026-11- 09 on Pith
- [14] Dmitrii Gluzdov. Gemma 4 task soundness and local results. Kag- gle dataset dmitriigluzdov/gemma-4-task-soundness-and-local-results (CC BY 4.0), 2026. URL https://www.kaggle.com/datasets/dmitriigluzdov/ gemma-4-task-soundness-and-local-results . Accessed 2026-11-09 on Pith
- [15] Google AI for Developers. Gemma 4 prompt formatting. Gemma documen- tation, accessed 2026-11-09, 2026. URL https://ai.google.dev/gemma/docs/core/ prompt-formatting-gemma4. Accessed 2026-11-09 on Pith
- [16] Google DeepMind. Google - the gemma 4 developer agent competition. Kaggle competition, on Pith
- [17] David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A. Smith, Hannaneh Hajishirzi, Kyle Lo, and Jesse Dodge. Signal and noise: A framework for reducing uncertainty in language model evaluation. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025)... on PithPith's reviewarXiv
- [18] hemantkumarbk (Kaggle user). Coding-agent thought retention: Reproduction data. Kaggle dataset, 2026. URL https://www.kaggle.com/datasets/hemantkumarbk/ thoughtkeep-reproduction-inputs on Pith
- [19] houchiunyo (Kaggle user). Gemma 4 agent resource: runs & checker. Kaggle dataset, 2026. URL https://www.kaggle.com/datasets/houchiunyo/gemma4-agent-silent-zeros on Pith
- [20] Hugging Face. Chat templates (transformers documentation). Online documentation, ac- cessed 2026-11-09, 2026. URL https://huggingface.co/docs/transformers/main/en/ chat_templating. Accessed 2026-11-09 on Pith
show 30 more
- [21] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations (ICLR 2024) , 2024. URL https:// arxiv.org/abs/23... on PithPith's reviewarXiv
- [22] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint, 2024. URL https://arxiv.org/abs/2407.01502. Accessed 2026-11-09. 36 on PithPith's reviewarXiv
- [23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principle... on Pithdoi:10.1145/3600006.3613165
- [24] James G. MacKinnon and Matthew D. Webb. Wild bootstrap inference for wildly different cluster sizes. Journal of Applied Econometrics , 32(2):233–254, 2017. doi: 10.1002/jae.2508. URL https://doi.org/10.1002/jae.2508. Accessed 2026-11-09 on Pithdoi:10.1002/jae.2508
- [25] Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz, and Harry Coppock. Quantifying frontier llm capabilities for container sandbox escape. In Proceedings of the 43rd International Conference on Machine Learning ... on Pith
- [26] marcofpereira (Kaggle user). Gemma 4 developer agent lab - instrumentos. Kaggle dataset marcofpereira/gemma4-lab-v1 (paper-track companion; MIT per its text), 2026. URL https: //www.kaggle.com/datasets/marcofpereira/gemma4-lab-v1. Accessed 2026-11-09 on Pith
- [27] Elan Markowitz, Bryan Perozzi, Benedek Rózemberczki, Glenn Cameron, Hadi Hemmati, Yuchen Li, Michael Galkin, Majid Farhadi, Ryan Holbrook, and Ashley Oldacre. Google - the gemma 4 developer agent paper track. Kaggle competition, 2026. URL https: //www.kaggle.com/competitions/g... on Pith
- [28] Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint, 2024. URL https://arxiv.org/abs/2411.00640. Accessed 2026-11-09 on PithPith's reviewarXiv
- [29] Mobassir. Too small to see: Resolution limits, leaky statements and re-sent reasoning when tun- ing a gemma 4 31b software-engineering agent. Gemma 4 Developer Agent Paper Track writeup and companion dataset ’SWE Agent Eval Validity: Gemma 4 Tasks & Results’ (Apache-2.0); auth... on Pith
- [30] moshegolderson (Kaggle user). Freshswe-py: Scorer-faithful tasks and three ways a coding leaderboard misleads. Gemma 4 Developer Agent Paper Track writeup; dataset ’FreshSWE- Py: post-cutoff SWE tasks for Gemma 4’, 2026. URL https://www.kaggle.com/datasets/ moshegolderson/fres... on Pith
- [31] OpenAI. Introducing swe-bench verified. OpenAI blog post (2024-08-13 per search index), 2024. URL https://openai.com/index/introducing-swe-bench-verified/ . Accessed 2026-11- 09 on Pith
- [32] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. In International Conference on Machine Learning (ICML 2025) , 2025. URL https://arxiv.org/abs/2412. 21139. Accessed 2026-11-09 on Pith
- [33] Jay Patel. Berth: Harness-compatible, post-cutoff python tasks for offline gemma 4 coding agents. Gemma 4 Developer Agent Paper Track, Kaggle; dataset jayp2005/berth-swe-tasks 37 (Apache-2.0), 2026. URL https://www.kaggle.com/datasets/jayp2005/berth-swe-tasks. Accessed 2026-11-09 on Pith
- [34] Rafiqul Rabin, Jesse Hostetler, Sean McGregor, Brett Weir, and Nick Judd. Sandboxeval: Towards securing test environment for untrusted code. arXiv preprint (preliminary version, working paper), 2025. URL https://arxiv.org/abs/2504.00018. Accessed 2026-11-09 on PithPith's reviewarXiv
- [35] rladeira. [litellm] role=’tool_responses’ for gemma 4 breaks tool calling on vertex ai model garden maas (400 invalid_argument) (issue #7977) and fix(models): use standard tool role for gemma 4 on vertex ai (pull request #7978). google/adk-python GitHub issue #7977 (opened 202... on Pith
- [36] ryanholbrook (Kaggle user). Getting started - Gemma 4 developer agent. Kaggle notebook, 2026. URL https://www.kaggle.com/code/ryanholbrook/ getting-started-gemma-4-developer-agent on Pith
- [37] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In International Conference on Learning Representations (ICLR 2024) , on Pith
- [38] Jayden Szeto, Daniel Guo, and Hayley Szeto. Scorer-faithful local evaluation for small coding agents. Gemma 4 Developer Agent Paper Track writeup; dataset ’Gemma 4 agent: scorer- faithful rig and run logs’ (Apache-2.0), 2026. URL https://www.kaggle.com/datasets/ jaydenszeto123... on Pith
- [39] Team Muhammed Idris. gemma4-agent-runs. Kaggle dataset (Apache-2.0), 2026. URL https: //www.kaggle.com/datasets/muhammedidris28/gemma4-agent-runs on Pith
- [40] The Agent Lightning (AGL) Team. No more retokenization drift: Returning token ids via the openai compatible api matters in agent rl. vLLM Blog, 2025-10-22, 2025. URL https: //vllm.ai/blog/2025-10-22-agent-lightning . Accessed 2026-11-09 on Pith
- [41] tyfeng1997 (Hugging Face user). gemma4-e4b-swe-sft-32k-competition-template. Hug- ging Face dataset, 2026. URL https://huggingface.co/datasets/tyfeng1997/ gemma4-e4b-swe-sft-32k-competition-template on Pith
- [42] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna... on PithPith's reviewarXiv
- [43] Matthew D. Webb. Reworking wild bootstrap-based inference for clustered errors. Canadian Journal of Economics/Revue canadienne d’économique , 56(3):839–858, 2023. doi: 10.1111/ caje.12661. URL https://doi.org/10.1111/caje.12661. Accessed 2026-11-09. 38 on Pithdoi:10.1111/caje.12661
- [44] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024) , pages 50528–50652... on PithPith's reviewarXiv
- [45] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025) ... on PithPith's reviewarXiv
- [46] Ethan G. Young, Pengfei Zhu, Tyler Caraza-Harter, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. The true cost of containing: A gvisor case study. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19) , 2019. URL https://www.usenix.org/ conference/hotc... on Pith
- [47] Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Ant... on PithPith's reviewarXiv
- [2008] doi: 10.1162/rest.90.3.414. URL https://doi.org/10.1162/rest.90.3.414. Accessed 2026-11-09 on Pithdoi:10.1162/rest.90.3.414
- [2024] URL https://arxiv.org/abs/2310.11324. Accessed 2026-11-09 on PithPith's reviewarXiv
- [2026] URL https://www.kaggle.com/competitions/gemma-4-developer-agent . Accessed 2026-11-09 on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment