{"id":"fc2f7e47-cfa0-4de4-94fe-84da7fe6eef2","arxiv_id":"2611.16485","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"A local evaluation kit and extensive null-result experiments show that small coding agents are insensitive to many prompt and configuration changes.","lead":"This paper introduces drydock-eval, a local evaluation toolkit for small coding agents that runs 250 agent evaluations per hour on a single GPU. It provides safety through containerized execution, conformance checks against a hosted scorer, and statistically rigorous comparisons, plus a record of over 9,000 agent runs that found mostly null results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader identified the same weakest assumption (workstation–scorer similarity) and still accepted the paper. The paper acknowledges this limitation and does not base its central claims on exact parity. The kit's value lies in internal consistency (high repeatability, conformance checks, paired statistics) and the null results are reported with honest intervals. All other points (low power, selection effects, repeated use of H) are explicitly discussed. Therefore no change in verdict is warranted.","tokens_in":45690,"tokens_out":2717,"duration_ms":24325,"concrete_test":"To further validate the workstation–scorer similarity, run a larger parity experiment: 100 tasks from the validated pool, 4 repeats each, on both the workstation (RTX 5090, 8 workers) and a 4×L4 notebook (serial). Compute the per-task difference in solve rates. If the 95% interval of the mean difference exceeds ±2 percentage points or if the cross-machine correlation of per-task solved counts is below 0.85, the assumption of interchangeability weakens. This test would quantify the residual environment effect that the 24-task check could not resolve.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper is transparent about its limitations and does not overclaim. The most plausible vulnerability—the environmental difference between the workstation and the hosted scorer—is explicitly acknowledged (Section 13), checked on 24 tasks (Section 8.3), and noted as insufficient to rule out small task-specific effects. This concern does not undermine the central claim because the kit's primary use is comparative evaluation within the same environment, not absolute score prediction. All other potential weaknesses (low power, selection effects, depletion of the task pool) are documented and discussed with appropriate caution. No unaddressed error or omission supports a change of verdict.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper presents drydock-eval, an evaluation kit for small coding agents in the Gemma 4 Developer Agent competition, and reports the full record of experiments conducted with it. The kit provides container-based safe execution (Contain), eight checks for local-stack fidelity with the hosted scorer (Conform), and a repository-clustered statistical comparison procedure with a frozen pass rule (Compare). The experimental record covers 5,024 agent runs across 31 blocks with the 31B and E4B models, plus a pre-registered 4,620-job case study measuring four prompt divergences. The central findings are null: six 31B agent configurations were indistinguishable on 70 development tasks; four prompt divergences caused no detectable solve-rate change for E4B (largest estimate -2.1 points, 98.75% interval [-5.1, +0.95]); and three candidate E4B improvements selected by screens were not confirmed on new tasks (gains -0.35, -0.39, and +4.10 points with p=0.0405, above the 0.025 threshold). The paper transparently documents limitations, missing rows, confounded arms, and post-hoc analyses.","tokens_in":45913,"tokens_out":2952,"duration_ms":26196,"significance":"If the null results hold—and the experimental design is rigorous—the paper makes three important contributions. First, it provides a validated local evaluation pipeline with known error rates and documented attack surfaces, which can serve as a template for competitions and agent development workflows. Second, it demonstrates that four documented prompt-rendering divergences (role, compaction, thought retention, template) do not materially affect the E4B agent's solve rate on a 231-task panel, an empirical finding that sharpens the community's understanding of which 'traps' matter for this model class. Third, it publishes a detailed noise budget: run-to-run variation, selection effects in screens, and the limited power of single hosted draws. The release of all 9,660 run rows enables others to plan powered comparisons. The significance is constrained by the specific model (E4B), a single agent bundle, and the use of Berth and development tasks, but within that scope the work is definitive.","major_comments":[{"comment":"The screen set S (48 tasks) is described as 'not fresh' (Section 13): 32 of its 48 tasks had been run in earlier blocks. Because the screen selects candidates from several variants, and prior exposure of those tasks could have informed the variant design (even if the paper states that variants were built from general ideas, not per-task results), the possibility of information leakage into the variant selection should be discussed more explicitly. The paper acknowledges the non-freshness in the limitations, but given that the screens are the sole mechanism for candidate selection, the degree to which previous runs on those tasks could have influenced the builds (e.g., through debugging or tuning) should be quantified or argued. Otherwise, the screening procedure's theoretical protection against selection bias is weakened.","section":"§10, Table 29 and Table 30","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The case study's intervals (98.75% simultaneous intervals) leave room for drops of up to 5.1 points (arm H) and gains of up to 3.5 points (arm T). The paper correctly calls these 'null results, not evidence of no effect.' However, the abstract and introduction state the result as 'no E4B solve-rate change was detected (largest estimate -2.1 points, 95% interval -4.4 to +0.4).' While technically accurate, a reader could misinterpret this as evidence of equivalence. The distinction between 'not detected' and 'no effect' is carefully made elsewhere, but the framing in the abstract could be tightened to avoid over-interpretation by readers who skip the detailed discussion.","section":"§9.1 (Table 18) and §9.9","authors_acknowledge":"It cannot show that a specific difference in the prompts has no effect.","authors_acknowledge_p":0.71}],"minor_comments":[{"comment":"The term 'points' for percentage points is defined, but the first occurrence in the abstract uses it without definition. Consider adding a brief clarification in the abstract or early in Section 1.","section":"§2.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The simulated pass rates for the 'no effect, exchangeable tasks' scenario show a family-wise false-claim rate of 0.23–1.90% for the small panel and 0.00–0.50% for the large panel. This is reported without an explicit justification of why the simulation uses a 0.025 threshold rather than the standard 0.05; the 2.5% level is mentioned later in Section 9.1. The connection would be clearer if the simulation description referenced the rule's requirement (p ≤ 0.025) and explained that the simulated size is the fraction of runs where p ≤ 0.025 under the null.","section":"Table 5 (Section 6)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The caption of Fig. 2 mentions 'disturbed measurement' for two cells, but the corresponding entry in Table 1 (8 streams, 8-192 context) is flagged with a short note. Consider adding a footnote to Fig. 2 that explains the KV-cache eviction and the unexplained cell, referencing the two sentences in the text.","section":"Fig. 2 and Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The recomputation of the case study intervals with the released kit shows that the upper end of the H arm's interval moves from +0.38 to +0.25 percentage points depending on task order. The paper notes this and calls the intervals consistent. However, the test decision (rejection or not) does not change, but the interval width fluctuation of ~0.13 points could be larger than the precision implied in the main report (which gives 95% intervals to 0.01 points). For clarity, state that the reported intervals are based on the registered panel order and that small Monte Carlo variations (up to 0.13 points) are typical.","section":"Appendix B (Table 34)","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper does several things well. It ships a working evaluation kit (drydock-eval) that is fast, safe, and checks fidelity to the hosted scorer at the token ID level. The empirical work is unusually careful: pre-registered protocols, frozen pass rules, clustering by repository, power simulations, and explicit reporting of null results. The noise measurements (run-to-run variation, screen lead as selection artefact) are genuinely useful for anyone building a similar pipeline. The authors clearly understand that a screen does not rank and that a hosted score is one noisy draw. They also release the full run-level data, which is rare and welcome.\n\nThe main vulnerability is the one the authors flag themselves: the workstation hardware (single RTX 5090, parallel jobs) is not the same as the hosted scorer (four L4 GPUs, serial). They check this on 24 tasks and find no gross difference, but cannot rule out small or task-specific effects. This does not undercut the kit's purpose (comparative evaluation within the same environment), but it limits what a local result says about an absolute hosted score. The paper is also very long—the extended version runs to 50+ pages with exhaustive tables—and the core findings are nearly all null. That is honest science, but it means the impact is mostly methodological: a model for how to do rigorous local evaluation, rather than a new discovery about agent performance.\n\nThe paper is clearly the work of a serious thinker. The design, the checks on the checks, the self-documented limitations, and the refusal to overclaim are all first-rate. Minor issues: some 31B arms are confounded with different blocks, one block's rows are missing, and the third candidate's p = 0.0405 is close but does not clear the prespecified bar—the treatment of that borderline case is fair.\n\nWho should read this? Anyone running local agent evaluations, especially in competition settings, and anyone designing evaluation protocols with small effects and noisy scores. The noise estimates and the screen-vs-confirmation comparison are practical takeaways.\n\nI would accept a peer review invitation for this paper. It is methodologically sound, data-rich, and transparent. The limitations are acknowledged. The contribution is real, even if most of the answers are \"no difference detected.\"","headline":"Thorough, transparent, and well-executed evaluation kit for coding agents, mostly reporting null results with honest limitations.","tokens_in":46350,"tokens_out":896,"duration_ms":9654},"pith_extraction":{"msc":["68T50","62F03","68N30"],"pacs":["89.20.Ff","07.05.Tp"],"model":"deepseek/deepseek-v4-flash","headline":"The drydock-eval kit enables fast local evaluation of small coding agents and reveals that most modifications do not improve solve rate.","keywords":["coding agents","evaluation kit","containar sandbox","prompt fidelity","null result","statistical testing","cluster bootstrap","software engineering agents"],"falsifier":"Run a different small language model (e.g., Gemma 4 2B) with the same drydock-eval kit and the same four prompt divergences on the same task set; if the null results replicate, the insensitivity is a property of the evaluation setup, not of E4B specifically. If a large shift appears, then E4B is unusually robust.","tokens_in":47210,"feed_emoji":"🧪","tokens_out":2804,"duration_ms":23774,"temperature":0.7,"pith_summary":"This paper presents drydock-eval, a local evaluation kit for small coding agents (specifically the Gemma 4 E4B model) in the context of the Gemma 4 Developer Agent competition. The kit has three modules: Contain (a secure container runner that passed 200 scripted escape tests), Conform (eight checks that verify a local stack reproduces the hosted scorer's prompts down to token IDs), and Compare (paired statistics with repository-clustered tests and a frozen pass rule). Using this kit, the author conducted extensive experiments: serving measurements on four L4 GPUs, comparisons of six 31B agent configurations (solving 37–44 of 70 tasks with no pairwise difference detected), a pre-registered five-arm case study on 231 tasks measuring four prompt divergences (no solve-rate change detected, largest estimate -2.1 points), and three attempts to improve the E4B baseline through screen-and-confirm programmes (none confirmed). The overwhelming result is null: most changes produce no detectable improvement, and the noise structure—most tasks never solved, few tasks drive variability—makes it hard to see any signal.","feed_headline":"Most coding agent tweaks fail to improve solve rate","feed_subtitle":"Local evaluation kit shows prompt changes and architecture shifts produce no detectable gain across thousands of runs.","key_machinery":"The central object is the drydock-eval kit, composed of three modules: Contan (container runner with no network, seccomp filter, read-only root, unprivileged user, eight hash matches before runs start), Conform (eight automated checks comparing local requests to the scorer path, including role, template rendering, thought handling, and compostion settings, with a mutation suite of 103 drifts and 18 controls verifying the checks), and Compare (paired tests with repositories as clusters, a wild cluster bootstrap-t with Rademacher weights, and a frozen pass rule requiring a gain of at least 3.45 points, one-sided p ≤ 0.025, at least three repositories gain, and no repository loses two task-equivalents). The machinery carries the argument by providing an instrument that—if the comparisons are faithful—allows decising whether a candidate agent is genuinely better than the baseline, and the experiments use it to show that most changes are not.","core_discovery":"The central claim is that the drydock-eval kit provides a fast (≈250 runs per hour on one RTX 5090), safe (network-less containers, 200 passed escape tests), and faithful (token-ID-level comparison against the scorer's configuration) local evaluation that can detect small differences between agent configurations, and that when applied, the largest measured effect of four prompt divergences on the E4B solve rate was -2.1 points (95% interval -4.4 to +0.4, not statistically significant after Holm correction. Three candidate improvements to the E4B agent led their screens but were not confirmed on new tasks (gains -0.35, -0.39, +4.10 points, none passing the pre-registered rule). The paper establishes that developing a small coding agent against a noisy hosted score requires a local evaluator with these properties, and that with such an evaluator most modifications show no detectable effect.","pith_inferences":["The null results on E4B are striking: even large prompt changes (affecting 85–90% of requests) did not shift solve rate detectably. This suggests that for this model on these tasks, the solve rate is not determined by the exact phrasing of the prompt but by the model's inherent ability to fix issues—a ceiling effect that might hold for other small models too.","The fact that the runner's high-severity vectors came through shared host resources—and were found by independent verifiers—highlights that container-based sandboxes for agent evaluation need adversarial testing of the host interface, not just of the container boundary.","The three screen-and-confirm programmes show that a 'sure thing' in a small screen (lead of +6.2 points) can vanish on new tasks; the community might adopt a pre-registered two-stage evaluation as a standard to reduce false positives from selection.","The measured run-to-run noise on fixed tasks (SD ≈ 1.01 points for one arm) and the repository-level clustering (tasks within a repository succeed together) suggest that existing benchmarks reporting point estimates without confidence intervals may be misleading; every evaluation should report clustered intervals."],"forward_implications":[],"supporting_citations":[{"why":"Supplies the census of repeated public draws of the BF5 files, giving the standard deviation of one hosted score (1.56 tasks of 58) used to compare local vs. hosted noise.","marker":"[7]"},{"why":"Provides the technical report of Gemma 4 E4B, the model used in the main experiments.","marker":"[12]"},{"why":"Describes vLLM, the serving engine used by both the scorer and the local kit, whose configuration determines prompt rendering.","marker":"[23]"},{"why":"Povides the Berth task set (579 tasks in competition format) from which all E4B evaluation panels are drawn.","marker":"[33]"},{"why":"The starter notebook that defines the scorer's configuration (template, compostion settings, model name) that Conform checks against.","marker":"[36]"},{"why":"Documents the role change in google-adk when model string contains 'gemma4', which is one of the four prompt divergences measured.","marker":"[35]"},{"why":"Introduces the wild cluster bootstrap-t used in the Compare module's pass rule.","marker":"[4]"},{"why":"Describes SWE-bench, the repository-level coding benchmark that tasks are modelled on, though this paper uses competition-specific tasks.","marker":"[21]"},{"why":"Advocates for clustered inference in model evaluations, which the Compare module implements.","marker":"[28]"}],"fun_headline_variants":["Coding agent tweaks rarely yield gains in local eval","Local eval kit: most agent changes show no detectable gain","Thousands of runs find few agent upgrades matter","Prompt shifts fail to boost small coding agent solve rate"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The similarity between the workstation (single RTX 5090, eight parallel jobs) and the hosted scorer (four L4 GPUs, serial jobs) is sufficient for results to generalize; the paper checks this on 24 tasks and finds no gross difference, but cannot rule out small or task-specific effects, and acknowledges that environment differences remain.","fun_headline_variants_meta":{"raw":{"variants":["Coding agent tweaks rarely yield gains in local eval","Local eval kit: most agent changes show no detectable gain","Thousands of runs find few agent upgrades matter","Prompt shifts fail to boost small coding agent solve rate"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"cost":0.000206388,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000206388,"upstream_inference_prompt_cost":0.000129948,"upstream_inference_completions_cost":7.644e-05},"total_tokens":1406,"prompt_tokens":1133,"completion_tokens":273,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":209}},"tokens_in":1133,"tokens_out":273,"cost_source":"openrouter_usage","duration_ms":3106,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":209,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T14:53:58.339427+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run a different small language model (e.g., Gemma 4 2B) with the same drydock-eval kit and the same four prompt divergences on the same task set; if the null results replicate, the insensitivity is a property of the evaluation setup, not of E4B specifically. If a large shift appears, then E4B is unusually robust.","supporting_citations":[{"cited_title":"Gemma 4 agent: Public config draws","cited_arxiv_id":null,"evidence_quote":"Supplies the census of repeated public draws of the BF5 files, giving the standard deviation of one hosted score (1.56 tasks of 58) used to compare local vs. hosted noise."},{"cited_title":"Gemma 4 Technical Report","cited_arxiv_id":"2607.02770","evidence_quote":"Provides the technical report of Gemma 4 E4B, the model used in the main experiments."},{"cited_title":"Gonzalez, Hao Zhang, and Ion Stoica","cited_arxiv_id":null,"evidence_quote":"Describes vLLM, the serving engine used by both the scorer and the local kit, whose configuration determines prompt rendering."},{"cited_title":"Berth: Harness-compatible, post-cutoff python tasks for offline gemma 4 coding agents","cited_arxiv_id":null,"evidence_quote":"Povides the Berth task set (579 tasks in competition format) from which all E4B evaluation panels are drawn."},{"cited_title":"Getting started - Gemma 4 developer agent","cited_arxiv_id":null,"evidence_quote":"The starter notebook that defines the scorer's configuration (template, compostion settings, model name) that Conform checks against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the role change in google-adk when model string contains 'gemma4', which is one of the four prompt divergences measured."},{"cited_title":"Colin Cameron, Jonah B","cited_arxiv_id":null,"evidence_quote":"Introduces the wild cluster bootstrap-t used in the Compare module's pass rule."},{"cited_title":"A unified approach to global program optimization","cited_arxiv_id":"2310.06770","evidence_quote":"Describes SWE-bench, the repository-level coding benchmark that tasks are modelled on, though this paper uses competition-specific tasks."},{"cited_title":"Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations","cited_arxiv_id":"2411.00640","evidence_quote":"Advocates for clustered inference in model evaluations, which the Compare module implements."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}