Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance

Yang Yang

REVIEW 3 major objections 4 minor 14 references

Output format, not just model skill, decides how often a coding agent succeeds.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:19 UTC pith:L6NIJQHD

load-bearing objection A transparent, well-scoped small experiment making a plausible case that output format is model-specific on one project; the general claim outruns the data and the uncalibrated verifiers are a real but addressable threat. the 3 major comments →

arxiv 2607.21674 v1 pith:L6NIJQHD submitted 2026-07-23 cs.SE

Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance

Yang Yang This is my paper

classification cs.SE

keywords output formatcoding agentsformat-model interactionJSON Patchunified difffull-file generationformat misuseLLM evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the format a coding agent uses to return code changes—full file, JSON Patch, or unified diff—is an active experimental variable, not an implementation detail. In a controlled single-round study with three coding models, three formats, and repeated runs on six tasks, the best format differed by model: one model jumped from 27% success with full-file output to 94% with JSON Patch, another did best with unified diff (66% vs. 35% for full file), and a third showed a smaller but real preference for full-file output. The authors conclude there is no universally optimal format and that output format should be chosen per model and treated as a guardrail. The nonzero evidence comes from a single Python progress-bar library; three other projects produced zero successes across all runs, a boundary the paper discusses as a failure gradient.

Core claim

The central claim is a format×model interaction: whether a coding agent succeeds depends jointly on which model is generating the change and which output format it uses. On the one project where any model produced nonzero success, the three models had opposite best formats—one at 94% with JSON Patch, one at 66% with unified diff, one at 50% with full-file—with effect sizes the paper treats as large at the top end. The paper also documents a distinct failure mechanism, 'format misuse,' in which a model correctly diagnoses a one-line fix but expresses it as a full-file replacement, and uses this to argue that output formats should constrain the scope of edits to the region the model actually l

What carries the argument

The load-bearing instrument is a 3×3 factorial experiment: three models × three output formats, with 20 independent runs per cell on each task, judged by whether the project's test suite passes. Each format goes through a different application pipeline—direct file overwrite for full file, a strict JSON Patch parser for JSON Patch, and the patch utility with fuzz tolerance for unified diff—and the paper measures format effects with chi-square and exact tests plus a standard effect-size statistic. The interaction itself is the central object: the paper does not claim a best format, only that the best format is model-dependent.

Load-bearing premise

The result rests on the assumption that the three verification pipelines are fair, equivalent measures of success; if the JSON Patch pipeline rejects correct output more readily than full-file overwrite or fuzzy patch tolerates broken output, the format×model interaction could be a measurement artifact.

What would settle it

Apply the same set of known-correct fixes in all three formats to the same code and compare pipeline failure rates; if the strict JSON Patch pipeline fails on valid patches significantly more often than the other two pipelines, the reported interaction is not a fair measure of format effects. A second check is to re-run the experiment on a held-out task where all formats plausibly work and see whether each model's best format persists.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Benchmarks that evaluate models with a single output format may report model rankings that would change under another format; the paper suggests format should be disclosed and analyzed as a dimension.
Adopting full-file output as a universal default can leave large, inexpensive gains on the table: the paper's strongest contrast was 27% to 94% for the same model and tasks.
Output format becomes a model-specific hyperparameter: testing a few formats for a new model costs roughly ¥15 in this setup and can identify gains of tens of percentage points.
Tool designers can enforce scope limits—rejecting a patch that edits far beyond the localized region—as a guardrail against format misuse, independent of model or task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

An inference beyond the paper: if the format×model interaction reproduces on other tasks, existing single-format leaderboards may be systematically biased toward models whose strengths match that format; this is a consequence the paper gestures at but does not test.
An inference: the three verification pipelines differ in strictness (strict JSON parsing vs. direct overwrite vs. fuzzy patch), so part of the observed effect could be measurement; the paper acknowledges this, and a control experiment with known-correct patches in each format would settle it.
A testable extension the paper leaves implicit: a dynamic format selector that picks a model's best format per task type could improve average success beyond any fixed format.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A transparent, well-scoped small experiment making a plausible case that output format is model-specific on one project; the general claim outruns the data and the uncalibrated verifiers are a real but addressable threat.

read the letter

The genuinely new thing here is a controlled ablation of output format across multiple models. Nobody in the cited literature did that, and the paper deserves credit for running it with real repetitions, reporting Wilson CIs and Fisher tests, and releasing the data. The within-tqdm numbers are credible as measured under the stated pipelines: Doubao at 94% with JSON Patch versus 27% with full file is a big effect, and DeepSeek's 66% with unified diff is consistent across tasks. The authors are also unusually honest about their limitations—they flag the non-deterministic sampling, the zero-success projects, and the verification-heterogeneity problem in §5.3 and Appendix C. That transparency is real and should count in their favor.

Soft spots, in proportion. First, the central claim is scoped too broadly. The confirmatory project was chosen after exploratory runs showed zero successes in three other projects. That is an adaptive design, and it is fine as an exploratory finding, but the abstract's "no universally optimal format" is supported by exactly one project with non-zero success. The title and abstract should say "on tqdm" or "in single-round tasks where any format succeeds." Second, the stress-test note is on target: the three verification pipelines have very different tolerance levels. Full-file output is extracted by regex and overwritten with no consistency check; JSON Patch must parse with json.loads on the first bracketed block; unified diff goes through patch -p1 --fuzz=3. A single known-correct patch passed through each pipeline would calibrate those tools, and the released data may already contain enough pipeline-failure annotations to test whether verifier strictness inflates the interaction. That analysis is missing and it is load-bearing—if the JSON Patch pipeline rejects valid patches at a higher rate, Doubao's 94% is partly an artifact. Third, they report per-model format effects but no formal format×model interaction test; the claim of an interaction is inferred from the pattern of per-model tests. Fourth, Qwen's effect is small (h=0.29), unadjusted for multiple comparisons, and its confidence interval nearly touches zero. Treat that as suggestive, not strong evidence.

The citation pattern is broadly fine. The self-citations to the author's own GitHub issues are a little unusual but not improper; the related-work discussion engages the benchmarks honestly. The paper ships data and templates, which is more than many benchmark papers do.

Who gets value from this: people building coding-agent pipelines and benchmark designers who care whether output format is a confound. It is a useful exploratory result, not a settled law of agent behavior. It deserves a serious referee, but the referee should ask for the verifier-calibration control, a scoped abstract, and either a formal interaction test or a softer conclusion. If the released data can answer the verifier question, this paper could become a solid empirical note.

Referee Report

3 major / 4 minor

Summary. This paper reports a single-round experiment in which three LLMs (DeepSeek V4, Doubao 2.0 Pro, Qwen 3.7 Max) produce code changes in three output formats (full-file replacement, JSON Patch, unified diff) on six tasks, repeated 20 times per cell. Across four projects, only tqdm yielded any successes; the authors therefore focus confirmatory analyses on tqdm. They report per-model format preferences (Doubao 94% JSON Patch vs 27% full file; DeepSeek 66% unified diff vs 35% full file; Qwen 50% full file vs 36% unified diff) and interpret this as a format×model interaction with no universally optimal format. They also identify "format misuse" as a failure mechanism and propose constraining output format semantics to the agent's own localization step. Data, templates, and parsing strategies are released.

Significance. If the headline effects are real, this is a useful empirical contribution: output format is commonly treated as an implementation detail, and the reported effect sizes are large enough to reorder model rankings. The manuscript has genuine strengths: all 4,013 runs and templates are released, Wilson CIs are reported, per-cell sample sizes are equalized at n=120 for the main analyses, and the limitations section is unusually candid about verification heterogeneity. However, the result is currently vulnerable on two fronts: the format comparisons use different, uncalibrated verification pipelines, and the central "interaction" claim lacks a formal interaction test. The single-project confirmatory basis further limits the scope of the abstract's generalization.

major comments (3)

[§5.3, Appendix A.6, Table 3] The central format×model interaction is measured through three format-specific verification pipelines whose failure modes are unquantified. As §5.3 concedes, full-file uses regex extraction plus direct overwrite with no patch-consistency check, json_patch requires json.loads to succeed on the first bracketed block with zero syntax tolerance, and unified_diff uses patch -p1 --fuzz=3, which is sensitive to line endings and context. No control experiment measures each pipeline's failure rate on known-correct patches. If, for example, the JSON Patch pipeline rejects syntactically valid output at a nontrivial rate while the full-file pipeline accepts any code block, Doubao's 94% vs 27% gap in Table 3 could be inflated by parser strictness rather than by the format's intrinsic suitability. Since this interaction is the paper's sole positive finding, the released data should be reanalyzed for p
[§4.1, Table 3] The claim of a "format×model interaction" is supported only by within-model χ² tests, one per row of Table 3. These tests establish that each model has a format preference, but they do not test whether the preferences differ significantly across models. The data could in principle be consistent with additive model and format main effects. A formal interaction test (e.g., logistic regression with model×format terms, or a permutation test on differences between model-specific effect sizes) is needed to substantiate the word "interaction." Given that the abstract, §5.1, and the conclusion repeatedly emphasize interaction, this is a load-bearing statistical gap.
[§3.1, §4.1, Abstract] All confirmatory statistics come from one project (tqdm), selected after exploratory runs showed zero successes on dotenv, requests, and jsoup. The abstract's global "no universally optimal format" wording overgeneralizes from a single-project result. The paper does state this limitation in §4.1 and Table 7, but the abstract and conclusion present the finding without this boundary. The abstract should condition the claim on tqdm, and the discussion should explicitly acknowledge that format×model interactions may be project-dependent. This is not a demand for more projects, but for an accurate scope statement.

minor comments (4)

[§3.1 vs §3.3] The power statements are inconsistent: §3.1 says the design can detect proportion differences ≥0.30 with 80% power, while §3.3 says 20 repeats provide >80% power for h≥0.38. Please reconcile or state the basis for each claim.
[Table 1] The tqdm row reports 1,462 runs and 61% success, while the core analysis in §4.1 uses 1,080 runs (n=120 per cell). The first-N deduplication rule is explained, but the table should make the discrepancy explicit. Also, the footnote about runs terminated by API billing interruptions should report those counts by project and model; if a substantial share of the non-tqdm zero-success runs are billing failures, the cross-project boundary claim is weaker.
[§5.5] The statement that Qwen's effect has a confidence interval whose "lower bound approaches zero" is not supported in the text: no confidence interval for Cohen's h is reported. Please provide the CI or clarify which interval is meant.
[Footnote 1, §A.5] The footnote says the API "returned up to 33,255 tokens" even though max tokens was set to 32,768. Clarify whether this is output tokens or total input+output tokens; as written it appears internally inconsistent.

Circularity Check

0 steps flagged

No circularity: the central format×model interaction is an observed empirical result derived from reported success counts, not from fitted parameters, self-citations, or definitional identities.

full rationale

The paper's central claim—a format×model interaction with no universally optimal format—rests on the success counts in Table 3 (e.g., Doubao 113/120 with JSON Patch, DeepSeek 79/120 with unified diff, Qwen 60/120 with full file). These are directly observed experimental outcomes, not outputs of a fitted model or a derivation that presupposes the conclusion. The statistical tests (χ², Fisher's exact test, Cohen's h) compare observed proportions; they do not estimate parameters that are then renamed as predictions. The 'model-specific output strategy' recommendation is an explicit post-hoc reading of the best-performing cell in the same experiment, but the paper labels these recommendations as 'directional, derived from controlled single-round experiments' and does not present them as an independent prediction. References [13] and [14] are self-citations by the same author, but they appear only in the 'Broader Vision' paragraph as companion work; they are not used to justify the format×model interaction or to rule out alternatives. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result is load-bearing. The verification-pipeline heterogeneity concern (Appendix A.6, §5.3) is a threat to internal validity and generalizability, but it is not circularity: even if the pipelines bias the counts, the headline numbers are still the measured outcomes, not quantities constructed to equal the paper's inputs. Therefore no circular step can be identified from the text, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters are fitted; the paper reports observed frequencies with standard statistics. The central claim rests on four domain assumptions: test-pass as success, single-project representativeness, verification fairness, and trial independence. No new entities are introduced.

axioms (4)

domain assumption pytest exit code 0 is an adequate success criterion for a coding-agent task
The central claim counts a run as success iff the specified pytest/mvn suite passes (Appendix A.8); the paper itself notes pytest pass is necessary but not sufficient for correctness.
domain assumption The six tqdm tasks are representative enough to support a general format×model interaction claim
All confirmatory statistics come from six tasks in one project; the abstract makes a general claim ("no universally optimal format") while zero successes on three other projects bound generalization.
domain assumption Format-specific verification pipelines are fair and do not systematically distort success rates
Full file, JSON Patch, and unified diff are applied by different tools with different tolerance (§5.3, A.6); no control experiment quantifies parser-induced failures on known-correct outputs.
domain assumption 20 repeated runs at API-default temperature are conditionally independent despite prompt caching
§3.3 uses stochastic sampling and 20 repetitions for statistics; §5.6 acknowledges DeepSeek's 99% cache-hit rate could weakly couple repeat outcomes.

pith-pipeline@v1.3.0-alltime-deepseek · 9851 in / 12766 out tokens · 117870 ms · 2026-08-01T08:19:35.219097+00:00 · methodology

0 comments

read the original abstract

Output format is not a neutral implementation detail -- it can reorder model rankings, amplify or suppress individual model differences, and determine whether a coding agent succeeds or fails. We conducted a controlled single-round experiment with 3 models (DeepSeek V4, Doubao 2.0 Pro, Qwen 3.7 Max) x 3 output formats (full file, JSON Patch, unified diff) x 6 tasks x 20 repetitions, totaling 4,013 runs across 4 open-source projects. Only one project (tqdm) yielded non-zero success rates: dotenv, requests, and jsoup yielded zero successes in 2,551 runs. Our central finding is a format x model interaction with no universally optimal format. Doubao achieves 94% success with JSON Patch (Cohen's h = 1.57, p < 0.001), DeepSeek excels at unified diff (66%, h = 0.63), and Qwen shows a small but significant full-file preference (50%, h = 0.29, p < 0.05). Beyond these headline results, we identify a distinct failure mechanism -- format misuse -- where agents correctly diagnose a problem but execute it with excessive scope, most vividly when a one-line fix is applied as a full-file replacement. We propose a model-specific output strategy, a tool-design principle that constrains format semantics to the agent's own localization step, and release all data and templates for reproducibility. All experimental data are available at https://doi.org/10.5281/zenodo.21505157.

Figures

Figures reproduced from arXiv: 2607.21674 by Yang Yang.

**Figure 2.** Figure 2: Cross-project success rates (all models combined). tqdm vs others: Fisher’s exact test, [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: DeepSeek V4: Task × Format Success Rate (n = 20 per cell) [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages

[1]

arXiv:2509.25370, 2025

Where LLM agents fail and how they can learn from failures. arXiv:2509.25370, 2025

arXiv 2025
[2]

arXiv:2607.18754, 2026

AgentDebugX: Execution-grounded debugging for code agents via test-driven repair. arXiv:2607.18754, 2026

Pith/arXiv arXiv 2026
[3]

arXiv:2604.08224, 2026

Externalization in LLM agents: A unified review of memory, skills, protocols and harness engineering. arXiv:2604.08224, 2026

Pith/arXiv arXiv 2026
[4]

Inference economics of enterprise coding agents: A case study of cloud vs

Peng et al. Inference economics of enterprise coding agents: A case study of cloud vs. on-premise LLMs. arXiv:2607.13080, 2026

Pith/arXiv arXiv 2026
[5]

CRITIC: Large language models can self-correct with tool-interactive critiquing

Zhibin Gou et al. CRITIC: Large language models can self-correct with tool-interactive critiquing. InProceedings of ICLR, 2024

2024
[6]

LiveCodeBench: Holistic and contamination free evaluation of large language models for code

Naman Jain et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv:2403.07974, 2024

Pith/arXiv arXiv 2024
[7]

Jimenez, John Yang, et al

Carlos E. Jimenez, John Yang, et al. SWE-bench: Can language models resolve real-world GitHub issues? InProceedings of ICLR, 2024

2024
[8]

Ed. P. Bryan. JavaScript Object Notation (JSON) Patch. RFC 6902, IETF, 2013

2013
[9]

OpenHands: An open platform for AI software developers as generalist agents

Xingyao Wang et al. OpenHands: An open platform for AI software developers as generalist agents. InProceedings of ICLR, 2025

2025
[10]

Willard et al

Brandon T. Willard et al. Efficient guided generation for large language models. arXiv:2307.09702, 2023

Pith/arXiv arXiv 2023
[11]

Agentless: Demystifying llm-based software engineering agents

Steven Xia et al. Agentless: Demystifying llm-based software engineering agents. arXiv:2407.01489, 2024

Pith/arXiv arXiv 2024
[12]

Jimenez, et al

John Yang, Carlos E. Jimenez, et al. SWE-agent: Agent-computer interfaces enable auto- mated software engineering. arXiv:2405.15793, 2024

Pith/arXiv arXiv 2024
[13]

Append-only coding: Prefix stability through diff-based agent output (Her- mes #66127).https://github.com/NousResearch/hermes-agent/issues/66127, 2026

Yang Yang. Append-only coding: Prefix stability through diff-based agent output (Her- mes #66127).https://github.com/NousResearch/hermes-agent/issues/66127, 2026. Zenodo:https://doi.org/10.5281/zenodo.21492628

work page doi:10.5281/zenodo.21492628 2026
[14]

Progressive system prompt disclosure (Hermes #64103).https://github

Yang Yang. Progressive system prompt disclosure (Hermes #64103).https://github. com/NousResearch/hermes-agent/issues/64103, 2026. Zenodo:https://doi.org/10. 5281/zenodo.21492624. 12 A Prompt Templates and Parsing Strategy A.1 System Instruction (Fixed) All three models used identical prompts. Only theOutput Formatsection varied. You are a coding agent. Mo...

2026

[1] [1]

arXiv:2509.25370, 2025

Where LLM agents fail and how they can learn from failures. arXiv:2509.25370, 2025

arXiv 2025

[2] [2]

arXiv:2607.18754, 2026

AgentDebugX: Execution-grounded debugging for code agents via test-driven repair. arXiv:2607.18754, 2026

Pith/arXiv arXiv 2026

[3] [3]

arXiv:2604.08224, 2026

Externalization in LLM agents: A unified review of memory, skills, protocols and harness engineering. arXiv:2604.08224, 2026

Pith/arXiv arXiv 2026

[4] [4]

Inference economics of enterprise coding agents: A case study of cloud vs

Peng et al. Inference economics of enterprise coding agents: A case study of cloud vs. on-premise LLMs. arXiv:2607.13080, 2026

Pith/arXiv arXiv 2026

[5] [5]

CRITIC: Large language models can self-correct with tool-interactive critiquing

Zhibin Gou et al. CRITIC: Large language models can self-correct with tool-interactive critiquing. InProceedings of ICLR, 2024

2024

[6] [6]

LiveCodeBench: Holistic and contamination free evaluation of large language models for code

Naman Jain et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv:2403.07974, 2024

Pith/arXiv arXiv 2024

[7] [7]

Jimenez, John Yang, et al

Carlos E. Jimenez, John Yang, et al. SWE-bench: Can language models resolve real-world GitHub issues? InProceedings of ICLR, 2024

2024

[8] [8]

Ed. P. Bryan. JavaScript Object Notation (JSON) Patch. RFC 6902, IETF, 2013

2013

[9] [9]

OpenHands: An open platform for AI software developers as generalist agents

Xingyao Wang et al. OpenHands: An open platform for AI software developers as generalist agents. InProceedings of ICLR, 2025

2025

[10] [10]

Willard et al

Brandon T. Willard et al. Efficient guided generation for large language models. arXiv:2307.09702, 2023

Pith/arXiv arXiv 2023

[11] [11]

Agentless: Demystifying llm-based software engineering agents

Steven Xia et al. Agentless: Demystifying llm-based software engineering agents. arXiv:2407.01489, 2024

Pith/arXiv arXiv 2024

[12] [12]

Jimenez, et al

John Yang, Carlos E. Jimenez, et al. SWE-agent: Agent-computer interfaces enable auto- mated software engineering. arXiv:2405.15793, 2024

Pith/arXiv arXiv 2024

[13] [13]

Append-only coding: Prefix stability through diff-based agent output (Her- mes #66127).https://github.com/NousResearch/hermes-agent/issues/66127, 2026

Yang Yang. Append-only coding: Prefix stability through diff-based agent output (Her- mes #66127).https://github.com/NousResearch/hermes-agent/issues/66127, 2026. Zenodo:https://doi.org/10.5281/zenodo.21492628

work page doi:10.5281/zenodo.21492628 2026

[14] [14]

Progressive system prompt disclosure (Hermes #64103).https://github

Yang Yang. Progressive system prompt disclosure (Hermes #64103).https://github. com/NousResearch/hermes-agent/issues/64103, 2026. Zenodo:https://doi.org/10. 5281/zenodo.21492624. 12 A Prompt Templates and Parsing Strategy A.1 System Instruction (Fixed) All three models used identical prompts. Only theOutput Formatsection varied. You are a coding agent. Mo...

2026