{"id":"299a448c-0b12-42e7-817b-8d343d94ff5c","arxiv_id":"2607.13080","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 56-day single-developer case study finds prompt caching makes the cloud agent cheaper per token than quantized on-premise serving, while the on-premise agent is associated with 2.6–4.9× higher defect-repair odds within every difficulty tier (OR_MH = 3.61).","lead":"Over 56 days, one developer ran a cloud AI coding agent for a month and an on-premise open-weights agent for the next month on the same production codebase, comparing costs, code quality, and developer effort. Prompt caching made the cloud API cheaper per token than the on-premise setup, but the on-premise agent produced far more defect-repair commits — the central trade-off for enterprise buyers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Defect-repair premium rests entirely on single-author commit-message regex; Period B's smaller, more numerous 'fix:' commits could manufacture the FCR gap. A blind diff-based oracle is needed.","rationale":"The paper is internally consistent and unusually careful: the cost arithmetic checks out ($8,785.21 API bill, $0.573/M effective, 88.6% cache saving, 40.1% shared-TCO saving), the sensitivity sweep is honest, and the authors repeatedly scope claims to the deployable bundle rather than to a single factor. The weakest point is not arithmetic or internal logic—it is the validity of the sole measurement instrument behind the quality headline. FCR, the defect taxonomy, the per-tier odds ratios, the debugging-spiral share, and the per-language FCR all reduce to regex matches on commit messages written by the single experimenter during two different agent-harness periods. The observed granularity shift (more, smaller commits in Period B, with many more 'fix:'-prefixed messages) makes label or packaging contamination a concrete, not merely hypothetical, threat. The paper's defenses are real but circular in an important sense: the same labels feed the taxonomy, the tier stratification, and the language breakdown, so none of those can independently confirm the labels. The Redmine cross-check only validates attribution, not defect semantics, and the CI gate constrains code quality, not message wording. Because the reader's verdict is already CONDITIONAL and names essentially this assumption as the reason, my stress-test does not move the verdict; it sharpens the condition: the quality claim should be reported as unverified until a blind, diff-based (or revert/issue-link-based) labeling protocol is run. This is an addressable concern, not a fatal one, and it does not undermine the cost-economics or developer-experience results, which derive from telemetry and timestamps rather than from fix labels.","tokens_in":28693,"tokens_out":5214,"duration_ms":52631,"concrete_test":"Have the authors run, on the confidential repository, a blind second-rater protocol over all 613 non-merge commits: a rater with no knowledge of period or hypothesis labels each commit as fix/non-fix using only the code diff and, where present, the linked Redmine context, with commit messages redacted. Compute Cohen's kappa between this diff-based oracle and the paper's regex labels, then recompute the Mantel–Haenszel OR and per-tier ORs using oracle labels only. If the oracle OR_MH falls materially below 2.0 or kappa is below 0.6, the defect premium should be treated as unverified; if the oracle reproduces OR_MH≈3.6 with CIs excluding 1, the label threat is bounded and the quality claim is provisionally supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quality claim—FCR 74.9% vs 45.9%, OR_MH=3.61, per-tier 2.6–4.9×—is produced entirely by applying /(fix|bug|issue|patch|correct)/i to commit messages (Eq. 1, §IV-C). The messages were authored by the sole, hypothesis-aware developer, and the first author manually reviewed every label without blinding (§IV-C). The paper's own data show Period B is not packaging-invariant: 367 vs 246 non-merge commits, 18.35 vs 11.18 commits/active day, median insertions/commit 25 vs 36 (Table X, p≈0.066/0.068), and 255 vs 101 'fix:'-prefixed commits (Table IV). A switch to Opencode/GLM—a different agent harness that generates Conventional-Commits-style messages and may commit at finer granularity—could inflate fix-commit counts without any change in underlying defect rate. The responses in §VII-C (identical CI gate, taxonomy shift, per-tier and per-language persistence) all reuse the same regex labels, so they cannot validate the labels; Redmine cross-validation checks attribution, not fix semantics; the CI gate is symmetric but does not constrain message wording or commit packaging. There is no independent defect oracle, and the authors explicitly acknowledge this (§VII-C). If commit-message behavior shifted between periods, every downstream quantity—FCR, per-tier ORs, debugging-spiral share, Table IX taxonomy—is jointly contaminated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a single-developer, non-randomized, 56-day longitudinal case study comparing two deployable coding-agent configurations on a production monorepo: Period A uses Claude Code with the Claude Opus 4.7/4.8 API; Period B uses Opencode with on-premise NVFP4-quantized GLM-5.1/5.2 served by vLLM on a Blackwell 4x B200 slice. The paper claims three main results: (1) prompt caching reduces the realized API cost to an effective $0.57 per million tokens, below the shared on-premise amortized cost of $2.83/M, although the API total bill is larger; (2) the local configuration is associated with a much higher defect-repair burden, with Fix Commit Ratio 74.9% vs. 45.9% and Mantel-Haenszel odds ratio 3.61 across difficulty tiers; (3) under Taiwan-market parameters and a symmetric labor model, shared on-premise allocation saves 40.1% of true TCO while dedicated reservation costs 43.8% more, and an offline replay of routing policies yields a monotonic cost-quality frontier rather than a dominating hybrid policy. The methodology is transparent: deterministic keyword rules, Git patch-id deduplication, Redmine cross-validation, and a fully specified cost model. The paper explicitly scopes its claims and acknowledges the absence of an independent defect oracle.","tokens_in":28963,"tokens_out":9220,"duration_ms":86064,"significance":"If the results hold, the paper makes two practically important contributions. First, it documents a concrete mechanism by which prompt caching inverts the per-token cost comparison between a frontier API and a self-hosted quantized cluster, with a detailed, reproducible cost accounting. Second, it provides an unusually complete longitudinal measurement of the defect-repair burden of an on-premise quantized agent relative to a cloud API, including per-tier stratification, per-language analysis, behavioral workload indicators, and a hybrid-routing counterfactual. The paper's methodological transparency is a genuine strength: the classification rules, telemetry extraction details, cost parameters, and bootstrap procedure are specified in enough detail to be reproduced on other repositories. The central caveat is that the quality comparison rests on a single measurement instrument, the commit-message keyword classifier, and the paper itself acknowledges that no independent defect oracle exists. Given the single-subject design and proprietary data, the empirical claims should be read as an exploratory case study; the statistical tests quantify within-developer consistency rather than po","major_comments":[{"comment":"The defect premium (FCR 74.9% vs. 45.9%; OR_MH=3.61) rests entirely on a keyword regex applied to commit messages authored by the sole, hypothesis-aware developer. Period B is not packaging-invariant by the paper's own data (Table IV: 255 vs. 101 fix:-prefixed; Table X: 367 vs. 246 commits, 11.18 vs. 18.35 commits/active day, median insertions/commit 25 vs. 36, p≈0.068). A switch to Opencode, which emits Conventional-Commits-style messages and may commit at finer granularity, could inflate fix counts without changing defect rates. The §VII-C defenses (CI gate, taxonomy, per-tier/per-language) reuse the same labels and cannot validate them; Redmine checks attribution, not fix semantics; no independent oracle exists (acknowledged in §VII-C). Because per-tier ORs, Table IX, §V-G, §VI-A, and §VI-C all derive from these labels, this is load-bearing. Required: a blind diff-based or issue-track","section":"§IV-C, Eq. (1)"},{"comment":"The offline replay treats the same regex-derived per-tier fix rates as transportable backend properties and pools commits across periods. The paper acknowledges the exchangeability assumption, but the bootstrap propagates only sampling variance, not label bias. Because these per-tier rates are exactly what the packaging confound of §IV-C can shift, the frontier ordering, the 'no dominating policy' claim, and all four TCO/FCR comparisons are contingent on label validity. Please re-run the replay under a conservative FCR band (or oracle-based rates) and show whether the frontier ordering survives.","section":"§VI-C, Table XIV"},{"comment":"Difficulty is derived from change footprint, and fix commits are systematically smaller (median 25 vs. 36 lines), mechanically enriching the Low tier with fixes, as the paper notes. This is a fair between-period contrast only if commit granularity is period-invariant; the data show it is not (more, smaller commits in Period B). Please report per-tier odds ratios with a packaging-invariant difficulty measure (e.g., normalized code-only change size or pre-spec complexity), or restrict to commits above a minimum size.","section":"§V-C, Table X / Fig. 5"}],"minor_comments":[{"comment":"The distinction between the fix:-prefix count and the regex-derived FCR count is easy to miss. Consider adding a short cross-tabulation of the two definitions and explicitly stating which count is used in each analysis (Table IX, §V-C, §VI-A).","section":"Table IV footnote"},{"comment":"The median insertions/commit contrast is reported with p=0.068 and Cliff's delta +0.087 (negligible). Please state the unit of analysis for this test and discuss why a negligible effect size accompanies a p-value close to 0.05.","section":"§V-C, Table X"},{"comment":"Equation (5) reuses the symbol TC from Eq. (2) with a different meaning (total cost vs. per-task cost). Rename one of them (e.g., TC_total and TC_task) to avoid confusion.","section":"§VI-A, Eq. (5)"},{"comment":"The cumulative net-line plot would benefit from annotated vertical lines for the mid-period model upgrades (Opus 4.8 on 2026-05-28; GLM-5.2 on 2026-06-13), since the text argues there is no structural break at these transitions.","section":"Fig. 3"},{"comment":"The statement that the cached API is cheaper per token than the shared on-premise amortization is utilization-dependent, as the paper later clarifies. Consider marking this with an asterisk at the first occurrence in the abstract so it cannot be read as a universal claim.","section":"Abstract / §V-D"},{"comment":"The recommendation of a blind second-rater protocol is welcome. Since the authors already have full Git history, a practical interim step would be to report inter-rater agreement on a sample of commit messages or to provide a small labeled sample in an appendix for independent verification.","section":"§VII-C, FCR validity"}],"recommendation":"major_revision","confidential_remarks":"This is a single-developer self-study on a proprietary codebase, with no data or code release. The manuscript is unusually transparent about its limitations, and the cost arithmetic is carefully presented. However, the central quality claim depends on a commit-message regex applied by a hypothesis-aware sole author, and the paper explicitly acknowledges the absence of an independent defect oracle. That gap is fixable within the manuscript's scope, but only with additional analysis; hence major revision rather than rejection. Editors should weigh whether the n=1 design and proprietary data fit the journal's expectations for empirical software-engineering contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: a serious, unusually honest case study, but the headline quality result — FCR 74.9% vs 45.9%, MH OR 3.61 — rests entirely on one developer's commit-message labels, and the paper's own data show the two periods weren't packaging-invariant. That doesn't kill the paper; the cost side is measured telemetry and the main confounds run conservative. But it should keep the quality claim conditional.\n\nWhat's actually new: the deployment-level finding that prompt caching (99.3% hit rate) inverts the per-token cost comparison — effective $0.57/M vs $2.83/M for the shared on-prem slice — and a difficulty-stratified defect analysis on production agentic workload. The TCO model under Taiwan parameters is transparent, the arithmetic checks out, and the offline hybrid-routing replay is a useful addition. The paper is also unusually disciplined in scoping: it explicitly treats the comparison as the deployable bundle, not any single factor, and it flags most of its own threats in Section VII.\n\nThe soft spot, in proportion: every downstream quality number comes from applying /(fix|bug|issue|patch|correct)/i to commit messages authored by the hypothesis-aware sole developer. Period B had more and smaller commits (367 vs 246; median insertions 25 vs 36) and far more 'fix:'-prefixed subjects (255 vs 101). A switch to a harness that generates Conventional-Commits-style messages could inflate fix counts without any change in underlying defect rate. The paper's defenses — identical CI gate, taxonomy shift, per-tier and per-language persistence — all reuse the same labels, so they can't validate the labels. The Redmine cross-check validates attribution, not fix semantics. The authors acknowledge there is no independent defect oracle. That is a real, load-bearing limitation. It is not fatal to the cost claims, which stand on telemetry, but it means the 3.6× repair premium is a well-formed hypothesis awaiting a blind diff-based oracle, not a settled measurement.\n\nMinor: GLM-5.1/5.2 are pooled on an untested 'no structural break' assertion, and no data or code ships, so none of the magnitudes can be independently double-checked. Both are addressable.\n\nWho this is for: enterprise architects choosing between API and on-prem coding agents, and researchers working on LLM cost/quality trade-offs. It deserves a serious referee — it is credible, well-scoped, and the quality claim, while fragile, is presented honestly enough to test. I'd send it to peer review with a request for a blind inter-rater protocol and ideally a multi-developer crossover.\n\nI'd cite the cost-inversion result, cautiously, and I'd bring the paper to a reading group.","headline":"A careful, honest single-developer case study with a real cost surprise and a defect-gap result that is plausible but probably inflated — the entire quality comparison hangs on one developer's commit-message labels.","tokens_in":29597,"tokens_out":3280,"would_cite":true,"duration_ms":26831,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a 56-day enterprise coding-agent comparison, prompt caching made a cloud API cheaper per token than a shared on-premise model, but the on-premise model carried roughly 3.6x higher defect-repair odds.","keywords":["coding agents","inference economics","prompt caching","total cost of ownership","fix commit ratio","LLM quantization","on-premise inference","defect repair"],"falsifier":"Strip the period labels from the 613 commits and have a blind second rater (or an independent oracle, such as issue-tracker-linked bug fixes) reclassify each commit as repair or feature; if the stratified odds ratio collapses toward 1 - or if a crossover with the two configurations in reversed period order shows no fix-commit gap - the paper's central defect-premium claim would be falsified.","tokens_in":28460,"feed_emoji":"⚙️","tokens_out":12120,"duration_ms":99264,"temperature":0.7,"pith_summary":"This paper tries to establish that the real economics of enterprise coding agents are not what nominal price sheets suggest. Using one developer and a single production codebase over two contiguous 28-day periods, it argues that prompt caching collapses the unit cost of a frontier cloud API to $0.57 per million processed tokens, below the $2.83 amortized per-token cost of a shared quantized on-premise cluster. The local configuration carries a far heavier defect-repair burden: a 74.9% fix-commit ratio versus 45.9%, with the odds of a commit being a repair 2.6-4.9 times higher within every difficulty tier (stratified odds ratio 3.61). Under shared GPU allocation the on-premise configuration still saves about 40% of true total cost of ownership, while dedicated reservation costs about 44% more than the cached API, so the genuine cost of going local is a measurable developer-experience burden rather than a dollar penalty. A reader should care because the result reframes cloud-versus-on-premise adoption decisions and maps a cost-quality frontier for routing work between the two backends.","feed_headline":"3.6x repair odds for on-premise coding agent, at 40% lower cost","feed_subtitle":"On-premise agent hit 74.9% fix commits vs 45.9% for the cloud API, yet shared-GPU total cost ran 40% lower.","key_machinery":"Two instruments carry the argument. The first is the Fix Commit Ratio (FCR), a deterministic keyword rule applied to commit messages that labels each commit as a repair or not; per-tier odds ratios compare FCR between the two periods within change-footprint difficulty strata, isolating the defect premium from task-mix differences. The second is a cost model that prices the API with cache-read discounts and attributes on-premise GPU cost through two rental scenarios - a dedicated 24/7 reservation and a shared monthly seat - which converts time-billed hardware into per-token and per-commit figures. A two-stage bootstrap stabilizes the offline routing replay that produces the cost-quality front","core_discovery":"The central discovery is a pair of linked inversions. With a 99.3% prompt-cache hit rate, the realized API cost fell 88.6% to $0.57 per million processed tokens, undercutting the $2.83 amortized unit cost of the shared on-premise slice; the API's total bill is larger only because the agent loop transmits about 17x more tokens. Meanwhile, at comparable gross code churn, the local configuration produced a defect-repair premium that survives stratification: 74.9% of its commits were repairs versus 45.9%, and the odds of a commit being a repair were 2.6-4.9x higher within every difficulty tier (stratified odds ratio 3.61). The paper reads these together as evidence that the cheaper-on-paper loca","pith_inferences":["If the 99.3% cache hit rate is typical of agentic development loops, then nominal API price sheets systematically overstate cloud costs for interactive coding agents; the per-token cost inversion likely weakens for batch workloads or fresh-context tasks with low cache reuse.","The observed bundle conflates base-model capability, quantization, and agent harness, so a natural test is an ablation that swaps one factor at a time (e.g., the same harness on both backends); the paper's design cannot attribute the repair premium to any single factor.","A multi-developer crossover study would reveal whether the repair premium and the 40% savings figure survive variation in skill, codebase, and period ordering; until then both should be read as single-site estimates.","The cost-quality frontier implies enterprises can treat defect rate as a tunable budget: routing only high-difficulty work to the cloud captures meaningful savings with modest FCR penalty, but each further shift to the local model trades quality for infrastructure dollars."],"forward_implications":["The cached frontier API is the cheaper option per processed token ($0.57 vs $2.83 per million), even though its total bill is larger because it transmits roughly 17x more tokens.","The local configuration's defect-repair premium is not a task-mix artifact: the odds of a commit being a repair were 2.6-4.9x higher within every difficulty tier.","Under shared GPU allocation, on-premise deployment lowers true total cost of ownership by about 40%; dedicated 24/7 reservation raises it by about 44% relative to the cached API.","No hybrid routing policy dominates the pure-API baseline: each tier shifted to the local model buys infrastructure savings at the price of a higher fix-commit ratio, tracing a monotonic cost-quality frontier.","The local configuration's developer-experience cost is objectively measurable: debugging-spiral share nearly doubles, the longest repair run grows from 18 to 58 commits, and median commit cadence slows from 5.9 to 12.7 minutes."],"fun_headline_variants":["On-prem LLMs: 40% TCO savings, 3.6x repair odds","Local coding agent costs 40% less but 3.6x more fixes","On-prem agent: 40% cheaper TCO, 3.6x repair risk","Local LLMs save 40% on TCO but triple defect fixes","Cloud vs on-prem agent: caching flips token cost, repairs 3.6x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That a deterministic keyword rule applied to commit messages written by the one developer labels repairs identically across both periods; if the developer packaged or phrased commits differently in the second period, the 45.9%-versus-74.9% gap could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["On-prem LLMs: 40% TCO savings, 3.6x repair odds","Local coding agent costs 40% less but 3.6x more fixes","On-prem agent: 40% cheaper TCO, 3.6x repair risk","Local LLMs save 40% on TCO but triple defect fixes","Cloud vs on-prem agent: caching flips token cost, repairs 3.6x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001066,"raw_usage":{"total_tokens":4412,"prompt_tokens":962,"completion_tokens":3450,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":3349}},"tokens_in":706,"tokens_out":3450,"duration_ms":22118,"temperature":1.0,"reasoning_tokens":3349,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T06:58:32.726706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Strip the period labels from the 613 commits and have a blind second rater (or an independent oracle, such as issue-tracker-linked bug fixes) reclassify each commit as repair or feature; if the stratified odds ratio collapses toward 1 - or if a crossover with the two configurations in reversed period order shows no fix-commit gap - the paper's central defect-premium claim would be falsified.","supporting_citations":[],"review_version":1}