Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

In a 56-day enterprise coding-agent comparison, prompt caching made a cloud API cheaper per token than a shared on-premise model, but the on-premise model carried roughly 3.6x higher defect-repair odds.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:58 UTC pith:PSH7ESST

load-bearing objection A careful, honest single-developer case study with a real cost surprise and a defect-gap result that is plausible but probably inflated — the entire quality comparison hangs on one developer's commit-message labels. the 3 major comments →

arxiv 2607.13080 v1 pith:PSH7ESST submitted 2026-07-13 cs.SE cs.AI

Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs

Sheng-Wei Peng , Yi-Hsun Lin , Yi-Pei Lee This is my paper
classification cs.SE cs.AI
keywords coding agentsinference economicsprompt cachingtotal cost of ownershipfix commit ratioLLM quantizationon-premise inferencedefect repair
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the real economics of enterprise coding agents are not what nominal price sheets suggest. Using one developer and a single production codebase over two contiguous 28-day periods, it argues that prompt caching collapses the unit cost of a frontier cloud API to $0.57 per million processed tokens, below the $2.83 amortized per-token cost of a shared quantized on-premise cluster. The local configuration carries a far heavier defect-repair burden: a 74.9% fix-commit ratio versus 45.9%, with the odds of a commit being a repair 2.6-4.9 times higher within every difficulty tier (stratified odds ratio 3.61). Under shared GPU allocation the on-premise configuration still saves about 40% of true total cost of ownership, while dedicated reservation costs about 44% more than the cached API, so the genuine cost of going local is a measurable developer-experience burden rather than a dollar penalty. A reader should care because the result reframes cloud-versus-on-premise adoption decisions and maps a cost-quality frontier for routing work between the two backends.

Core claim

The central discovery is a pair of linked inversions. With a 99.3% prompt-cache hit rate, the realized API cost fell 88.6% to $0.57 per million processed tokens, undercutting the $2.83 amortized unit cost of the shared on-premise slice; the API's total bill is larger only because the agent loop transmits about 17x more tokens. Meanwhile, at comparable gross code churn, the local configuration produced a defect-repair premium that survives stratification: 74.9% of its commits were repairs versus 45.9%, and the odds of a commit being a repair were 2.6-4.9x higher within every difficulty tier (stratified odds ratio 3.61). The paper reads these together as evidence that the cheaper-on-paper loca

What carries the argument

Two instruments carry the argument. The first is the Fix Commit Ratio (FCR), a deterministic keyword rule applied to commit messages that labels each commit as a repair or not; per-tier odds ratios compare FCR between the two periods within change-footprint difficulty strata, isolating the defect premium from task-mix differences. The second is a cost model that prices the API with cache-read discounts and attributes on-premise GPU cost through two rental scenarios - a dedicated 24/7 reservation and a shared monthly seat - which converts time-billed hardware into per-token and per-commit figures. A two-stage bootstrap stabilizes the offline routing replay that produces the cost-quality front

Load-bearing premise

That a deterministic keyword rule applied to commit messages written by the one developer labels repairs identically across both periods; if the developer packaged or phrased commits differently in the second period, the 45.9%-versus-74.9% gap could be inflated.

What would settle it

Strip the period labels from the 613 commits and have a blind second rater (or an independent oracle, such as issue-tracker-linked bug fixes) reclassify each commit as repair or feature; if the stratified odds ratio collapses toward 1 - or if a crossover with the two configurations in reversed period order shows no fix-commit gap - the paper's central defect-premium claim would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The cached frontier API is the cheaper option per processed token ($0.57 vs $2.83 per million), even though its total bill is larger because it transmits roughly 17x more tokens.
  • The local configuration's defect-repair premium is not a task-mix artifact: the odds of a commit being a repair were 2.6-4.9x higher within every difficulty tier.
  • Under shared GPU allocation, on-premise deployment lowers true total cost of ownership by about 40%; dedicated 24/7 reservation raises it by about 44% relative to the cached API.
  • No hybrid routing policy dominates the pure-API baseline: each tier shifted to the local model buys infrastructure savings at the price of a higher fix-commit ratio, tracing a monotonic cost-quality frontier.
  • The local configuration's developer-experience cost is objectively measurable: debugging-spiral share nearly doubles, the longest repair run grows from 18 to 58 commits, and median commit cadence slows from 5.9 to 12.7 minutes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 99.3% cache hit rate is typical of agentic development loops, then nominal API price sheets systematically overstate cloud costs for interactive coding agents; the per-token cost inversion likely weakens for batch workloads or fresh-context tasks with low cache reuse.
  • The observed bundle conflates base-model capability, quantization, and agent harness, so a natural test is an ablation that swaps one factor at a time (e.g., the same harness on both backends); the paper's design cannot attribute the repair premium to any single factor.
  • A multi-developer crossover study would reveal whether the repair premium and the 40% savings figure survive variation in skill, codebase, and period ordering; until then both should be read as single-site estimates.
  • The cost-quality frontier implies enterprises can treat defect rate as a tunable budget: routing only high-difficulty work to the cloud captures meaningful savings with modest FCR penalty, but each further shift to the local model trades quality for infrastructure dollars.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper reports a single-developer, non-randomized, 56-day longitudinal case study comparing two deployable coding-agent configurations on a production monorepo: Period A uses Claude Code with the Claude Opus 4.7/4.8 API; Period B uses Opencode with on-premise NVFP4-quantized GLM-5.1/5.2 served by vLLM on a Blackwell 4x B200 slice. The paper claims three main results: (1) prompt caching reduces the realized API cost to an effective $0.57 per million tokens, below the shared on-premise amortized cost of $2.83/M, although the API total bill is larger; (2) the local configuration is associated with a much higher defect-repair burden, with Fix Commit Ratio 74.9% vs. 45.9% and Mantel-Haenszel odds ratio 3.61 across difficulty tiers; (3) under Taiwan-market parameters and a symmetric labor model, shared on-premise allocation saves 40.1% of true TCO while dedicated reservation costs 43.8% more, and an offline replay of routing policies yields a monotonic cost-quality frontier rather than a dominating hybrid policy. The methodology is transparent: deterministic keyword rules, Git patch-id deduplication, Redmine cross-validation, and a fully specified cost model. The paper explicitly scopes its claims and acknowledges the absence of an independent defect oracle.

Significance. If the results hold, the paper makes two practically important contributions. First, it documents a concrete mechanism by which prompt caching inverts the per-token cost comparison between a frontier API and a self-hosted quantized cluster, with a detailed, reproducible cost accounting. Second, it provides an unusually complete longitudinal measurement of the defect-repair burden of an on-premise quantized agent relative to a cloud API, including per-tier stratification, per-language analysis, behavioral workload indicators, and a hybrid-routing counterfactual. The paper's methodological transparency is a genuine strength: the classification rules, telemetry extraction details, cost parameters, and bootstrap procedure are specified in enough detail to be reproduced on other repositories. The central caveat is that the quality comparison rests on a single measurement instrument, the commit-message keyword classifier, and the paper itself acknowledges that no independent defect oracle exists. Given the single-subject design and proprietary data, the empirical claims should be read as an exploratory case study; the statistical tests quantify within-developer consistency rather than po

major comments (3)
  1. [§IV-C, Eq. (1)] The defect premium (FCR 74.9% vs. 45.9%; OR_MH=3.61) rests entirely on a keyword regex applied to commit messages authored by the sole, hypothesis-aware developer. Period B is not packaging-invariant by the paper's own data (Table IV: 255 vs. 101 fix:-prefixed; Table X: 367 vs. 246 commits, 11.18 vs. 18.35 commits/active day, median insertions/commit 25 vs. 36, p≈0.068). A switch to Opencode, which emits Conventional-Commits-style messages and may commit at finer granularity, could inflate fix counts without changing defect rates. The §VII-C defenses (CI gate, taxonomy, per-tier/per-language) reuse the same labels and cannot validate them; Redmine checks attribution, not fix semantics; no independent oracle exists (acknowledged in §VII-C). Because per-tier ORs, Table IX, §V-G, §VI-A, and §VI-C all derive from these labels, this is load-bearing. Required: a blind diff-based or issue-track
  2. [§VI-C, Table XIV] The offline replay treats the same regex-derived per-tier fix rates as transportable backend properties and pools commits across periods. The paper acknowledges the exchangeability assumption, but the bootstrap propagates only sampling variance, not label bias. Because these per-tier rates are exactly what the packaging confound of §IV-C can shift, the frontier ordering, the 'no dominating policy' claim, and all four TCO/FCR comparisons are contingent on label validity. Please re-run the replay under a conservative FCR band (or oracle-based rates) and show whether the frontier ordering survives.
  3. [§V-C, Table X / Fig. 5] Difficulty is derived from change footprint, and fix commits are systematically smaller (median 25 vs. 36 lines), mechanically enriching the Low tier with fixes, as the paper notes. This is a fair between-period contrast only if commit granularity is period-invariant; the data show it is not (more, smaller commits in Period B). Please report per-tier odds ratios with a packaging-invariant difficulty measure (e.g., normalized code-only change size or pre-spec complexity), or restrict to commits above a minimum size.
minor comments (6)
  1. [Table IV footnote] The distinction between the fix:-prefix count and the regex-derived FCR count is easy to miss. Consider adding a short cross-tabulation of the two definitions and explicitly stating which count is used in each analysis (Table IX, §V-C, §VI-A).
  2. [§V-C, Table X] The median insertions/commit contrast is reported with p=0.068 and Cliff's delta +0.087 (negligible). Please state the unit of analysis for this test and discuss why a negligible effect size accompanies a p-value close to 0.05.
  3. [§VI-A, Eq. (5)] Equation (5) reuses the symbol TC from Eq. (2) with a different meaning (total cost vs. per-task cost). Rename one of them (e.g., TC_total and TC_task) to avoid confusion.
  4. [Fig. 3] The cumulative net-line plot would benefit from annotated vertical lines for the mid-period model upgrades (Opus 4.8 on 2026-05-28; GLM-5.2 on 2026-06-13), since the text argues there is no structural break at these transitions.
  5. [Abstract / §V-D] The statement that the cached API is cheaper per token than the shared on-premise amortization is utilization-dependent, as the paper later clarifies. Consider marking this with an asterisk at the first occurrence in the abstract so it cannot be read as a universal claim.
  6. [§VII-C, FCR validity] The recommendation of a blind second-rater protocol is welcome. Since the authors already have full Git history, a practical interim step would be to report inter-rater agreement on a sample of commit messages or to provide a small labeled sample in an appendix for independent verification.

Circularity Check

0 steps flagged

No load-bearing circularity; FCR and cost results are direct measurements, the TCO model is explicit with swept parameters, and the offline replay is a labeled counterfactual simulation.

full rationale

The paper's central results do not reduce to their inputs. The FCR contrast (74.9% vs 45.9%, OR_MH = 3.61) is a direct application of the deterministic regex in Eq. (1) to mined commit messages; it is a measurement, not a fitted prediction, and the paper openly flags the absence of an independent defect oracle (§VII-C) as a construct-validity threat rather than claiming the regex is validated by its own downstream uses. Per-tier and per-language stratifications re-aggregate the same labels and are presented as descriptive subgroup analyses, with the Low-tier/base-rate coupling explicitly acknowledged (§V-C). The per-token cost inversion ($0.57/M vs $2.83/M) is arithmetic on measured token telemetry and disclosed prices/amortization, with total spend and TCO stated as the robust quantities (§V-D). The TCO model uses explicit, swept parameters including a symmetric 15-minute review time grounded in the same cadence stream; because the time is symmetric and the conclusion is robust across the 10-25 min sweep, the model does not take its conclusion as input. The offline routing replay is transparently a counterfactual reweighting of measured per-tier fix rates (§VI-C), labeled as a projection, not a new empirical prediction. The only self-citation ([5]) is background on Copilot RCTs and is not load-bearing. Overall: no circular step is exhibited; the acknowledged single-instrument FCR limitation is a measurement-validity concern, not a derivation cycle.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The core measured quantities (token volumes, cache-hit rates, commit counts, FCR) come from telemetry and Git mining, not from assumptions. On top of that, the paper adds an accounting layer with five parameterized inputs (per-fix review time, wage, GPU basis, ops overhead, transportable per-tier fix rates) and three domain assumptions about the validity of commit labels, difficulty tiers, and granularity invariance. All are disclosed in the text and most are swept or argued conservative; only the label-validity and granularity-invariance assumptions are load-bearing for the central quality claim. No invented entities: the routing gateway is an explicitly unevaluated design sketch and E_net = 1−FCR is a defined algebraic ratio, not a new postulated thing.

free parameters (5)
  • Per-fix review time = 0.25 h (15 min), symmetric across both configurations
    Chosen uniformly and grounded in the observed median inter-commit interval preceding fix commits (14.1 vs 13.2 min, §VI-A). Drives the repair-labor terms ($988.75 vs $2,406.25) and hence the 40.1% / 43.8% TCO conclusions; a 10–25 min sweep keeps the shared-allocation conclusion positive.
  • Developer wage R_dev = $35/hr (NT$1,120/hr) employer cost
    Taiwan-market senior-engineer compensation from DGBAS statistics [28]; a market input rather than a data-fitted value, but still a chosen parameter of the TCO model, swept from $27–45/hr.
  • GPU cost basis = $16/hr per 4×B200 tray; shared seat $2,560/mo (160 h); dedicated $10,752 (672 h)
    Rental-equivalent scenarios built from market B200 rates ($3.70–5.90/GPU-h [26]). The shared-vs-dedicated split is what decides a 40.1% saving versus a 43.8% excess cost; the paper declares the per-token on-premise rate utilization-dependent.
  • Operational overhead C_ops = $893/developer/28 days (power $813 incl. PUE 1.5; MLOps $30; facilities $50)
    Composed of Taipower rates, an assumed 6.0 kW tray draw, PUE 1.5, and amortization across a ~100-developer cohort; the authors note the bases differ deliberately and are conservative against the savings conclusion.
  • Per-tier fix rates used in the offline replay = Claude 15.1/61.2/63.4%; GLM 46.5/80.6/86.9% (High/Med/Low)
    Measured from the two periods' commits, then treated as transportable properties of each (difficulty tier, backend) pair in the §VI-C counterfactual replay — measured parameters, not fitted to the conclusion, but structurally assumed exchangeable across periods.
axioms (6)
  • domain assumption Commit-message keyword regex (/(fix|bug|issue|patch|correct)/i) is a valid indicator of defect-repair commits in both periods.
    Load-bearing for FCR, OR_MH = 3.61, and all six workload indicators (§IV-C, §V-C). Misclassification is acknowledged ([36]) and argued conservative (Period A's test/CI share is larger), but there is no independent defect oracle (§VII-C), and the messages are authored by the hypothesis-aware sole developer.
  • domain assumption Difficulty tiers derived from change footprint (packages touched, files, net lines) are comparable across periods and not mechanistically coupled to the outcome.
    §IV-E defines High/Med/Low from footprint; §V-C concedes the Low tier is mechanically enriched with fixes (fix commits are smaller), so tiers read as change-size strata rather than intrinsic difficulty. The between-period contrast is argued unaffected, but the 'within every difficulty tier' claim inherits this coupling.
  • domain assumption Commit granularity and repair-labeling behavior are invariant across the two periods.
    FCR and run-structure indicators assume the developer packaged work into commits comparably and used 'fix'-type subjects similarly in both periods. §VII-C argues content-defined properties (taxonomy shift, per-language FCR) bound this, but it is not directly testable on the same data.
  • domain assumption Symmetric labor model: equal per-fix review time in both periods; initial feature-development labor cancels in the between-configuration comparison.
    §VI-A instantiates TC = compute + ops + T_dev × R_dev and makes the labor difference driven entirely by fix-commit volume. The 'True TCO' is therefore differential repair labor, not an absolute lifecycle cost — the paper says so explicitly.
  • domain assumption Transportability and linearity in the offline replay: each backend's per-tier fix rate transfers to rerouted tasks, and infrastructure cost scales linearly with commit count.
    §VI-C acknowledges these as first-order approximations ('treats the tiers as exchangeable across periods'); the two-stage bootstrap quantifies sampling variance only, not this structural assumption.
  • standard math Standard frequentist machinery (χ², Mantel–Haenszel, Cliff's Δ, bootstrap) is applied to non-independent single-subject units.
    The authors themselves state the units are not independent replicates (§V-C, scope of inference) and that p-values quantify within-subject consistency, not population inference. The machinery is standard; the interpretive claim is appropriately scoped.

pith-pipeline@v1.3.0-alltime-deepseek · 28063 in / 17020 out tokens · 149870 ms · 2026-08-02T06:58:32.726706+00:00 · methodology

0 comments
read the original abstract

Autonomous coding agents force engineering organizations to choose between API-based frontier models -- strong reasoning at high token cost -- and on-premise quantized open-weights models, which promise low-marginal-cost scaling and data sovereignty at some loss of reasoning fidelity. We study this trade-off through a single-developer, non-randomized longitudinal case study over two contiguous 28-day periods on a production monorepo: an API-based Claude Opus 4.7/4.8 configuration using Claude Code versus an on-premise GLM-5.1/5.2 configuration using Opencode, quantized to NVFP4, on NVIDIA Blackwell hardware. Analyzing LLM telemetry and Git history, we find that prompt caching (99.3% hit rate) cuts realized API cost by 88.6% to an effective \$0.57 per million tokens -- below even the \$2.83 amortized unit cost of the shared on-premise slice (a utilization-dependent inversion; total realized spend and total cost of ownership (TCO) are the robust quantities). At comparable gross code churn, the local configuration was associated with a far higher defect-repair burden: a Fix Commit Ratio (FCR) of 74.9% versus 45.9%, with the odds of a commit being a repair 2.6 to 4.9 times higher within every difficulty tier (Mantel-Haenszel OR = 3.61). Under Taiwan-market parameters and a symmetric labor model, on-premise deployment nonetheless saves 40.1% of true TCO under shared GPU allocation, whereas dedicated reservation costs 43.8% more than the cached API. Under shared allocation, the genuine penalty is not monetary but a measurable developer-experience burden -- timestamp indicators show more work trapped in debugging spirals and a slower commit cadence -- and an offline replay shows hybrid routing gateways trade defect rate for infrastructure savings along a cost-quality frontier rather than dominate the pure-API baseline.

Figures

Figures reproduced from arXiv: 2607.13080 by Sheng-Wei Peng, Yi-Hsun Lin, Yi-Pei Lee.

Figure 1
Figure 1. Figure 1: Deployable-configuration architectures compared in this study. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end measurement pipeline: Git history (all branches and PR [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cumulative net line growth over the 28-day periods (daily Git data). [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Percentage distribution of defect categories classified from fix [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Forest plot of the fix-commit odds ratio (Period B [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Speed–quality trade-off (Artificial Analysis hosted-endpoint measure [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Objective behavioral workload indicators expressed as the Period-B-to [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Cost–quality frontier of the four routing policies (Table XIV). The [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Simulated True TCO breakdown on the pooled 613-commit workload [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Two-stage-bootstrap distributions (normal approximation of [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Weekly FCR trend across Period A (Weeks 1–4) and Period B [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Proposed Tiered Routing Gateway: complex tasks route to the public [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance

    cs.SE 2026-07 conditional novelty 6.0

    On the tqdm benchmark tasks, the best output format depends on the model: Doubao thrives with JSON Patch (94%), DeepSeek with unified diff (66%), and Qwen with full file (50%).

Reference graph

Works this paper leans on

37 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    NVIDIA Blackwell architecture: Technical brief,

    NVIDIA, “NVIDIA Blackwell architecture: Technical brief,” NVIDIA Corporation, Tech. Rep., 2024, https://www.nvidia.com/ en-us/data-center/technologies/blackwell-architecture/

    2024

  2. [2]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inAdv. Neural Information Processing Systems (NeurIPS), 2024

    2024

  3. [3]

    Agentless: De- mystifying LLM-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying LLM-based software engineering agents,”arXiv preprint arXiv:2407.01489, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inProc. Int. Conf. Learning Representations (ICLR), 2024

    2024

  5. [5]

    The impact of AI on developer productivity: Evidence from GitHub Copilot,

    S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The impact of AI on developer productivity: Evidence from GitHub Copilot,”arXiv preprint arXiv:2302.06590, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Measuring the impact of early-2025 AI on experienced open-source developer productivity,

    J. Becker, N. Rush, E. Barnes, and D. Rein, “Measuring the impact of early-2025 AI on experienced open-source developer productivity,” arXiv preprint arXiv:2507.09089, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” inProc. 29th ACM Symp. Operating Systems Principles (SOSP), 2023, pp. 611–626

    2023

  8. [8]

    Orca: A distributed serving system for transformer-based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in Proc. 16th USENIX Symp. Operating Systems Design and Implementa- tion (OSDI), 2022

    2022

  9. [9]

    AWQ: Activation-aware weight quantiza- tion for on-device LLM compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware weight quantiza- tion for on-device LLM compression and acceleration,” inProc. Machine Learning and Systems (MLSys), 2024

    2024

  10. [10]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in Proc. Int. Conf. Learning Representations (ICLR), 2023

    2023

  11. [11]

    SmoothQuant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” inProc. Int. Conf. Machine Learning (ICML), 2023

    2023

  12. [12]

    GLM-5.2-NVFP4 official model configuration,

    NVIDIA and Zhipu AI, “GLM-5.2-NVFP4 official model configuration,” https://huggingface.co/nvidia/GLM-5.2-NVFP4, 2026, accessed 2026- 07-07

    2026

  13. [13]

    Evaluating the impact of post-training quantization on large language models for code generation,

    A. Giagnorio, A. Mastropaolo, S. Afrin, M. Di Penta, and G. Bavota, “Evaluating the impact of post-training quantization on large language models for code generation,”arXiv preprint arXiv:2503.07103, 2025

    arXiv 2025

  14. [14]

    Artificial Analysis: LLM model intelligence, per- formance and price index,

    Artificial Analysis, “Artificial Analysis: LLM model intelligence, per- formance and price index,” https://artificialanalysis.ai/models, 2026, accessed 2026-07-07

    2026

  15. [15]

    SWE-bench Pro leaderboard (2026): Model scores,

    Morph, “SWE-bench Pro leaderboard (2026): Model scores,” https:// www.morphllm.com/swe-bench-pro, 2026, accessed 2026-07-07

    2026

  16. [16]

    GPT-5.5 model documentation,

    OpenAI, “GPT-5.5 model documentation,” https://developers.openai. com/api/docs/models/gpt-5.5, 2026, accessed 2026-07-11

    2026

  17. [17]

    Prompt caching: Speed up and lower the cost of your API requests,

    Anthropic, “Prompt caching: Speed up and lower the cost of your API requests,” https://claude.com/blog/prompt-caching, 2024, accessed 2026-07-07

    2024

  18. [18]

    Claude Code: Agentic coding documentation,

    Anthropic, “Claude Code: Agentic coding documentation,” https://code. claude.com/docs, 2026, accessed 2026-07-07

    2026

  19. [19]

    opencode: AI coding agent built for the terminal,

    Anomaly / SST, “opencode: AI coding agent built for the terminal,” https://opencode.ai/docs/, 2026, accessed 2026-07-07

    2026

  20. [20]

    GLM-5.1-NVFP4 model con- figuration,

    Zhipu AI and community quantization, “GLM-5.1-NVFP4 model con- figuration,” https://huggingface.co/lukealonso/GLM-5.1-NVFP4, 2026, accessed 2026-07-07

    2026

  21. [21]

    Langfuse: Open source LLM engineering platform for telemetry and pricing audit,

    Langfuse Team, “Langfuse: Open source LLM engineering platform for telemetry and pricing audit,” https://langfuse.com, 2025, accessed 2026- 07-07

    2025

  22. [22]

    Identifying reasons for software changes using historic databases,

    A. Mockus and L. G. V otta, “Identifying reasons for software changes using historic databases,” inProc. Int. Conf. Software Maintenance (ICSM), 2000, pp. 120–130

    2000

  23. [23]

    B. W. Boehm,Software Engineering Economics. Prentice-Hall, 1981

    1981

  24. [24]

    Recommendations for analysing and meta-analysing small sample size software engineering experiments,

    B. Kitchenham and L. Madeyski, “Recommendations for analysing and meta-analysing small sample size software engineering experiments,” Empirical Softw. Eng., vol. 29, no. 6, p. 137, 2024

    2024

  25. [25]

    Claude platform pricing,

    Anthropic, “Claude platform pricing,” https://platform.claude.com/docs/ en/about-claude/pricing, 2026, accessed 2026-07-07

    2026

  26. [26]

    NVIDIA B200 cloud pricing comparison across providers,

    GetDeploying, “NVIDIA B200 cloud pricing comparison across providers,” https://getdeploying.com/gpus/nvidia-b200, 2026, accessed 2026-07-07

    2026

  27. [27]

    Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research,

    S. G. Hart and L. E. Staveland, “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research,” inHuman Mental Workload, P. A. Hancock and N. Meshkati, Eds. North-Holland, 1988, pp. 139–183

    1988

  28. [28]

    Earnings and productivity statistics,

    Directorate-General of Budget, Accounting and Statistics (DGBAS), Taiwan, “Earnings and productivity statistics,” https://earnings.dgbas. gov.tw/, 2026, accessed 2026-07-07

    2026

  29. [29]

    Electricity tariff tables (effective 2025-10- 01),

    Taiwan Power Company, “Electricity tariff tables (effective 2025-10- 01),” https://www.taipower.com.tw/2289/2290/46940/, 2025, accessed 2026-07-07

    2025

  30. [30]

    Estimating the carbon footprint of BLOOM, a 176B parameter language model,

    A. S. Luccioni, S. Viguier, and A.-L. Ligozat, “Estimating the carbon footprint of BLOOM, a 176B parameter language model,”J. Mach. Learn. Res., vol. 24, 2023

    2023

  31. [31]

    FrugalGPT: How to use large language models while reducing cost and improving performance,

    L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,”Trans. Mach. Learn. Res., 2024

    2024

  32. [32]

    RouteLLM: Learning to route LLMs with preference data,

    I. Ong, A. Almahairi, V . Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica, “RouteLLM: Learning to route LLMs with preference data,” inProc. Int. Conf. Learning Representations (ICLR), 2025

    2025

  33. [33]

    AI agents that matter,

    S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan, “AI agents that matter,”arXiv preprint arXiv:2407.01502, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    A large scale study of programming languages and code quality in GitHub,

    B. Ray, D. Posnett, V . Filkov, and P. Devanbu, “A large scale study of programming languages and code quality in GitHub,” inProc. 22nd ACM SIGSOFT Symp. Foundations of Software Engineering (FSE), 2014, pp. 155–165

    2014

  35. [35]

    On the impact of programming languages on code quality: A reproduction study,

    E. D. Berger, C. Hollenbeck, P. Maj, O. Vitek, and J. Vitek, “On the impact of programming languages on code quality: A reproduction study,”ACM Trans. Program. Lang. Syst., vol. 41, no. 4, pp. 21:1–21:24, 2019

    2019

  36. [36]

    It’s not a bug, it’s a feature: How misclassification impacts bug prediction,

    K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: How misclassification impacts bug prediction,” inProc. 35th Int. Conf. Software Engineering (ICSE), 2013, pp. 392–401

    2013

  37. [37]

    CodeT: Code generation with generated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “CodeT: Code generation with generated tests,” inProc. Int. Conf. Learning Representations (ICLR), 2023. 20

    2023