API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

Daniel E. Ho; Jennifer Wang; Joachim Baumann; Sanmi Koyejo

REVIEW

API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo

Artificial Intelligence · Submitted 8 Sep 2026

Pith's verdict: Solid, well-designed audit showing API–interface gaps; the checkpoint caveat is real but transparently handled and doesn't sink the paper.

T0 · AI review

How this review was made: An AI referee (deepseek/deepseek-v4-flash) read the paper on September 9, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

T0 AI review An AI read the paper and answered a fixed list of questions.
T1 Audit Automatic checks that the references and files the paper relies on exist and match.
T2 Compute The paper's code or calculations re-run with the same results.
T3 Formal statement The claims written out in a formal math language a computer can read.
T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 5 minor · Ask about this paper · The compiled record · Read the full review →

Challenge this review Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2609.08861 (version 3)

Pith IDTNLEIV2H, Pith's permanent record of this paper and its review

CitationDaniel E. Ho, Jennifer Wang, Joachim Baumann, Sanmi Koyejo. "API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces." arXiv:2609.08861 (2026). https://pith.science/paper/TNLEIV2H

Cite this reviewBibTeX for Pith's review of this paper

Claim paper

Why Pith says this

Highlight any text to comment on it.

Editor's note

read the letter

This is the first large-scale controlled comparison of API vs. chatbot-interface benchmark scores across multiple providers. The methodology is genuinely careful—synchronized queries, fresh sessions, disabled personalization and tools, output normalization, human-validated extraction. The main finding, a ~3.4 pp accuracy gap and ~2.1 pp test–retest gap, is convincingly documented. The paper’s strongest move is showing that exposed API controls (system prompts, sampling, reasoning budget) don’t reliably close the gap, which strengthens the case that something in the deployment stack beyond these knobs is driving the difference.

The stress-test concern about checkpoint identity is real, and the authors flag it as a central limitation in Section 7. But they don’t overclaim: they call it a “context-validity gap” rather than attributing it to any single layer, and they design the study to estimate the combined effect of all access-surface differences, not to isolate checkpoint identity. If providers serve different model variants across surfaces, that is itself a meaningful finding about what benchmark scores actually measure. The comparison to model-version differences (GPT 5.3 vs 5.4) is clearly labeled as a perspective check, not a causal decomposition. So the caveat reduces confidence in attributing the gap wholly to interface-layer effects, but it does not undermine the paper’s core claim that API scores do not reliably predict interface behavior.

Soft spots worth noting: the audit covers one subscription tier per platform, so tier-based routing could produce different gaps for other users. The benchmarks are standardized, not naturalistic conversational interactions. And the open-source tool may face terms-of-service barriers for others. But these are minor relative to the contribution. The paper is transparent about all of them.

This paper is for anyone who uses benchmark scores to evaluate or purchase LLM systems—regulators, procurement officers, practitioners. It deserves a serious referee. The experiments are reproducible, the data is shared, and the analysis is sound. I would cite it in my own work on evaluation validity.

Recommendation for peer review: accept with minor revisions. The checkpoint identity caveat should stay prominent; the authors might add a sentence noting that if providers do route different checkpoints, that itself validates the paper’s central point about benchmark context-dependence.

Referee report

Minor objections (5)

[§5.1, Table 11] The system-prompt ablation produces some large deviations from the interface (e.g., GPT 5.3 Instant on HellaSwag: interface–SP gap +28.3 pp). The paper overall concludes that system prompts do not close the accuracy gap, but this particular large deviation is not discussed. A brief comment explaining why the prompt apparently worsened performance relative to both API and interface would be informative.
[§4.1, Table 9] The Spearman correlation for GSM8K (ρ=0.07) and MMLU (ρ=0.11) is noted as indicating weak rank correspondence, but this is likely influenced by near-ceiling scores (range approximately 92–96% for MMLU). The paper mentions this in passing, but adding a direct statement of the score range for these benchmarks in the main text would help readers interpret the correlation magnitude.
[§3.4] The description of the temporal-drift control ('per-query barrier') is brief. Adding a sentence on how synchronization was verified (e.g., timestamp logging or network-level check) would increase reproducibility.
[Figure 3] The heatmap is informative, but the color scale legend lacks explicit annotation of units (percentage points). Adding '∆ (pp)' to the legend would improve clarity.
[§7] The checkpoint-identity limitation is well and transparently stated. Consider adding a brief discussion of how plausible checkpoint differences might be (e.g., known instances of providers serving quantized or differently fine-tuned variants across surfaces) to help readers calibrate the severity of this limitation.

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

The paper shows that identical benchmark prompts score 3.4 accuracy points and 2.1 consistency points higher through model APIs than through the corresponding chatbot interfaces, so API scores should not be read as product behavior.

The reading

This paper challenges a working assumption behind nearly every model release: that a benchmark score measured through a model's API tells you how that model behaves in the chatbot product people actually use. The authors sent identical prompts through both access paths for seven systems across ChatGPT, Claude, and Gemini, using nine benchmarks covering general capability, social bias, and sycophancy, while controlling for personalization, tool use, timing, and output formatting. They report that API evaluations score on average 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test-retest agreement than interface evaluations, and that for ChatGPT the access-surface gap exceeds the API-only gap between GPT 5.3 and GPT 5.4. Exposed API parameters (system prompts, sampling settings, reasoning budgets) shift behavior but do not reliably close the gap. If the finding holds, API scores describe the API configuration rather than the deployed product, and any benchmark claim that omits the access surface is incomplete.

What carries the argument

The study's organizing object is context validity, defined in the paper as whether measurements obtained in one evaluation environment generalize to another, and the specific context dimension it manipulates is the access surface, the technical route through which identical prompts reach a model's weights. The argument is carried by a paired audit design in which each benchmark item is submitted to the API and the interface through a per-query synchronization barrier, across five independent trials with fresh accounts, browser profiles, and chat windows, tools disabled, outputs normalized to plain text, and both conditions graded by the same extraction pipeline. Gaps are estimated with linear probability mixed-effects models carrying random intercepts for item, system, and benchmark; test-retest agreement is the average probability that two runs on the same item agree on binary correctness; and ranking transfer is measured by the Spearman correlation between the two surfaces. Two ablation families, prepending approximated public system prompts and sweeping temperature, top-p, and reasoning budgets, act as diagnostics of whether exposed API controls can reproduce interface behavior, and the study finds that they cannot do so reliably.

Extended reading notes

Core claim

The paper's central claim is that a context-validity gap separates API measurements from interface measurements: the same benchmark items, prompt texts, and scoring pipeline produce lower and less stable scores when issued through deployed chat interfaces than through the corresponding model APIs. Across seven systems and nine benchmarks, API accuracy exceeded interface accuracy for every system, with a pooled gap of +3.4 points (p=0.002); per-cell gaps ranged up to +28.4 points on the moral-sycophancy benchmark AITA for Claude Haiku 4.5 and +19.0 points on WinoGrande for GPT 5.4 Thinking, while API test-retest agreement averaged 96.9% versus 94.8% for interfaces. The surface also reorders systems, with a mean Spearman correlation of 0.52 between API and interface rankings, and same-provider orderings invert on several benchmarks. Prepending approximated public system prompts to API calls moves response consistency toward interface levels but leaves the accuracy gap essentially unchanged (6.5 to 6.4 points), and sweeping temperature, top-p, and reasoning budgets changes accuracy in only one of eight tested cells. The authors conclude that the divergence sits in deployment layers that external evaluators cannot observe or configure, so API evaluations are valid statements about the API configuration, not about the deployed product.

Load-bearing premise

The load-bearing premise, which the paper itself flags in its limitations section, is that each interface system and its matched API identifier actually serve the same underlying model checkpoint; if providers route different model variants to different surfaces, part or all of the measured gap could be model difference rather than an effect of the interface layer.

Editorial extensions

If this is right

Benchmark reports that omit the access surface can mislead: across the nine benchmarks the Spearman correlation between API and interface system rankings is only 0.52, so leaderboard order can change substantially depending on which surface was measured.
For ChatGPT, the API-versus-interface gap for GPT 5.4 Thinking is larger than the API-only accuracy difference between GPT 5.3 and GPT 5.4, meaning an access-surface change can look like a full model-generation downgrade.
Deployed chatbots are less repeatable than API measurements suggest, with test-retest agreement of 94.8% versus 96.9%, so applications that need stable answers will see more run-to-run variation in products than in API evaluation.
Because exposed system prompts, sampling parameters, and reasoning budgets do not reliably close the gap, external evaluators cannot currently reproduce deployed interface behavior from the API; bridging this requires provider-side transparency about routing, checkpoints, and hidden instruction stacks.
API-only assessments of user-facing risks understate what users encounter: on the sycophancy benchmark AITA the interface falls as much as 28.4 points behind the API, meaning deployed chatbots side with users more readily than the API numbers suggest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: replicating this audit with open-weight models served behind a self-hosted interface would settle the checkpoint caveat directly, since the model snapshot would be fully known and identical on both surfaces; the paper's design cannot make that separation.
Editorial extension: the directionality of the gap (API at or above interface in most of the 63 cells) suggests a systematic deployment-layer behavior rather than noise, although the paper deliberately avoids attributing the gap to any single mechanism such as interface instruction stacks, output policies, or routing heuristics.
Editorial extension: studies that simulate users through the API inherit this surface bias, so research claims about personalization, bias, or sycophancy should be re-validated against product surfaces before they inform policy or product decisions.
Editorial extension: a testable follow-up would check whether the gap widens on safety-relevant or ambiguous content, since the largest cells here are AITA, BBQ, and WinoGrande, which would indicate the deployment layer is reshaping exactly the responses that matter most for harm assessment.

Original abstract

Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model version. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,428 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.36 · human participants 0.09 · controlled comparison 0.51 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.79 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy.	background 1.00	0.04	0.02	asserted 1.00
s1	Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems.	background 0.81	0.05	0.03	asserted 0.99
s2	We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy.	contribution 0.75	0.95	0.04	asserted 0.96
s3	We find systematic API--interface differences in both accuracy and consistency.	result 1.00	0.96	0.25	asserted 1.00
s4	On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations.	result 1.00	0.96	0.14	asserted 1.00
s5	For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4.	result 0.81	0.93	0.07	asserted 1.00
s6	Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation.	interpretation 0.94	0.81	0.13	asserted 0.87
s7	We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings.	method 0.95	0.95	0.02	asserted 0.74
s8	These controls shift behavior in some cases but do not reliably eliminate the gap.	result 0.99	0.96	0.06	asserted 0.96
s9	Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.	interpretation 0.99	0.94	0.96	asserted 0.61

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

Figures

**Figure 1.** Chatbot systems systematically underperform their corresponding APIs. Across platforms, identical prompts receive lower-scoring and less consistent responses through the interfaces than through the corresponding model APIs. This use of benchmarks, however, assumes that the signal they provide transfers across contexts (Saxon et al., 2024). A score measured in one setting is treated as evidence about how a model or system will perform in another. Benchmark measurements, however, are inherently contextual (Brundage et al., 2026). They depend on the model snapshot, prompt format, sam

**Figure 2.** Chat interfaces underperform their corresponding APIs across all systems. Across nine benchmarks, every system has lower mean accuracy through the interface than through the API. after controlling for benchmark, system, and item clustering (p = 0.002). At the system level, all seven systems show significantly higher accuracy under API evaluation (all p < 0.001). The LME-estimated gap (βˆ 1) ranges from 1.9% for Gemini 3 Flash Thinking to 4.6% for GPT 5.4 Thinking, with GPT 5.3 Instant (2.4%) and Claude Opus 4.6 (2.4%) near the low end and Claude Haiku 4.5 (3.7%) and Gemini 3…

**Figure 3.** API–interface gaps are widespread but model- and benchmark-dependent. Most system–benchmark pairs show higher API scores, but the magnitude and direction vary across benchmarks. Values report API score minus interface score (pp). Warmer cells indicate API advantages; cooler cells indicate interface advantages. Asterisks denote Benjamini–Hochberg-adjusted significance across the 63 cell-level tests: ∗ q < 0.05 and ∗∗q < 0.01. for the broader trend. The test-retest agreement within each comparison also remains high across configurations (mean R = 98.2% for sampling sweeps and R = 96.3%

Figures reproduced from arXiv:2609.08861 by the authors.

References

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

"If I Had to Buy Just ONE: Galaxy S26 Ultra": Auditing AI-Generated Product Recommendations
cs.CY 2026-09 conditional novelty 7.0 of 10

AI chatbots vary drastically in how they recommend products and cite sources, and their APIs cannot substitute for auditing the consumer-facing interfaces.
Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans
cs.HC 2026-10 accept novelty 6.0 of 10

ImpactBench is an open, expert-guided benchmark platform that evaluates AI models on 809 behavioral metrics across psychological, physical, and social well-being, finding that models are better at fostering positive o...
Open Problems in Chatbots and Mental Health
cs.CL 2026-11 accept novelty 4.0 of 10

A structured survey of open problems and recommendations for researchers and policymakers on the mental health effects of general-purpose chatbots.

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.