{"id":"fdfe7ca9-1435-49f8-a77a-2d8e55a54ff5","arxiv_id":"2609.08861","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"API benchmark scores systematically overestimate chatbot interface performance by an average of 3.4 percentage points and show higher test-retest consistency.","lead":"This paper compares benchmark performance of large language models accessed via API versus their web-based chatbot interfaces. It finds systematic differences in accuracy and consistency, showing that API scores are not reliable proxies for real-world chatbot behavior.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Checkpoint identity between API and interface is unverifiable; if they differ, the gap's attribution to interface-layer effects is weakened.","rationale":"The reader identified precisely this checkpoint-identity concern as the weakest assumption, and our analysis confirms it is the most load-bearing threat to the paper's causal interpretation. However, the paper's central finding—that API and interface evaluations produce different accuracy and consistency scores—remains robust even if checkpoints differ, because the comparison is between the deployed system and the closest documented API configuration. The headline magnitude (3.4 pp accuracy gap) and the practical conclusion (API scores do not reliably transfer to interfaces) survive this concern. The concrete test we propose would either reinforce the paper's interpretation (if cross-surface agreement is high) or reveal a serious confound (if it is low), but in either case the main empirical result stands. Therefore, no change to the ACCEPT verdict is warranted; the paper transparently acknowledges the limitation, and our proposed test provides a clear path to strengthen the evidence.","tokens_in":35509,"tokens_out":4114,"duration_ms":40309,"concrete_test":"Compute per-item answer agreement (proportion of items where both surfaces yield the same correct/incorrect outcome) for each model–benchmark cell, using the existing run-level data. Compare this cross-surface agreement to the within-surface test–retest agreement reported in Table 8. If cross-surface agreement is substantially lower than within-API agreement (e.g., by more than 3 pp on average), that would strongly suggest that different checkpoints or fundamentally different pipeline components are being served, undermining the attribution of the gap solely to interface-layer effects. This test is a straightforward reanalysis of already-collected data and does not require new experiments. If the cross-surface agreement closely matches within-API agreement, then the checkpoint identity assumption is supported, and the gap can more confidently be attributed to interface-layer mechanisms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that API evaluations systematically differ from interface evaluations—rests on the assumption that the API and interface access paths serve the same underlying model checkpoint. The authors explicitly acknowledge in Section 7 that 'we cannot verify that matched API and interface identifiers always correspond to the same underlying checkpoint.' This is not a minor caveat; it is a direct threat to the causal interpretation that the gap is due to 'interface-layer effects' (system prompts, routing, tool policies, etc.). If providers serve different model variants across surfaces, then some or all of the observed gap (mean 3.4 pp accuracy, 2.1 pp test-retest) could reflect model differences rather than context-validity failures. For example, Table 1 shows that the interface model 'GPT 5.4 Thinking' is matched to API identifier 'gpt-5.4-2026-03-05', but the actual checkpoint served via the chat interface may be a later or earlier snapshot, a quantized variant, or a different fine-tune. The paper's control experiments (system prompts, sampling, reasoning) test whether exposed API parameters can close the gap, but if the checkpoints themselves differ, no amount of parameter tuning can fully reconcile the behaviors. This concern is particularly acute for the headline comparison to model-version gaps (GPT 5.3 vs GPT 5.4): if the interface actually runs an older or newer variant, the claim that 'switching access surfaces can degrade performance as much as downgrading a full model generation' may be misleading.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper presents a large-scale controlled audit comparing LLM benchmark scores obtained through API access versus deployed chatbot interfaces across seven systems (ChatGPT, Claude, Gemini) and nine benchmarks spanning general capability, social bias, and sycophancy. The authors find systematic differences: API evaluations score on average 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test-retest agreement than corresponding interface evaluations. For ChatGPT, the API-interface gap exceeds the API-only difference between GPT 5.3 and GPT 5.4. The paper further tests whether exposed API controls (system prompts, sampling, reasoning settings) can close the gap, finding that these controls shift behavior in some cases but do not reliably eliminate the difference. The authors release an open-source audit tool and document a 'context-validity gap' that complicates the use of API evaluations as proxies for deployed chatbot behavior.","tokens_in":35923,"tokens_out":2463,"duration_ms":24063,"significance":"The work addresses a critically understudied dimension of benchmark validity—context validity across access surfaces—with rigorous experimental design. The paper controls for request routing, session personalization, tool invocation, temporal drift, and output normalization; uses appropriate mixed-effects models and bootstrapping; and validates the extraction pipeline against human annotation (98.7% agreement). The transparency about the central limitation (unverifiable checkpoint identity) is a strength. If the findings replicate broadly, they have substantial implications for how benchmark scores are interpreted in purchasing decisions, policy, and research. The public release of an interface-audit tool also provides practical value.","major_comments":[],"minor_comments":[{"comment":"The system-prompt ablation produces some large deviations from the interface (e.g., GPT 5.3 Instant on HellaSwag: interface–SP gap +28.3 pp). The paper overall concludes that system prompts do not close the accuracy gap, but this particular large deviation is not discussed. A brief comment explaining why the prompt apparently worsened performance relative to both API and interface would be informative.","section":"§5.1, Table 11"},{"comment":"The Spearman correlation for GSM8K (ρ=0.07) and MMLU (ρ=0.11) is noted as indicating weak rank correspondence, but this is likely influenced by near-ceiling scores (range approximately 92–96% for MMLU). The paper mentions this in passing, but adding a direct statement of the score range for these benchmarks in the main text would help readers interpret the correlation magnitude.","section":"§4.1, Table 9"},{"comment":"The description of the temporal-drift control ('per-query barrier') is brief. Adding a sentence on how synchronization was verified (e.g., timestamp logging or network-level check) would increase reproducibility.","section":"§3.4"},{"comment":"The heatmap is informative, but the color scale legend lacks explicit annotation of units (percentage points). Adding '∆ (pp)' to the legend would improve clarity.","section":"Figure 3"},{"comment":"The checkpoint-identity limitation is well and transparently stated. Consider adding a brief discussion of how plausible checkpoint differences might be (e.g., known instances of providers serving quantized or differently fine-tuned variants across surfaces) to help readers calibrate the severity of this limitation.","section":"§7"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is the first large-scale controlled comparison of API vs. chatbot-interface benchmark scores across multiple providers. The methodology is genuinely careful—synchronized queries, fresh sessions, disabled personalization and tools, output normalization, human-validated extraction. The main finding, a ~3.4 pp accuracy gap and ~2.1 pp test–retest gap, is convincingly documented. The paper’s strongest move is showing that exposed API controls (system prompts, sampling, reasoning budget) don’t reliably close the gap, which strengthens the case that something in the deployment stack beyond these knobs is driving the difference.\n\nThe stress-test concern about checkpoint identity is real, and the authors flag it as a central limitation in Section 7. But they don’t overclaim: they call it a “context-validity gap” rather than attributing it to any single layer, and they design the study to estimate the combined effect of all access-surface differences, not to isolate checkpoint identity. If providers serve different model variants across surfaces, that is itself a meaningful finding about what benchmark scores actually measure. The comparison to model-version differences (GPT 5.3 vs 5.4) is clearly labeled as a perspective check, not a causal decomposition. So the caveat reduces confidence in attributing the gap wholly to interface-layer effects, but it does not undermine the paper’s core claim that API scores do not reliably predict interface behavior.\n\nSoft spots worth noting: the audit covers one subscription tier per platform, so tier-based routing could produce different gaps for other users. The benchmarks are standardized, not naturalistic conversational interactions. And the open-source tool may face terms-of-service barriers for others. But these are minor relative to the contribution. The paper is transparent about all of them.\n\nThis paper is for anyone who uses benchmark scores to evaluate or purchase LLM systems—regulators, procurement officers, practitioners. It deserves a serious referee. The experiments are reproducible, the data is shared, and the analysis is sound. I would cite it in my own work on evaluation validity.\n\nRecommendation for peer review: accept with minor revisions. The checkpoint identity caveat should stay prominent; the authors might add a sentence noting that if providers do route different checkpoints, that itself validates the paper’s central point about benchmark context-dependence.","headline":"Solid, well-designed audit showing API–interface gaps; the checkpoint caveat is real but transparently handled and doesn't sink the paper.","tokens_in":36370,"tokens_out":890,"would_cite":true,"duration_ms":10237,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper shows that identical benchmark prompts score 3.4 accuracy points and 2.1 consistency points higher through model APIs than through the corresponding chatbot interfaces, so API scores should not be read as product behavior.","keywords":["API evaluation","chatbot interface","benchmark validity","context validity","test-retest reliability","ecological validity","access surface","LLM auditing"],"falsifier":"The decisive check is a checkpoint-exact replication: run the same nine-benchmark audit with one model snapshot served behind both surfaces, either by a provider that locks and discloses the checkpoint for both access paths or by a self-hosted deployment of open-weight models behind a custom interface. If the 3.4-point accuracy gap and the 2.1-point test-retest gap shrink to zero under identical checkpoints, the context-validity claim fails; a partial confirmation would come from provider trace logs showing interface requests reaching the same model identifier and inference configuration as API requests while the gap persists.","tokens_in":37321,"feed_emoji":"📉","tokens_out":14860,"duration_ms":132857,"temperature":0.7,"pith_summary":"This paper challenges a working assumption behind nearly every model release: that a benchmark score measured through a model's API tells you how that model behaves in the chatbot product people actually use. The authors sent identical prompts through both access paths for seven systems across ChatGPT, Claude, and Gemini, using nine benchmarks covering general capability, social bias, and sycophancy, while controlling for personalization, tool use, timing, and output formatting. They report that API evaluations score on average 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test-retest agreement than interface evaluations, and that for ChatGPT the access-surface gap exceeds the API-only gap between GPT 5.3 and GPT 5.4. Exposed API parameters (system prompts, sampling settings, reasoning budgets) shift behavior but do not reliably close the gap. If the finding holds, API scores describe the API configuration rather than the deployed product, and any benchmark claim that omits the access surface is incomplete.","feed_headline":"Identical prompts score 3.4 points higher via API than chatbot","feed_subtitle":"Switching access surfaces cuts accuracy and repeatability as much as a full model downgrade.","key_machinery":"The study's organizing object is context validity, defined in the paper as whether measurements obtained in one evaluation environment generalize to another, and the specific context dimension it manipulates is the access surface, the technical route through which identical prompts reach a model's weights. The argument is carried by a paired audit design in which each benchmark item is submitted to the API and the interface through a per-query synchronization barrier, across five independent trials with fresh accounts, browser profiles, and chat windows, tools disabled, outputs normalized to plain text, and both conditions graded by the same extraction pipeline. Gaps are estimated with linear probability mixed-effects models carrying random intercepts for item, system, and benchmark; test-retest agreement is the average probability that two runs on the same item agree on binary correctness; and ranking transfer is measured by the Spearman correlation between the two surfaces. Two ablation families, prepending approximated public system prompts and sweeping temperature, top-p, and reasoning budgets, act as diagnostics of whether exposed API controls can reproduce interface behavior, and the study finds that they cannot do so reliably.","core_discovery":"The paper's central claim is that a context-validity gap separates API measurements from interface measurements: the same benchmark items, prompt texts, and scoring pipeline produce lower and less stable scores when issued through deployed chat interfaces than through the corresponding model APIs. Across seven systems and nine benchmarks, API accuracy exceeded interface accuracy for every system, with a pooled gap of +3.4 points (p=0.002); per-cell gaps ranged up to +28.4 points on the moral-sycophancy benchmark AITA for Claude Haiku 4.5 and +19.0 points on WinoGrande for GPT 5.4 Thinking, while API test-retest agreement averaged 96.9% versus 94.8% for interfaces. The surface also reorders systems, with a mean Spearman correlation of 0.52 between API and interface rankings, and same-provider orderings invert on several benchmarks. Prepending approximated public system prompts to API calls moves response consistency toward interface levels but leaves the accuracy gap essentially unchanged (6.5 to 6.4 points), and sweeping temperature, top-p, and reasoning budgets changes accuracy in only one of eight tested cells. The authors conclude that the divergence sits in deployment layers that external evaluators cannot observe or configure, so API evaluations are valid statements about the API configuration, not about the deployed product.","pith_inferences":["Editorial extension: replicating this audit with open-weight models served behind a self-hosted interface would settle the checkpoint caveat directly, since the model snapshot would be fully known and identical on both surfaces; the paper's design cannot make that separation.","Editorial extension: the directionality of the gap (API at or above interface in most of the 63 cells) suggests a systematic deployment-layer behavior rather than noise, although the paper deliberately avoids attributing the gap to any single mechanism such as interface instruction stacks, output policies, or routing heuristics.","Editorial extension: studies that simulate users through the API inherit this surface bias, so research claims about personalization, bias, or sycophancy should be re-validated against product surfaces before they inform policy or product decisions.","Editorial extension: a testable follow-up would check whether the gap widens on safety-relevant or ambiguous content, since the largest cells here are AITA, BBQ, and WinoGrande, which would indicate the deployment layer is reshaping exactly the responses that matter most for harm assessment."],"forward_implications":["Benchmark reports that omit the access surface can mislead: across the nine benchmarks the Spearman correlation between API and interface system rankings is only 0.52, so leaderboard order can change substantially depending on which surface was measured.","For ChatGPT, the API-versus-interface gap for GPT 5.4 Thinking is larger than the API-only accuracy difference between GPT 5.3 and GPT 5.4, meaning an access-surface change can look like a full model-generation downgrade.","Deployed chatbots are less repeatable than API measurements suggest, with test-retest agreement of 94.8% versus 96.9%, so applications that need stable answers will see more run-to-run variation in products than in API evaluation.","Because exposed system prompts, sampling parameters, and reasoning budgets do not reliably close the gap, external evaluators cannot currently reproduce deployed interface behavior from the API; bridging this requires provider-side transparency about routing, checkpoints, and hidden instruction stacks.","API-only assessments of user-facing risks understate what users encounter: on the sycophancy benchmark AITA the interface falls as much as 28.4 points behind the API, meaning deployed chatbots side with users more readily than the API numbers suggest."],"supporting_citations":[{"why":"Supplies the three-dimension ecological-validity framework (content, task, context) from which the paper takes its definition of context validity.","marker":"(Schmuckler, 2001)"},{"why":"Identifies the 'missing middle' deployment layers between user input and model response that the paper treats as the unobservable source of the surface gap.","marker":"(Neumann and Singh, 2025)"},{"why":"Metabench's sparse distillation of the six OpenLLM benchmarks (858 items instead of 28,659) is what makes repeated interface evaluation tractable.","marker":"(Kipnis et al., 2025)"},{"why":"Source of the six general-capability benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, WinoGrande) used for the capability group.","marker":"(Beeching et al., 2023)"},{"why":"Provides ELEPHANT and the AITA original-versus-flipped pair design that defines the moral-sycophancy metric.","marker":"(Cheng et al., 2026)"},{"why":"Prior evidence that identical prompts behave differently in stateless API calls versus personalized chat sessions, the concern this work turns into a benchmark-level measurement.","marker":"(Wang et al., 2025)"},{"why":"Circulated public system prompts used to approximate the hidden interface instruction stack in the system-prompt ablation.","marker":"(Neumann et al., 2026)"},{"why":"Definition of test-retest agreement as a repeated-measures consistency metric, which becomes the paper's reliability headline.","marker":"(Berchtold, 2016)"},{"why":"The false-discovery-rate procedure applied across the 63 cell-level accuracy tests and the system-prompt contrasts.","marker":"(Benjamini and Hochberg, 1995)"},{"why":"Supplies the AA-Omniscience benchmark and its detailed LLM grading template used for cross-domain knowledge reliability.","marker":"(Jackson et al., 2025)"}],"fun_headline_variants":["API benchmarks overstate chatbot accuracy by 3.4 points","Identical prompts score 3.4 points higher via API than chat","Switching from API to chatbot drops accuracy like model downgrade","API scores mislead: chatbots rank differently, gap hits 28 points","Benchmark gap: API evaluations don't match deployed chatbots"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise, which the paper itself flags in its limitations section, is that each interface system and its matched API identifier actually serve the same underlying model checkpoint; if providers route different model variants to different surfaces, part or all of the measured gap could be model difference rather than an effect of the interface layer.","fun_headline_variants_meta":{"raw":{"variants":["API benchmarks overstate chatbot accuracy by 3.4 points","Identical prompts score 3.4 points higher via API than chat","Switching from API to chatbot drops accuracy like model downgrade","API scores mislead: chatbots rank differently, gap hits 28 points","Benchmark gap: API evaluations don't match deployed chatbots"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"cost":0.000200088,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000200088,"upstream_inference_prompt_cost":0.000116648,"upstream_inference_completions_cost":8.344e-05},"total_tokens":1336,"prompt_tokens":1038,"completion_tokens":298,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":208}},"tokens_in":1038,"tokens_out":298,"cost_source":"openrouter_usage","duration_ms":4561,"temperature":1.0,"reasoning_tokens":208,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T07:47:30.700966+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"falsifier":"The decisive check is a checkpoint-exact replication: run the same nine-benchmark audit with one model snapshot served behind both surfaces, either by a provider that locks and discloses the checkpoint for both access paths or by a self-hosted deployment of open-weight models behind a custom interface. If the 3.4-point accuracy gap and the 2.1-point test-retest gap shrink to zero under identical checkpoints, the context-validity claim fails; a partial confirmation would come from provider trace logs showing interface requests reaching the same model identifier and inference configuration as API requests while the gap persists.","supporting_citations":[],"review_version":1}