What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

Markus Schedl; Monorama Swain; Passant Shafaei; Shahed Masoudian

REVIEW 2 major objections 5 minor 16 references

What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Changing only the inference backend—from a direct HuggingFace generate call to a pipeline, LangChain wrapper, vLLM, or Ollama—can significantly change a model's benchmark scores, even when greedy decoding removes sampling noise.

desk verdict Solid controlled evidence that inference backend changes benchmark scores, though the headline 39% share is approximate and slightly entangled with output length. read the letter →

arxiv 2608.04714 v1 pith:CZQYCG2B submitted 2026-08-05 cs.SE cs.AIcs.LG

Shahed Masoudian , Passant Shafaei , Monorama Swain , Markus Schedl This is my paper

classification cs.SEcs.AIcs.LG

keywords LLMbenchmarkreproducibilityinferencebackendgreedydecodinggeneration-modeladderstructuraleffectframeworkdefaultsvvsHuggingFacereporting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Benchmark scores are typically reported as if they said something about the model alone. This paper argues that the inference backend—the software that loads the weights, tokenizes the prompt, and generates the answer—is itself a significant factor in those scores. In a fully crossed study of three instruction-tuned models, five inference frameworks, six benchmarks, and four generation modes, the authors show that even with greedy decoding and no sampling noise, swapping backends changes which items a model answers correctly. They estimate that the backend's structural contribution accounts for roughly 39% of the variance a practitioner sees with out-of-the-box default settings, with sampling noise and framework-specific defaults making up the rest. If this is right, published benchmark numbers are only meaningful when the backend, its version, and the full generation configuration are disclosed.

What carries the argument

The load-bearing device is the generation-mode ladder: four conditions that differ in exactly one source of variability at a time—Deterministic (greedy, temperature=0, no sampling), Fix (matched sampling hyper-parameters across backends), Token=256 (only max_new_tokens fixed, rest framework defaults), and Default (each backend's full default configuration). By subtracting variances across adjacent modes within each (model, benchmark) pair, the authors attribute the Deterministic-mode variance of 0.00074 to the structural backend effect, the increase to 0.00112 to sampling noise, and the final increase to 0.00190 to framework-default hyper-parameters. The companion measures are the per-item divergence from a fixed reference backend, the binary disagreement rate, and a 'severe' threshold of more than 5% of items flipped; these turn variance numbers into practical significance counts.

What would settle it

Re-run the Deterministic-mode comparison of vLLM against the HuggingFace reference on QWEN2.5-1.5B-INST, holding the chat template and tokenizer identical and forcing the same attention kernel on both backends. If the 0.025 mean divergence and 10-of-11 significant metrics disappear, the structural effect is an arithmetic-kernel artifact; if they persist, the effect lives in the backend's decoding logic rather than its kernels.

Watch

Extended reading notes

Core claim

The paper's central claim is that benchmark scores are a joint product of the model and the inference backend, not a property of the model alone. In a fully crossed experiment—three instruction-tuned 1B-scale models, five backends, six benchmarks, and four generation modes—the authors find that even under greedy decoding with no sampling noise, moving from a direct HuggingFace generate call to a pipeline wrapper, a LangChain layer, vLLM, or Ollama changes which items a model answers correctly. The structural backend component alone accounts for a within-cell variance of 0.00074, which is roughly 39% of the 0.00190 variance seen under out-of-the-box default settings; sampling noise and framework-specific defaults add the rest. The effect is strongly model-dependent: LLAMA-3.2-1B-INST shows mean absolute divergences up to 0.055 and per-item disagreement up to 48%, while GEMMA-3-1B-IT stays below 0.010 divergence on almost every task under deterministic decoding. Divergences are larger on factual benchmarks than on social-bias benchmarks, and most severe cases lie above the y=x line, meaning aggregate score differences understate how many individual answers change.

Load-bearing premise

The 39% figure stands on the assumption that each generation mode adds exactly one new source of variability, so subtracting variances between adjacent modes cleanly separates the backend's own effect from sampling noise and framework defaults.

Editorial extensions

If this is right

Published benchmark numbers should be read as a joint property of the model, backend, backend version, and generation configuration, not of the model alone.
Cross-backend comparisons should use deterministic greedy decoding with matched generation parameters; otherwise avoidable sampling and default-configuration variance is mixed into the comparison.
Because the backend effect is strongly model-dependent, a backend that is safe for one model family can change conclusions for another; reproducibility checks should cover the model family being claimed about.
Factual benchmarks are more likely than bias benchmarks to flip conclusions when the backend or generation defaults change, so factual claims need the strictest reporting.
Per-item disagreement can exceed the aggregate score difference, so averaged scores can hide how many individual answers a backend change actually flips.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 39% share is best read as an upper-bound estimate: the paper never measures interaction terms, so interactions between backend behavior and sampling settings could shift the true structural share in either direction.
If the deterministic-mode divergence at 1B scale comes largely from kernel arithmetic and tokenizer differences, the same protocol on larger models and longer outputs (code, math, multi-turn) could show larger backend effects than the paper observes.
A natural follow-up experiment is a temperature and top-p sweep under matched settings; if backend divergence grows faster than sampling noise alone, averaging a few random samples will not make benchmark numbers backend-agnostic.
The paper's reporting recommendation implies a practical standard: every leaderboard entry should pin backend name, version, and generation configuration, just as software builds pin dependency versions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. This paper gives the most carefully controlled evidence I've seen that the inference backend is a genuine factor in LLM benchmark scores, visible even under greedy decoding. The second thing: the headline 39% structural-backend share is an approximation and could shift under re-analysis, but the central claim survives regardless.

What's new: the fully-crossed design (3 models × 5 backends × 6 benchmarks × 4 generation modes) and the generation-mode ladder that separates structural, sampling, and default-hyperparameter contributions. Pape et al. already showed backends matter; this paper adds the cross factors, per-item disagreement rates up to 40%, and the task-family asymmetry (factual tasks more sensitive than bias tasks). That's a real advance for the reproducibility literature.

What it does well: fixed hardware and precision, matched parameter sets, BH-corrected paired tests, five repetitions for stochastic modes, and an honest handling of the Ollama GGUF confound (treated as part of the backend cost, not removed). The appendices are thorough — effect sizes, agreement, clustering, and a dedicated length-confounding analysis. The citation pattern is honest: Pape et al. is credited and the incremental contribution is stated precisely.

The soft spots are proportionate. The 39% is Var(Deterministic)/Var(Default), computed by subtracting variances across adjacent modes. As Section 6 admits, this assumes the modes form an additive ladder, and that assumption is untested. The stress-test concern is fair: Appendix G shows Default-mode differences are entangled with output length on exactly the ROUGE/F1 metrics with the largest divergences, so the structural/default separation is partly a length effect. The 39% might move materially. Also, code and data aren't released yet, there are small internal inconsistencies (30 vs 33 comparisons; Token=256 vs Token=512 in figure labels), and the headline number has no uncertainty estimate. None of this undermines the main message.

Who's it for: anyone running or citing LLM benchmarks. It deserves a serious referee — conditional accept, with requested code/data release, fixed reporting, and ideally a within-mode re-decomposition to check the 39%. I'd cite it.

Referee Report

2 major / 5 minor

Summary. This paper reports a fully-crossed empirical study of how the choice of inference backend affects LLM benchmark scores. Three instruction-tuned models around 1B parameters (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-3-1B) are evaluated on six benchmarks (MMLU, TriviaQA, TruthfulQA MC1/Gen, BBQ, StereoSet) across five backends (hf_raw, hf_pipeline, langchain_hf, vllm, ollama) under four generation modes (Deterministic, Fix, Token=256, Default), with hardware, precision, and prompts held constant. The authors find that backends diverge even under greedy, sampling-free decoding; that the divergence is strongly model-dependent (Llama most affected, Gemma least); that per-item answer flips often exceed 5% (Figure 1, Table 4); and that a mode-by-mode variance decomposition (Table 3) attributes roughly 39% of the Default-mode variance to the backend (0.00074/0.00190), with sampling noise and framework defaults accounting for the rest. They conclude that benchmark numbers are not backend-agnostic and recommend disclosing the backend, its version, and the full generation configuration, and using greedy decoding for cross-backend comparison.

Significance. The qualitative central finding — that the inference backend changes per-item answers under greedy decoding, with strong model dependence — is convincing and, if it generalizes beyond the 1B scale and single GPU, directly challenges the common practice of reporting benchmark numbers without backend disclosure. The design is a genuine strength: a controlled, fully-crossed setup with fixed hardware (RTX 3090), fixed 16-bit precision, and fixed prompt formatting; five repetitions in the stochastic modes; a clear hf_raw baseline with the hf_pipeline no-op sanity check; BH-corrected paired significance tests (McNemar/Wilcoxon); effect sizes; and a per-item disagreement and severity analysis that goes beyond mean-score differences. The task-family split (factual vs. social-bias) and the appendix analyses (Cohen's kappa, Jaccard error-set overlap, PCA, response-length diagnostics) add interpretive value. The quantitative headline — the roughly 39% variance share — is currently the weakest link and needs the additional analysis described in the major comments.

major comments (2)

[Section 4.2, Table 3] The headline claim that the structural backend effect accounts for 'roughly 39%' of Default-mode variance (0.00074/0.00190) requires the four generation modes to behave as an additive ladder, i.e., Var(Default) = Var(Deterministic) + sampling-noise variance + default-hyperparameter variance with zero covariance and no backend-by-mode interaction. None of these conditions is tested, and Section 6 concedes the ladder is 'an approximation rather than an exact decomposition.' The entanglement is concrete: the largest single increment (Default minus Token=256, 0.00072) is attributed to default hyper-parameters, but Table 6 shows Default max_new_tokens differs across backends (HF family 20, vLLM 16, Ollama unbounded), and Appendix G identifies response length as a confound for ROUGE-L/F1 on exactly the generative tasks with the largest divergences (Figure 4 shows TruthfulQA-Gen divergences of -0.17 to -0.25 in Default mode). In addition, the Deterministic variance rests on a single greedy run per cell (Section 6), so the ratio 0.00074/0.00190 is reported without any uncertainty estimate. The five repetitions collected in the stochastic modes enable a direct test — e.g., between-backend variance of repetition means versus within-backend variance across repetitions, or a backend-by-mode mixed model with an interaction term — and a length-matched variant of Default would show how much of the 39% is an artifact of unmatched max_new_tokens. The paper should run those checks or, at minimum, present the 39% as a conditional estimate with an uncertainty bound rather than as the paper's headline number.
[Section 3.3, Fix mode] The Fix-minus-Deterministic variance increment is labeled 'sampling noise,' but identical parameter names need not imply identical sampling distributions across the backends, since vLLM 0.8.5 and llama.cpp/Ollama implement decoding and sampling kernels independently of HuggingFace generate(). The increment therefore also absorbs backend-specific sampling-implementation differences and their interactions with the structural effect, so the row labels in Table 3 (and the '61%' remainder in Section 4.2) overstate the share attributable purely to irreducible sampling noise and to default parameters. A within-mode partition using the five repetitions — within-backend variance across repetitions versus between-backend variance of repetition means — would resolve this and should be reported before the decomposition claims are made.

minor comments (5)

[Section 3.6] Specify how the five responses per item in the stochastic modes are reduced for the per-item paired significance tests and the disagreement-rate computation (e.g., mean, majority vote, or a single draw); Tables 1, 2, and 4 depend on this rule.
[Table 2 caption] The caption says 'out of 30 primary-metric comparisons (3 models x 11 metrics)', but the product is 33 and the table header reads 'out of 33'; correct the caption (the same inconsistency appears in Table 7's 'six metrics').
[Appendix Figures 2, 5, 7, 8] These figures label the mode 'Token=512', while the main text and Table 3 use 'Token=256'; use a single consistent label throughout.
[Abstract and Section 4.3] The abstract states that divergences are 'more pronounced on factual than on social-bias benchmarks' without qualification, but Table 5 shows the reverse ranking for LLAMA-3.2-1B-INST and GEMMA-3-1B-IT under Deterministic decoding (bias 0.0368 vs factual 0.0123; bias 0.0040 vs factual 0.0025); qualify the abstract statement by mode.
[Throughout] There are several typos and wording slips that should be cleaned up: 'This word addresses the gap' (Section 1), 'quantify investigate' (Section 1), 'models are suing the exact same model' (Section 4.1), 'differntiate' (Section 2), 'paramters' (Section 5), and Section 3.2's 'For the four HuggingFace-ecosystem backends namely hf_raw, hf_pipeline, langchain and for vllm', which is confusing because vLLM is not a HuggingFace-ecosystem wrapper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline quantities are direct measurements, the 39% share is a descriptive ratio of variances measured in the same experiment, and the mode-ladder caveat is a validity limitation, not a self-referential derivation.

full rationale

The paper's derivation chain is empirical rather than formal. The central claim — that backend choice changes benchmark scores even under greedy decoding — is established by directly measuring per-item divergence from a fixed hf_raw baseline (Section 3.6, Tables 1 and 2), with no parameter fitted to a subset of data and then relabeled as a prediction. The headline 39% figure is a ratio of two variances (0.00074 / 0.00190) computed from the same experimental cells under different generation modes (Section 4.2, Table 3); it is a descriptive attribution, not the output of an inverse problem whose solution was baked into its inputs. The ladder separation between structural effects, sampling noise, and default hyper-parameters is an untested additivity assumption, and the paper itself concedes in Section 6 that the decomposition is 'an approximation rather than an exact decomposition'; but an admitted approximation is a correctness or validity risk, not circularity, because the 39% figure is not defined in terms of the conclusion it supports. The 5% severity threshold is borrowed from external prior work (Mekala et al., 2025; Sclar et al., 2024), and no load-bearing argument rests on a self-citation: the reference list contains no works by the present authors, and the closest related work (Pape et al., 2026) is authored by others. The potential response-length confound for ROUGE/F1 metrics (Appendix G) concerns whether some divergences are caused by length mismatch; it does not make any measured quantity equivalent to its own input by construction. Overall, the paper is a self-contained controlled measurement, so the circularity burden is minimal.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a small number of domain assumptions, most importantly the additive decomposition of variance across generation modes and the treatment of hf_raw as baseline. There are two hand-chosen analysis choices (severity threshold, composite-score calibration), no fitted generative-model parameters, and no invented entities. The GGUF weight confound is acknowledged and folded into the backend cost.

free parameters (2)

Severity threshold for per-item disagreement = 0.05
Used in Section 3.6 to classify a backend-mode divergence as severe; threshold borrowed from prior work (Mekala et al. 2025; Sclar et al. 2024) but still a hand-chosen cutoff that determines the severe-configuration counts in Table 4.
Calibration mappings for composite metrics = 1 - raw for invalid_rate and ambig_bias_score; 1 - 2|raw - 0.5| for ss
Section 3.5 defines nonlinear mappings to a common [0,1] scale for metrics with non-standard ideal values. The shape of these mappings affects all variance and divergence numbers, including the 39% share, and is chosen by the authors rather than derived.

assumptions (5)

domain assumption The generation-mode ladder (Deterministic, Fix, Token=256, Default) cleanly and additively separates backend, sampling, and default-parameter variance.
Section 3.3 defines the ladder; Section 4.2 uses differences in variance across modes to attribute shares; Section 6 acknowledges this is an approximation, not an exact decomposition. If interactions are nontrivial, the 39% backend share is not a valid causal attribution.
domain assumption hf_raw is an appropriate reference baseline; all divergences are relative and there is no external ground truth for 'correct' backend behavior.
Section 3.6 defines divergence from hf_raw; Section 6 notes all divergences are relative. The paper's recommendations assume that the differences are meaningful even without ground truth.
domain assumption GGUF weight conversion for Ollama is part of the backend cost rather than a confound.
Section 3.2 states that Ollama's fp16 GGUF weights differ numerically from the other backends' SafeTensors and that this is 'part of the backend cost.' This assumption is needed to interpret Ollama divergences as backend effects rather than as weight-representation artifacts.
domain assumption The fixed computational environment (RTX 3090, CUDA 12.6, PyTorch 2.6, fp16) makes the observed differences attributable to backend rather than environment.
Section 3.1 states all experiments share hardware and numerical configuration. The paper relies on this to isolate backend effects, while acknowledging in Section 6 that the absolute magnitudes may differ on other accelerators.
standard math Standard paired statistical tests (McNemar, Wilcoxon) with Benjamini-Hochberg correction are appropriate for per-item score comparisons.
Section 3.6 describes the significance testing; this relies on standard assumptions of paired tests and multiple-testing control.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend." pith.science (2026). https://pith.science/paper/CZQYCG2B

@misc{pith2026260804714,
  author       = {Pith},
  title        = {Pith review of: What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZQYCG2B}},
  note         = {Machine review of arXiv:2608.04714}
}

read the original abstract

Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39\%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.

Figures

Figures reproduced from arXiv: 2608.04714 by the authors.

**Figure 1.** |∆| from hf_raw (x axis) versus the per-item disagreement rate (y axis) for each (model, backend, dataset, mode) cell. Marker shape encodes the backend and colour the benchmark. The shaded quarter-circle (radius 0.05) is the “not-severe” zone and a divergence that flips fewer than 5% of items. The dashed line is y = x where rate of flips is equivalent to |∆| score. The variance decomposition establishes that backend… view at source ↗

**Figure 2.** Average divergence from hf_raw on the calibrated primary metric of each benchmark, for each generation mode. Note that contrary to main body of paper here we report the ∆ instead of |Delta| to capture negative and positive influences. ∗ = at least one backend BH-adjusted significant (p < 0.05). MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-RcBBQ-AccBBQ-DisBBQ-UnkBBQ-Bias SS-LMS SS-Inv SS-SS SS-ICAT Metric… view at source ↗

**Figure 3.** Signed divergence from hf_raw (green = backend higher, red = lower) per backend (rows) and task (columns), one panel per model family in Deterministic mode. ∗ marks a task with at least one BH-significant metric. MMLU TriviaQA-EM TriviaQA-F1 TruthfulQA TQAGen-Rb TQAGen-RcBBQ-AccBBQ-DisBBQ-UnkBBQ-Bias SS-LMS SS-Inv SS-SS SS-ICAT Metric hf_pipeline langchain_hf vllm ollama Backend -0.00 -0.00 -0.00 -0.00 -0.13* -0.16*… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Signed divergence from hf_raw per backend (rows) and task (columns), one panel per model family in Default mode. Divergences are larger than under Deterministic decoding, especially for vllm and ollama on Llama’s factual tasks. hf_pipeline langchain_hf vllm ollama Gemm…

**Figure 5.** Figure 5: Cohen’s κ between each non-baseline backend and hf_raw for correct/wrong answer. Values are averaged over all conditions not held constant [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Jaccard similarity of the error sets (incorrectly answered questions) between each non-baseline backend [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 7.** Figure 7: PCA of per-question calibrated score vectors (one vector per backend per model [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Mean response length (characters) as compound heatmaps: model [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

[4]

D Inter-Backend Agreement Analysis To complement aggregate mean-divergence met- rics, we assess whether backends agree onwhich questionsa modelanswers correctly

the cells deepen most sharply for vllm and ollama on Llama’s factual tasks reflecting the framework-default sampling gaps and added sampling noise discussed in Sec- tion 4.3. D Inter-Backend Agreement Analysis To complement aggregate mean-divergence met- rics, we assess whether backends agree onwhich questionsa modelanswers correctly. For each (model, mod...

work page
[7]

InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8086–8098

Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8086–8098. Associa- tion for Computational Linguistics. Anmol Mekala, Anirudh Atmakuru, Yixi...

work page 2022
[9]

Stereoset: Measuring stereotypical bias in pretrained language models. InProceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference on Nat- ural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5356–5371. Association for Computat...

work page 2021
[11]

InFind- ings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, volume ACL 2022 ofFindings of ACL, pages 2086–2105

BBQ: A hand- built bias benchmark for question answering. InFind- ings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, volume ACL 2022 ofFindings of ACL, pages 2086–2105. Asso- ciation for Computational Linguistics. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr

work page 2022
[12]

InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

work page 2024
[13]

InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 ofProceedings of Machine Learning Research, pages 50043–50064

One prompt is not enough: Automated construction of a mixture-of-expert prompts. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 ofProceedings of Machine Learning Research, pages 50043–50064. PMLR / OpenReview.net. Martijn Wieling, Josine Rawee, and Gertjan van No- ord

work page 2024
[14]

Understanding and mitigating numer- ical sources of nondeterminism in LLM inference. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, De- cember 2-7, 2025 / Mexico City, Mexico, November 30 - December 5,

work page 2025
[15]

Calibrate before use: Improving few-shot performance of language models. InProceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 ofProceedings of Machine Learning Re- search, pages 12697–12706. PMLR. Table 6: Default-mode generation parameters, per backend and model. Backend Modeltemp...

work page 2021

Show all 16 references

[2017]

InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601–1611

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601–1611. Assoc...

2017
[2020]

Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek V ossen, and Nuno Freire

Fine-tuning pretrained language models: Weight initial- izations, data orders, and early stopping.arXiv preprint arXiv:2002.06305. Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek V ossen, and Nuno Freire

2002 arXiv
[2021]

In9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7,

Measuring massive multitask language under- standing. In9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7,

2021
[2022]

InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252

Truthfulqa: Measuring how models mimic human false- hoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics....

2022
[2023]

InProceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–626

Efficient mem- ory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–626. ACM. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipr...

2023
[2024]

Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach

Lessons from the Trenches on Reproducible Evaluation of Language Models.arXiv preprint arXiv:2405.14782. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach

arXiv
[2025]

Association for Computa- tional Linguistics

Does quan- tization affect models’ performance on long-context tasks? InProceedings of the 2025 Conference on Em- pirical Methods in Natural Language Processing, pages 9422–9470, Suzhou, China. Association for Computa- tional Linguistics. Moin Nadeem, Anna Bethke, and Siva Reddy

2025
[2026]

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R

The silent hy- perparameter: Quantifying the impact of inference backends on LLM reproducibility.arXiv preprint arXiv:2605.19537. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [4]

D Inter-Backend Agreement Analysis To complement aggregate mean-divergence met- rics, we assess whether backends agree onwhich questionsa modelanswers correctly

the cells deepen most sharply for vllm and ollama on Llama’s factual tasks reflecting the framework-default sampling gaps and added sampling noise discussed in Sec- tion 4.3. D Inter-Backend Agreement Analysis To complement aggregate mean-divergence met- rics, we assess whether backends agree onwhich questionsa modelanswers correctly. For each (model, mod...

work page

[2] [7]

InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8086–8098

Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8086–8098. Associa- tion for Computational Linguistics. Anmol Mekala, Anirudh Atmakuru, Yixi...

work page 2022

[3] [9]

Stereoset: Measuring stereotypical bias in pretrained language models. InProceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference on Nat- ural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5356–5371. Association for Computat...

work page 2021

[4] [11]

InFind- ings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, volume ACL 2022 ofFindings of ACL, pages 2086–2105

BBQ: A hand- built bias benchmark for question answering. InFind- ings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, volume ACL 2022 ofFindings of ACL, pages 2086–2105. Asso- ciation for Computational Linguistics. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr

work page 2022

[5] [12]

InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

work page 2024

[6] [13]

InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 ofProceedings of Machine Learning Research, pages 50043–50064

One prompt is not enough: Automated construction of a mixture-of-expert prompts. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 ofProceedings of Machine Learning Research, pages 50043–50064. PMLR / OpenReview.net. Martijn Wieling, Josine Rawee, and Gertjan van No- ord

work page 2024

[7] [14]

Understanding and mitigating numer- ical sources of nondeterminism in LLM inference. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, De- cember 2-7, 2025 / Mexico City, Mexico, November 30 - December 5,

work page 2025

[8] [15]

Calibrate before use: Improving few-shot performance of language models. InProceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 ofProceedings of Machine Learning Re- search, pages 12697–12706. PMLR. Table 6: Default-mode generation parameters, per backend and model. Backend Modeltemp...

work page 2021