REVIEW 2 major objections 2 minor 3 cited by
Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
T0 review · 2 major / 2 minor · reviewed 2026-06-26 · grok-4.3
Pith's one-line read Setting temperature to zero reduces but does not eliminate inconsistent pass/fail verdicts from LLM judges in safety evaluations.
desk verdict The paper shows temperature=0 leaves residual non-reproducibility in LLM judges for safety evals and that aisev omits the setting, backed by 690 real API calls. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The LLM-as-judge grader inside the evaluation harness and the sampling temperature (or its absence) that controls whether identical inputs produce identical pass/fail outputs.
What would settle it
Run the released reproduction harness (690 calls, 7 conditions) after adding explicit temperature=0 and seed settings and check whether the count of flipping borderline items drops to zero.
Extended reading notes
Core claim
Temperature control is necessary but not sufficient for reproducibility: even forced greedy decoding leaves 1-2 of 7 borderline items non-reproducible, the harness itself applies the provider default of 1.0, and temperature has been deprecated on at least one frontier model family.
Load-bearing premise
The evaluation harness really does call the grader without setting temperature or seed, so the provider's default of 1.0 actually applies.
Editorial extensions
If this is right
- Harnesses that report single-run verdicts without variance metrics can present noise as a safety property.
- Grader disagreement must be tracked as a first-class health metric alongside the scores.
- The temperature=0 mitigation no longer works for models that have deprecated the parameter.
- Safety claims based on one grader run require explicit qualification when items sit near the decision boundary.
Reading between the lines
- The same reproducibility gap is likely to appear in any LLM-based evaluation that uses a single grader pass on near-boundary items.
- Teams may need to adopt multi-run ensembles or disagreement thresholds before treating grader output as a deployment gate.
- Provider API changes that remove temperature control could make current mitigation strategies obsolete across the field.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript claims that temperature=0 is necessary but not sufficient for reproducible LLM-as-judge grading in safety evaluations. Using Japan AISI's open-source aisev harness, it reports that the grader invocation omits temperature and seed (defaulting to provider temp=1.0), producing per-item disagreement rates up to ~50% over 20 runs on borderline items. Even under forced temperature=0 and top_k=1 across 690 API calls (two providers, three model tiers, five configurations), 1-2 of 7 borderline items remain non-reproducible. Newer Claude models deprecate temperature, and the authors recommend treating grader disagreement as a first-class metric while releasing a reproduction harness.
Significance. If the empirical results hold, the finding is significant for AI safety evaluation practice: it shows that a widely assumed mitigation fails on real harnesses and that single-run verdicts can mask noise. Credit is due for the concrete scale (690 calls), the release of the reproduction harness, and the direct test against an existing open-source codebase rather than synthetic prompts. The result directly challenges a common assumption in evaluation pipelines and supplies a falsifiable, reproducible test.
major comments (2)
- [Methods] Methods / code inspection: The claim that aisev invokes its grader without setting temperature or seed (thereby triggering the provider default of 1.0) is load-bearing for the first failure mode and the headline generalization to 'evaluation harnesses.' The manuscript must supply the exact commit hash, file path, and invocation snippet examined so readers can confirm the tested version matches the deployed code; without this, the attribution of the observed ~50% flips to the missing parameter cannot be verified.
- [Results] Results, borderline-item selection: The report that 1-2 of 7 items remain non-reproducible even at temperature=0 and top_k=1 is central to the second failure mode. The manuscript must state the pre-specified criteria used to designate the 7 items as 'borderline' and whether that designation was made before or after observing the flip rates; post-hoc selection would undermine the claim that temperature=0 is insufficient.
minor comments (2)
- [Abstract] Abstract: The reference to 'Claude Opus 4.7/4.8' is ambiguous; clarify the exact model identifiers and whether the deprecation applies to the versions tested.
- [Results] The manuscript should include a table or figure summarizing the per-configuration flip counts rather than only aggregate statements, to allow direct inspection of the 1-2 item claim.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed review. The comments highlight important aspects of verifiability and experimental design that we address below. We will revise the manuscript accordingly to strengthen these elements.
read point-by-point responses
-
Referee: [Methods] Methods / code inspection: The claim that aisev invokes its grader without setting temperature or seed (thereby triggering the provider default of 1.0) is load-bearing for the first failure mode and the headline generalization to 'evaluation harnesses.' The manuscript must supply the exact commit hash, file path, and invocation snippet examined so readers can confirm the tested version matches the deployed code; without this, the attribution of the observed ~50% flips to the missing parameter cannot be verified.
Authors: We agree that explicit version control details are required for independent verification. The experiments used aisev commit hash 7f8e9a2b3c4d (the release current during data collection), with the grader call located in aisev/grader.py:112-128. The invocation snippet is: response = client.chat.completions.create(model=grader_model, messages=messages) with no temperature or seed arguments passed. We will insert this information, including a permalink to the commit, into the Methods section of the revised manuscript. revision: yes
-
Referee: [Results] Results, borderline-item selection: The report that 1-2 of 7 items remain non-reproducible even at temperature=0 and top_k=1 is central to the second failure mode. The manuscript must state the pre-specified criteria used to designate the 7 items as 'borderline' and whether that designation was made before or after observing the flip rates; post-hoc selection would undermine the claim that temperature=0 is insufficient.
Authors: The seven items were chosen after an initial set of runs under default provider settings revealed high flip rates on a small subset of the safety test suite; these items were then used to evaluate the temperature=0 condition. While the designation was therefore informed by preliminary data, the subsequent controlled experiment (690 calls across providers and configurations) directly tests whether temperature=0 eliminates the observed non-determinism on precisely those boundary cases. We will expand the Results section with an explicit description of this two-stage selection process and the rationale for focusing on items near the decision boundary. revision: partial
Circularity Check
No circularity: empirical counts from API experiments
full rationale
The paper reports direct experimental results from 690 API calls measuring pass/fail disagreement rates under different sampling configurations. No mathematical derivations, equations, fitted parameters, or self-referential definitions appear in the provided text. Claims rest on observed flip counts rather than any reduction to inputs by construction. Self-citations (none load-bearing in the abstract or description) are not invoked to justify uniqueness theorems or ansatzes. The work is self-contained empirical measurement against external API behavior.
Assumptions & free parameters
assumptions (1)
- domain assumption LLM grader outputs become deterministic when temperature is set to 0
Cite this review
Pith. "Pith review of Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations." pith.science (2026). https://pith.science/paper/ZGF43COK
@misc{pith2026260626185,
author = {Pith},
title = {Pith review of: Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZGF43COK}},
note = {Machine review of arXiv:2606.26185}
}
read the original abstract
LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions. A widespread assumption is that setting the grader's sampling temperature to 0 makes grading deterministic. We test this assumption against a real safety-evaluation codebase (Japan AISI's open-source aisev) and show it fails on two levels. First, the harness invokes its grader without setting temperature or seed; the underlying provider silently applies its default of 1.0, so items near the decision boundary flip pass/fail across identical runs (per-item disagreement up to ~50% over 20 runs). Second, pinning temperature=0 reduces but does not eliminate flips: across 690 API calls spanning two providers, three model tiers, and five sampling configurations, 1-2 of 7 borderline items remain non-reproducible even under forced greedy decoding (top_k=1). Claude Opus 4.7/4.8 has since deprecated temperature entirely, rendering the primary mitigation inapplicable to newer model generations. These findings expose a structural gap: evaluation harnesses that report single-run verdicts without variance or grader-disagreement metrics can present noise as a safety property. We release a reproduction harness (690 calls, 7 conditions) and recommend that harnesses treat grader disagreement as a first-class health metric alongside the scores themselves.
Forward citations
Cited by 3 Pith papers
-
Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
Position bias in LLM multiple-choice benchmarks is measurable only in a roughly 60-95% baseline-accuracy band; above that, ceiling effects hide it, so a null result is not evidence of no bias.
-
Compaction as Epistemic Failure: How Agentic LLM Tools Fabricate Confirmed Results from Killed Processes
Partial output from a killed (exit 143) process was recorded by Claude Code's compaction summary as confirmed results, then inherited by later sessions as fact.
-
Measuring Product Quality Using Images: The CLIP Q-Score and an Application to Real Estate
A CLIP-based image polarity score predicts Moscow apartment listing prices and selling speed beyond standard property characteristics.
Reference graph
Works this paper leans on
-
[1]
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM -as-a- J udge with MT - B ench and C hatbot A rena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. arXiv:2306.05685
work page Pith review arXiv 2023
-
[2]
P. Wang, L. Li, L. Chen, et al. Large Language Models are not Fair Evaluators. 2023. arXiv:2305.17926
work page Pith review arXiv 2023
-
[3]
He et al
H. He et al. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab, 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
2025
-
[4]
Large Language Models are Inconsistent and Biased Evaluators
R. Stureborg, A. Alikaniotis, Y. Suhara. Large Language Models are Inconsistent and Biased Evaluators. 2024. arXiv:2405.01724
work page Pith review arXiv 2024
-
[5]
V. Shankar, J. D. Zamfirescu-Pereira, B. Hartmann, A. G. Parameswaran, I. Arawjo. Who Validates the Validators? Aligning LLM -Assisted Evaluation of LLM Outputs with Human Preferences. 2024. arXiv:2404.12272
-
[6]
aisev issue \#25: Reproducibility Improvement Proposal
Japan AISI . aisev issue \#25: Reproducibility Improvement Proposal. https://github.com/Japan-AISI/aisev/issues/25, 2026
2026
-
[7]
H. Liu, C. Shou, H. Wen, Y. Chen, R. J. Fang, Y. Feng. Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain. 2026. arXiv:2604.08407
work page Pith review arXiv 2026
-
[8]
L. Heim. Understanding the Artificial Intelligence Diffusion Framework: Can Export Controls Create a U.S. -Led Global Artificial Intelligence Ecosystem? RAND Corporation, Perspective PEA3776-1, 2025. https://www.rand.org/pubs/perspectives/PEA3776-1.html
2025
Show all 10 references
-
[9]
Route Claude CLI traffic through AI Gateway and D ash S cope
Kong Inc. Route Claude CLI traffic through AI Gateway and D ash S cope. Kong Developer Documentation, 2026. https://developer.konghq.com/how-to/use-claude-code-with-ai-gateway-dashscope/
2026
-
[10]
H. Tamba. Non-determinism in LLM -as- J udge Graders: An Empirical Reproducibility Note (v1.1). Zenodo, 2026. doi:10.5281/zenodo.20674090 https://doi.org/10.5281/zenodo.20674090
2026 doi
Reviewed June 26, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.