Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality

Andrew Ilyas; Bangya Liu; Gokul Swamy; Kimberly Le Truong; Longtian Ye; Minghao Liu; Minglai Yang; Riccardo Fogliato; Steven Wu; Weijian Zhang

arxiv: 2604.04418 · v2 · submitted 2026-04-06 · 💻 cs.HC · cs.AI

Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality

Xiaoyuan Zhu , Kimberly Le Truong , Riccardo Fogliato , Gokul Swamy , Weijian Zhang , Minglai Yang , Longtian Ye , Bangya Liu

show 3 more authors

Minghao Liu Andrew Ilyas Steven Wu

This is my paper

Pith reviewed 2026-05-10 19:54 UTC · model grok-4.3

classification 💻 cs.HC cs.AI

keywords error verifiabilityLLM justificationshuman evaluationmathematical reasoningfactual QAmodel scalingpost-trainingresponse quality

0 comments

The pith

Error verifiability is a distinct LLM quality dimension that does not improve with scaling or post-training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper defines error verifiability as the property that lets model justifications help users accurately judge whether an answer is correct or incorrect. It introduces a balanced metric v_bal that scores this property and validates it against human raters who agree strongly on answer correctness. Standard improvements such as larger models or further training leave verifiability unchanged. The authors instead present reflect-and-rephrase for mathematical reasoning and oracle-rephrase for factual QA; both raise verifiability by supplying domain-appropriate external information. The results show that verifiability requires dedicated, task-specific techniques rather than general accuracy gains.

Core claim

Error verifiability is formalized as the capacity of justifications to enable raters to distinguish correct answers from incorrect ones. The balanced metric v_bal quantifies this capacity and is validated by high-agreement human judgments. Neither post-training nor model scaling raises v_bal scores. Reflect-and-rephrase for mathematical reasoning and oracle-rephrase for factual QA both succeed by incorporating suitable external information. The work therefore treats error verifiability as a response-quality dimension separate from accuracy that demands domain-aware interventions.

What carries the argument

error verifiability, defined as the degree to which model justifications enable human raters to correctly assess answer correctness, quantified by the balanced metric v_bal

If this is right

Common scaling and post-training leave error verifiability unchanged across tested settings.
Reflect-and-rephrase improves verifiability for mathematical reasoning by adding external information.
Oracle-rephrase improves verifiability for factual QA by adding external information.
Verifiability forms a quality dimension that does not follow automatically from higher accuracy.
Domain-aware methods are required to raise verifiability in high-stakes deployments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Evaluation suites for LLMs should add verifiability tests alongside accuracy metrics.
Similar rephrasing techniques could be tested in domains such as code or medical advice.
Systems that automatically surface external context might reduce user errors in real deployments.
The gap between convincing and verifiable responses points to continued risk when users lack verification aids.

Load-bearing premise

High-agreement human judgments on answer correctness provide a sufficient test that the justifications truly help distinguish right answers from wrong ones.

What would settle it

A replication study in which human raters using justifications from reflect-and-rephrase or oracle-rephrase achieve no higher accuracy at identifying correct answers than with baseline justifications would falsify the reported improvements.

Figures

Figures reproduced from arXiv: 2604.04418 by Andrew Ilyas, Bangya Liu, Gokul Swamy, Kimberly Le Truong, Longtian Ye, Minghao Liu, Minglai Yang, Riccardo Fogliato, Steven Wu, Weijian Zhang, Xiaoyuan Zhu.

**Figure 1.** Figure 1: vbal and accuracy across post-training checkpoints for Tulu3.1-8B and OLMo2-7B. 0.30 0.60 0.90 0.60 0.75 vbal GSM8K 0.30 0.60 0.90 MATH500 0.30 0.60 0.90 MMLU 0.30 0.60 0.90 MMLU-Pro 0.30 0.60 0.90 TruthfulQA Accuracy DeepSeek-V3 Llama3.1-8b Llama4-Maverick OLMo2-7b Qwen3-8b Tulu3.1-8b grok-4 [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

**Figure 2.** Figure 2: Accuracy vs. vbal across models and datasets. 7 Does Improving Model Capability Improve Verifiability? We examine whether standard approaches to improving model capability, namely posttraining (§7.1) and model scaling (§7.2), also improve verifiability. 7.1 Post-Training Does Not Consistently Improve Verifiability Setup. To isolate the effect of post-training on verifiability, we track two open-weight mod… view at source ↗

**Figure 3.** Figure 3: Effect of calibrating linguistic confidence on [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

read the original abstract

As LLMs are deployed in high-stakes settings, users must judge the correctness of individual responses, often relying on model-generated justifications such as reasoning chains or explanations. Yet, no standard measure exists for whether these justifications help users distinguish correct answers from incorrect ones. We formalize this idea as error verifiability and propose $v_{\text{bal}}$, a balanced metric that measures whether justifications enable raters to accurately assess answer correctness, validated against human raters who show high agreement. We find that neither common approaches, such as post-training and model scaling, nor more targeted interventions recommended improve verifiability. We introduce two methods that succeed at improving verifiability: reflect-and-rephrase (RR) for mathematical reasoning and oracle-rephrase (OR) for factual QA, both of which improve verifiability by incorporating domain-appropriate external information. Together, our results establish error verifiability as a distinct dimension of response quality that does not emerge from accuracy improvements alone and requires dedicated, domain-aware methods to address.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Error verifiability is framed as a distinct LLM quality axis that scaling misses, with two simple domain-specific fixes that move the needle where standard approaches do not.

read the letter

The main thing here is that the paper treats error verifiability as its own property—whether justifications actually help users separate correct answers from incorrect ones—and shows that scaling and post-training leave it flat while two targeted interventions improve it. They formalize the concept, define the balanced v_bal metric, and validate it against human raters who reach high agreement on correctness judgments. The reflect-and-rephrase method for math and oracle-rephrase for factual QA both work by injecting relevant external information, and the negative results on common techniques are documented clearly enough to be useful. This is the part that lands: it gives practitioners something concrete to try instead of waiting for bigger models to solve verification on their own. The human validation step adds grounding that pure synthetic metrics often lack. The soft spot is the validation itself. Rater agreement on answer correctness does not automatically prove the justification is what enabled the distinction; raters could be relying on the answer text, surface cues, or their own knowledge. The abstract and stress-test note leave open whether the experiments included blinding to ground truth, domain-expertise screening, or ablations that remove the justification. If those controls are absent or weak in the full methods, the claim that verifiability is cleanly separate from accuracy loses force. Generalization beyond math and factual QA is also untested. This is for people who build or evaluate LLMs for settings where users must verify outputs themselves. Readers focused on explanation quality, trustworthiness metrics, or high-stakes deployment would find the metric and the intervention results worth reading. It has enough new formalization and testable claims to deserve a serious referee, mainly to check the controls and see how far the methods extend.

Referee Report

3 major / 2 minor

Summary. The paper introduces 'error verifiability' as a distinct dimension of LLM response quality, formalized as the balanced metric v_bal that quantifies whether model-generated justifications enable human raters to accurately judge answer correctness. It reports that neither model scaling nor post-training improves v_bal, while two new domain-aware interventions—reflect-and-rephrase (RR) for mathematical reasoning and oracle-rephrase (OR) for factual QA—succeed by incorporating external information. The metric is validated via high inter-rater agreement on correctness judgments, and the results are used to argue that verifiability does not emerge automatically from accuracy gains.

Significance. If the validation and intervention results hold under scrutiny, the work is significant for highlighting a gap in current LLM training and evaluation practices for high-stakes applications. It provides concrete evidence that verifiability is orthogonal to accuracy and offers targeted, domain-specific methods (RR and OR) that demonstrably improve it. The introduction of v_bal as an externally validated metric could influence future work on explanations and human-AI trust.

major comments (3)

[Abstract / Methods] Abstract and Methods: The central validation of v_bal rests on 'high agreement' among human raters judging answer correctness, yet no details are provided on experimental controls such as rater blinding to ground truth, domain-expertise screening, justification ablation conditions, or statistical tests for agreement (e.g., Cohen's kappa or Fleiss' kappa values). Without these, high agreement does not isolate whether raters are using the justification or their own knowledge/surface features of the answer, directly threatening the claim that v_bal measures justification utility distinctly from answer obviousness.
[Results] Results section (intervention comparisons): The claim that RR and OR improve verifiability while scaling/post-training do not is load-bearing, but requires explicit reporting of effect sizes, confidence intervals, and controls showing that accuracy does not improve in tandem (or that verifiability gains persist after accuracy matching). If accuracy also rises under RR/OR, the distinctness argument is weakened.
[Definition of v_bal] § on v_bal definition: The metric is described as 'balanced' and 'validated against human raters,' but the exact formula, balancing procedure, and how it differs from simple accuracy or agreement metrics must be shown with an equation; without it, reproducibility and the 'parameter-free' aspect cannot be assessed.

minor comments (2)

[Notation / Methods] Notation for v_bal and the two interventions (RR, OR) should be introduced with explicit equations or pseudocode in the main text rather than relying on prose descriptions.
[Figures/Tables] Figure or table captions for human evaluation results should include exact sample sizes, rater counts, and exclusion criteria to allow assessment of the reported high agreement.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed comments, which have identified key areas where the manuscript can be strengthened for clarity, rigor, and reproducibility. We address each major comment point by point below and will incorporate revisions as indicated.

read point-by-point responses

Referee: [Abstract / Methods] Abstract and Methods: The central validation of v_bal rests on 'high agreement' among human raters judging answer correctness, yet no details are provided on experimental controls such as rater blinding to ground truth, domain-expertise screening, justification ablation conditions, or statistical tests for agreement (e.g., Cohen's kappa or Fleiss' kappa values). Without these, high agreement does not isolate whether raters are using the justification or their own knowledge/surface features of the answer, directly threatening the claim that v_bal measures justification utility distinctly from answer obviousness.

Authors: We agree that the manuscript would benefit from expanded details on the human evaluation protocol to better substantiate the validation of v_bal and address potential confounds. In the revised manuscript, we will add a dedicated subsection in Methods describing: rater recruitment and domain-expertise screening criteria; blinding procedures (raters judged correctness using only the model justification, without access to ground truth or external verification tools in the primary condition); any justification ablation conditions tested; and quantitative inter-rater agreement statistics including Fleiss' kappa. We will also discuss how the design helps isolate justification utility from answer obviousness or surface features. These additions directly respond to the concern. revision: yes
Referee: [Results] Results section (intervention comparisons): The claim that RR and OR improve verifiability while scaling/post-training do not is load-bearing, but requires explicit reporting of effect sizes, confidence intervals, and controls showing that accuracy does not improve in tandem (or that verifiability gains persist after accuracy matching). If accuracy also rises under RR/OR, the distinctness argument is weakened.

Authors: We concur that stronger statistical reporting and controls are needed to support the orthogonality claim. The revised Results section will report effect sizes (e.g., Cohen's d) and 95% confidence intervals for v_bal changes under all conditions, including RR, OR, scaling, and post-training. We will also add explicit accuracy metrics for each intervention, with statistical comparisons showing that RR and OR yield minimal or no accuracy gains (consistent with our data). Where feasible, we will include accuracy-matched subgroup analyses to confirm verifiability improvements persist independently. These changes will reinforce that verifiability does not automatically follow from accuracy. revision: yes
Referee: [Definition of v_bal] § on v_bal definition: The metric is described as 'balanced' and 'validated against human raters,' but the exact formula, balancing procedure, and how it differs from simple accuracy or agreement metrics must be shown with an equation; without it, reproducibility and the 'parameter-free' aspect cannot be assessed.

Authors: We will revise the v_bal definition section to include the explicit mathematical formula presented as an equation. The updated text will detail the balancing procedure (normalizing performance across correct and incorrect answer cases to avoid bias from accuracy levels) and contrast v_bal with standard accuracy (which does not assess justification utility) and raw agreement metrics (which lack the human correctness judgment component). This will also highlight the parameter-free design. The revision ensures full reproducibility and addresses the assessment concern. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation of v_bal or improvement claims

full rationale

The paper proposes v_bal as an independently defined balanced metric for error verifiability, explicitly validated against external human rater judgments with high inter-rater agreement rather than derived from or fitted to the same data used for improvement claims. The two introduced methods (RR and OR) are presented as empirical interventions that incorporate domain-appropriate external information, with results compared against baselines like scaling and post-training; no equations, self-definitions, or load-bearing self-citations reduce these claims to inputs by construction. The distinction from accuracy is supported by direct empirical contrasts without renaming or smuggling via prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the new definition of error verifiability and the empirical effectiveness of RR and OR; the abstract supplies no free parameters, relies on the domain assumption that human raters provide ground truth for verifiability, and introduces error verifiability as a new entity without external falsifiable handles beyond the reported human study.

axioms (1)

domain assumption Human raters can reliably judge answer correctness when shown model justifications
v_bal is validated against these judgments; if raters cannot do this reliably the metric loses meaning.

invented entities (1)

error verifiability no independent evidence
purpose: A distinct dimension of LLM response quality measuring whether justifications enable users to distinguish correct from incorrect answers
Newly introduced concept whose independent evidence is limited to the paper's own human-rater agreement results.

pith-pipeline@v0.9.0 · 5516 in / 1551 out tokens · 43713 ms · 2026-05-10T19:54:28.223071+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We formalize this idea as error verifiability and propose v_bal, a balanced metric that measures whether justifications enable raters to accurately assess answer correctness, validated against human raters who show high agreement.
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Verifiability does not follow accuracy. Post-training and model scaling greatly improve accuracy but do not change or worsen verifiability.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

URLhttps://arxiv.org/abs/2502.12668. 19 Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh J...

work page doi:10.1145/3706598 2022
[2]

3 Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda

URLhttps://arxiv.org/abs/2602.23248. 3 Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of llm outputs, 2024. URL https://arxiv. org/abs/2407.13692. 3 Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan...

work page arXiv 2024
[3]

Proceedings of the 28th International Conference on Intelligent User Interfaces , pages =

doi: 10.1145/3581641.3584066. URL http://dx.doi.org/10.1145/3581641.3584066. 2 Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, an...

work page doi:10.1145/3581641.3584066 2025
[4]

No I have never in my life been drunk

URLhttps://arxiv.org/abs/2306.13063. 8 Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. On verbalized confidence scores for llms, 2024. URLhttps://arxiv.org/abs/2412.14737. 9 Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, and Daniel E. Ho. A reasoning-focused legal retrieval benchmark. In...

work page doi:10.1145/3709025.3712219 2024
[5]

arXiv preprint arXiv:2510.22922 (2025)

URLhttps://arxiv.org/abs/2510.22922. 3, 7 13 Appendix Table of Contents §A Full Post-Training Checkpoint Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 §B Full Cross-Model Comparison Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 §C Stylistic Rephra...

work page arXiv
[6]

Do the final answers agree or disagree?

work page
[7]

What are the key steps or assumptions in the main response? Are they different from the alternatives? Could any of them be wrong?

work page
[8]

If there are differences between responses, which reasoning seems more reliable?

work page
[9]

Analysis: Rephrase prompt

How confident should we be in the main response? Is it likely correct, uncertain, or likely wrong? If uncertain or likely wrong, identify the specific problematic step(s) and what the correct steps might be. Analysis: Rephrase prompt. You are a helpful assistant that rewrites responses to help readers judge whether the response’s answer is correct. Your g...

work page 2022
[10]

A response is FINISHED if it clearly states a final numerical or mathematical answer

work page
[11]

If FINISHED, extract the final answer exactly as it appears (just the value, no ‘‘The answer is’’ prefix)

A response is UNFINISHED if there is no clear final answer stated. If FINISHED, extract the final answer exactly as it appears (just the value, no ‘‘The answer is’’ prefix). If UNFINISHED, output ‘‘N/A’’. Output format (exactly one line): FINISHED:〈answer〉OR UNFINISHED: N/A Answer comparison prompt. You are comparing two mathematical answers to determine ...

work page
[12]

Consider mathematical equivalence, not string equality

work page
[13]

Compensation

Ignore formatting differences (spaces, commas in numbers, etc.). Reply with ONLY ‘‘Yes’’ or ‘‘No’’. J.4 LLM-as-a-Judge Configurations and Prompts All three rater models (GPT-4.1-mini, Claude-Haiku-4.5, Gemini-2.5-Flash-Lite) share the same prompts and decoding configuration: temperature 0.0, maximum 30 output tokens. Direct mode (single-turn).The rater re...

work page
[14]

terminated

Instructions.[You will verify whether proposed answers to math problems are correct. You will review 16 math questions with 3 minutes per question. Each question has the AI’s proposed answer. Your task is to determine whether each proposed answer is correct or incorrect.] 3.Active.Participants answer all 17 items. 4.Completed.Participants submit and exit ...

work page

[1] [1]

URLhttps://arxiv.org/abs/2502.12668. 19 Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh J...

work page doi:10.1145/3706598 2022

[2] [2]

3 Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda

URLhttps://arxiv.org/abs/2602.23248. 3 Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of llm outputs, 2024. URL https://arxiv. org/abs/2407.13692. 3 Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan...

work page arXiv 2024

[3] [3]

Proceedings of the 28th International Conference on Intelligent User Interfaces , pages =

doi: 10.1145/3581641.3584066. URL http://dx.doi.org/10.1145/3581641.3584066. 2 Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, an...

work page doi:10.1145/3581641.3584066 2025

[4] [4]

No I have never in my life been drunk

URLhttps://arxiv.org/abs/2306.13063. 8 Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. On verbalized confidence scores for llms, 2024. URLhttps://arxiv.org/abs/2412.14737. 9 Lucia Zheng, Neel Guha, Javokhir Arifov, Sarah Zhang, Michal Skreta, Christopher D. Manning, Peter Henderson, and Daniel E. Ho. A reasoning-focused legal retrieval benchmark. In...

work page doi:10.1145/3709025.3712219 2024

[5] [5]

arXiv preprint arXiv:2510.22922 (2025)

URLhttps://arxiv.org/abs/2510.22922. 3, 7 13 Appendix Table of Contents §A Full Post-Training Checkpoint Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 §B Full Cross-Model Comparison Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 §C Stylistic Rephra...

work page arXiv

[6] [6]

Do the final answers agree or disagree?

work page

[7] [7]

What are the key steps or assumptions in the main response? Are they different from the alternatives? Could any of them be wrong?

work page

[8] [8]

If there are differences between responses, which reasoning seems more reliable?

work page

[9] [9]

Analysis: Rephrase prompt

How confident should we be in the main response? Is it likely correct, uncertain, or likely wrong? If uncertain or likely wrong, identify the specific problematic step(s) and what the correct steps might be. Analysis: Rephrase prompt. You are a helpful assistant that rewrites responses to help readers judge whether the response’s answer is correct. Your g...

work page 2022

[10] [10]

A response is FINISHED if it clearly states a final numerical or mathematical answer

work page

[11] [11]

If FINISHED, extract the final answer exactly as it appears (just the value, no ‘‘The answer is’’ prefix)

A response is UNFINISHED if there is no clear final answer stated. If FINISHED, extract the final answer exactly as it appears (just the value, no ‘‘The answer is’’ prefix). If UNFINISHED, output ‘‘N/A’’. Output format (exactly one line): FINISHED:〈answer〉OR UNFINISHED: N/A Answer comparison prompt. You are comparing two mathematical answers to determine ...

work page

[12] [12]

Consider mathematical equivalence, not string equality

work page

[13] [13]

Compensation

Ignore formatting differences (spaces, commas in numbers, etc.). Reply with ONLY ‘‘Yes’’ or ‘‘No’’. J.4 LLM-as-a-Judge Configurations and Prompts All three rater models (GPT-4.1-mini, Claude-Haiku-4.5, Gemini-2.5-Flash-Lite) share the same prompts and decoding configuration: temperature 0.0, maximum 30 output tokens. Direct mode (single-turn).The rater re...

work page

[14] [14]

terminated

Instructions.[You will verify whether proposed answers to math problems are correct. You will review 16 math questions with 3 minutes per question. Each question has the AI’s proposed answer. Your task is to determine whether each proposed answer is correct or incorrect.] 3.Active.Participants answer all 17 items. 4.Completed.Participants submit and exit ...

work page