Pith. sign in

REVIEW 4 major objections 8 minor 18 references

Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary

T0 review · 4 major / 8 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ACT-Eval decomposes LLM chess commentary into atomic claims, routes them to deterministic chess tools and expert gold atoms, and shows that tool access cuts hallucination rates roughly in half while conceptual coverage stays low.

desk verdict A real and honestly reported evaluation framework for LLM chess commentary, but the headline no-tool error rates rest on a single judge whose accepted claims experts only half agree with. read the letter →

arxiv 2608.04240 v1 pith:2WCC3KF6 submitted 2026-08-04 cs.CL cs.AI

S. Ashwin Hebbar , Peiyao Sheng , Sewoong Oh , Pramod Viswanath This is my paper · ORCID
classification cs.CLcs.AI
keywords chesscommentaryhallucinationdetectiontool-augmentedevaluationatomicclaimdecompositionLLM-as-a-judgefactualverificationexpertgoldannotationsengines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that current large language models cannot reliably explain chess moves without external help: they produce fluent but false sub-claims at high rates, and ordinary evaluation judges cannot tell. The proposed remedy, ACT-Eval, slices commentary into individual factual atoms, verifies each with deterministic chess tools (board-state queries and Stockfish evals), and compares conceptual content to expert-verified gold atoms. Using this on 325 position–move pairs, the paper reports that tool access reduces the strongest model's sub-claim error rate from 22.0% to 9.2% and largely fixes move-quality stance, but even the best model covers only 60.9% of expert ideas. A reader should care because the framework turns 'is this explanation true' into a checkable computation rather than another LLM opinion.

What carries the argument

The load-bearing objects are atomic claims ('atoms'): single position-specific factual statements, each carrying the full move sequence from the root position so it can be checked without context. The judge model decomposes generated commentary into these atoms, splits each into verifiable sub-claims, and is forced to call a suite of more than fifteen chess tools (piece-on-square, attacks, legality, variation simulation, and Stockfish evaluation) before issuing a verdict; an atom counts as correct only if all sub-claims pass. Verified atoms are then matched against expert-verified gold atoms to compute atomic recall, and the commentary's stance toward the played move is compared to engine-derived labels for move-quality F1. The key move is routing computationally decidable claims to deterministic oracles so the judge's parametric chess knowledge is not the evidence.

What would settle it

Build a test set of 100 atomic claims with unanimous expert verdicts (50 clearly correct, 50 clearly false) and run ACT-Eval's judge on them; if its agreement with experts on the clearly-correct subset does not exceed 57%, the top of the range the paper itself reports for accepted claims, or if the model ordering flips under a second judge model, the central claim that factual judgments fall within inter-human agreement does not hold.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a large share of what sounds like confident chess expertise in LLM commentary is factually wrong, and that the error can be made measurable. The authors show that an LLM judge given only reference text and raw engine numbers—the prior standard—rewards hallucinated commentary with near-perfect relevance scores, while the same commentary fails atom-level verification. ACT-Eval replaces relevance with factual precision (the share of tool-verified atoms divided by the total), completeness with atomic recall against expert gold atoms, and adds move-quality F1 against engine-derived labels. With this stack, factual error rates fall from 22.0% to 9.2% for the strongest tool-augmented model and move-quality F1 rises from 12.5 to 97.6, while recall against expert ideas remains between 44% and 61%.

Load-bearing premise

The framework assumes the LLM judge can correctly break commentary into the right checkable claims and read the chess tool's answers without systematic bias; if judge errors are systematic rather than random, the reported error rates and model rankings would be distorted.

Editorial extensions

If this is right

  • Tool-augmented atomic verification can serve as a reliable automatic substitute for human fact-checking of chess commentary, within inter-human agreement bounds.
  • Adding chess tools to a commentary model cuts sub-claim errors roughly in half for frontier models, so deployment should assume no-tool commentary is unreliable.
  • Move-quality stance becomes nearly solved with engine access (F1 from 12.5 to 97.6), but selecting which expert ideas to mention does not improve much: conceptual coverage stays below 61% everywhere.
  • Vanilla LLM-as-a-judge scores should not be trusted for expert-domain explanations; on hallucinated commentary that ACT-Eval flags, the prior judge still gives high relevance scores 39.8% of the time.
  • The decompose-and-verify pattern is a candidate template for any domain where part of the truth is computationally decidable and the rest can be pinned to expert annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own calibration implies reported error rates are lower bounds: because the judge accepts many claims experts reject (agreement drops to 14–57% on accepted claims), true hallucination rates are probably higher than the table shows, and cross-model gaps could shrink if judge bias varies by model.
  • The sharp rise in move-quality F1 with tools measures stance calibration, not chess understanding; a testable extension is to separate 'gave the right verdict' from 'explained why the verdict is right'.
  • The same route—one judge decomposing claims, deterministic tools checking the decidable ones, gold atoms anchoring the rest—could transfer to domains like proof explanation, clinical reasoning, or code review, provided each has a reliable oracle and curated expert annotations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper presents ACT-Eval, a framework for evaluating LLM chess commentary by decomposing commentary into atomic claims, routing verifiable claims to chess tools (board-state queries, move simulation, Stockfish evaluation), and measuring factual precision, atomic recall against expert gold atoms, and move-quality F1. A new benchmark of 325 position–move pairs is released, comprising 75 textbook positions with Chernev annotations, 50 positions from a Lichess Candidates study, and 200 Maia2-selected critical positions, with 125 positions carrying expert-verified gold atoms. Six LLMs are evaluated with and without tool access. The headline results are that no-tool sub-claim error rates range from 10.8% (Gemini 3.1 Pro) to 55.5% (Qwen3-8B), tool access roughly halves error rates for frontier models and raises move-quality F1 substantially (e.g., GPT-5.4: 12.5 to 97.6), and atomic recall remains below 61% for all configurations. Human calibration shows the judge's flagged errors align with experts 83–94% of the time, but accepted claims align only 14–57% of the time; coverage recall correlates with two annotators at r≈0.73–0.75.

Significance. If the findings hold, ACT-Eval is a meaningful step toward oracle-grounded evaluation in expert domains, and chess is a well-chosen testbed because a substantial fraction of claims are computationally decidable. The paper's strengths include a fully specified tool suite and exact model identifiers (Table 5), position-clustered bootstrap confidence intervals for every main metric, a documented human calibration protocol, a cross-judge robustness analysis (Section D.1), a judge ablation separating tool value from parametric judgment (Section 6.2), and an unusually candid limitations section that reports missed errors and rejected valid claims rather than hiding them. These properties make the framework and benchmark immediately usable by the community. The remaining validation gaps concern the generality of the absolute error-rate claims, not the value of the framework as a relative evaluation tool.

major comments (4)
  1. [Section 5.1; Section D.1; Table 6] The headline absolute rates in Table 2 (e.g., GPT-5.4 no-tool sub-claim error 22.0%, Qwen3-8B 55.5%) and the claimed tool benefits ('tool access substantially reduces factual errors', Section 4.2) rest solely on GPT-5.4 as both generator and judge. Section 5.1 reports that when the judge accepts a claim, expert annotators agree only 14–57% of the time, and Section D.2 documents both missed errors and valid claims rejected by misread tool output. The cross-judge rerun (Table 6) was restricted to tool-augmented outputs, so the no-tool condition that drives the headline hallucination and tool-benefit claims was never scored by an independent judge. Table 6 further shows a self-leniency pattern (GPT-5.4 judged at 9.5 by itself vs. 12.8 by Gemini; Gemini at 3.2 by itself vs. 7.5 by GPT-5.4), making judge–generator alignment a live confound. If this bias extends to no-tool outputs, the reported tool benefit could be inflated. I request that the no-tool outputs (or a pre-registered, representative subsample) be scored by the independent Gemini judge, and that the paper report both judges' absolute rates for the no-tool condition.
  2. [Section D.1; Table 5] The claim that the cross-judge analysis 'reran the full pipeline' (Section D.1) is stronger than what was done. Table 5 shows that the independent Gemini judge was used only at Stage 2 (tool-augmented verification); Stage 1 atomic decomposition and semantic classification remained GPT-5.4 for both judges. Since the limitations section concedes that LLM-based decomposition 'can produce incorrect or under-contextualized atoms', and since a biased decomposition could systematically advantage the generator that shares the judge's identity, the robustness check does not cover this entry point. Please either run Gemini-based decomposition on a sample and measure the agreement of the resulting atoms and verdicts, or revise the robustness conclusion to state explicitly that only the verification stage was varied.
  3. [Table 2; Section C.1] All headline error-rate and quality-F1 numbers are aggregates over three datasets with very different properties, yet no per-dataset breakdown is reported. The Critical set (200 of 325 positions) is deliberately built around positions where Maia2 predicts a human inaccuracy or mistake (Section C.1), so it is not a neutral sample of chess commentary; pooling it with the textbook and Candidates50 positions makes the aggregate 'pervasive hallucination' claim depend on the dataset mix. Per-dataset error rates, precision, and quality-F1 should be reported (or at least supplied for the contrast between Critical and the natural-commentary datasets) so that readers can separate the framework's ability to flag errors on adversarial positions from its estimate of hallucination in ordinary commentary.
  4. [Appendix B.4; Table 3] The move-quality metric depends on the stance extraction rule in the decomposition prompt that maps 'not the best' to 'good'. Because the metric requires an explicit 'bad' stance to count a mistake or blunder as detected, a model that describes a blunder as 'not the best' is recorded as a true negative, which may systematically depress no-tool F1 and inflate the tool-vs-no-tool gap in Table 3. Please provide a sensitivity analysis in which hedged criticisms ('not the best', 'far from ideal', 'dubious') are treated as 'bad' or at least 'inconclusive', and report the resulting F1 values.
minor comments (8)
  1. [Section 3.1] There are two typos: 'proposeACT-Eval' and 'AtomiCandTool-augmented Evaluation' should read 'propose ACT-Eval' and 'Atomic and Tool-augmented Evaluation'.
  2. [Section 6.1] The text says errors are normalized by 'decisive sub-claims'; the term 'decisive' is not defined and appears to be a typo for 'verifiable' or 'decidable' sub-claims.
  3. [Tables 2 and 3] F1 values are reported without percent signs, while the text reads 'F1 ranges from 12.5 to 40.9'; please make the units consistent.
  4. [Abstract; Appendix C.3] The abstract states that 'smaller open-weight models exceed 40%' error without the caveat from Appendix C.3 that the Qwen models were run in No-Think mode because thinking mode exhausted the generation budget; the headline should carry this configurational qualification.
  5. [Section 5] The human calibration sample is small (50 claims, 25 positions, four annotators for Study 1 and two for Study 2), and the annotators are described as 1800–2200 Lichess blitz players rather than grandmaster-level; the abstract's 'within the observed range of inter-human agreement' claim should acknowledge the sample size and expertise level.
  6. [Table 6] The note says n=225 but does not explain how the subset of positions scored by both judges was selected from the 325 total; please clarify the overlap rule.
  7. [References] The reference to the Lichess accuracy metric would benefit from a persistent URL or version identifier rather than a bare web-page citation.
  8. [Figure 1] The caption reports precision and recall values without defining which metric each refers to; adding 'factual precision' and 'atomic recall' to the caption would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ACT-Eval's factual, coverage, and move-quality metrics are anchored to external chess oracles and expert gold annotations, and the one same-model judge confound is explicitly tested and conceded.

full rationale

ACT-Eval's derivation chain is not circular. The three headline metrics are defined against independent external anchors: factual precision and sub-claim error rate are computed by tool-augmented verification against Stockfish 14.1 and python-chess board-state queries; atomic recall is measured by matching generated atoms against expert-verified gold atoms extracted from Chernev and Lichess annotations; and move-quality F1 is scored against engine-derived Lichess win-probability labels. None of these quantities is obtained by fitting a parameter to the data it later 'predicts,' and the paper contains no equation that reduces a reported result to its own input by construction. The only self-reference is that GPT-5.4 is used both as a generator and as the ACT-Eval judge. That is a real validity concern, but the paper directly probes it in Section D.1 by rerunning the pipeline with Gemini 3.1 Pro as an independent judge, finding the same model ordering under both judges, and the Limitations section explicitly concedes that absolute rates are judge-calibrated rather than judge-invariant. The low accepted-claim agreement (14-57%) and the documented judge failures in Appendix D.2 are measurement-accuracy limitations, not evidence that a prediction is equivalent to an input. There are no load-bearing self-citations or imported uniqueness theorems; the reference list contains no prior work by the present authors. A skeptical reviewer can question whether no-tool error rates were independently judged, but that is a correctness/robustness gap, not circularity under the definitions used here.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

Two hand-chosen thresholds, both inherited from Lichess conventions, affect the move-quality metric and the Critical dataset composition; no parameters are fitted to make the main results come out. The framework relies on domain assumptions: Stockfish as ground truth, the Lichess win-probability mapping, single-source expert annotations as coverage targets, and LLM decomposition with human review as a valid atomizer. The judge is an LLM, so its interpretation of tools is a maintained assumption rather than an external guarantee, and the paper documents residual errors (Appendix D.2). No invented entities are introduced.

free parameters (2)
  • wp_loss classification thresholds for move quality (good, inaccuracy, mistake, blunder): 10, 20, 30 = 10, 20, 30 (win-probability loss, from Lichess)
    Chosen by hand from Lichess conventions (Appendix A.1). These thresholds determine which 60 of 325 positions are excluded from Move Quality F1 and define the positive class (mistake/blunder) for the central move-quality claims in Table 2.
  • Maia2 sampling threshold for Critical dataset = Elo 2000, inaccuracy probability > 10%
    Chosen by hand (Appendix C.1) to construct the 200 critical position-move pairs. This skews the dataset toward human mistakes and thus affects the reported no-tool move-detection F1 values.
assumptions (4)
  • domain assumption Stockfish evaluations at depths 18-22 define ground truth for move quality and tactical facts
    Used for move-quality labels (Appendix A.1), the tool suite (Appendix B.2), and dataset filtering, e.g., rejecting annotations that contradict engine evaluation (Section 3.2). Engines are strong but not infallible.
  • domain assumption The Lichess win-probability mapping from centipawns is an adequate basis for classifying moves
    Adopted from Lichess in Appendix A.1; the 10/20/30 thresholds and the mapping itself determine every move-quality label in the benchmark.
  • domain assumption Chernev's book and the Lichess Candidates study annotations represent the key ideas a commentary should cover
    These annotations are the gold atoms (Section 3.2, Appendix C.1). Recall is defined only against this single source per position, so alternative valid analyses are not credited (Appendix A.3).
  • ad hoc to paper GPT-5.4-based decomposition, with human expert review, produces valid atomic claims and gold atoms
    The entire pipeline depends on LLM decomposition (Section 3.3, Appendix B.4); the paper's own calibration and Appendix D.2 show residual decomposition and judge errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary." pith.science (2026). https://pith.science/paper/2WCC3KF6

@misc{pith2026260804240,
  author       = {Pith},
  title        = {Pith review of: Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WCC3KF6}},
  note         = {Machine review of arXiv:2608.04240}
}
read the original abstract

Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike. Large language models could, in principle, bridge this gap, but they frequently hallucinate due to limited domain-specific knowledge, and standard reference-based or LLM-as-a-judge frameworks cannot reliably detect these errors. In this work, we present ACT-Eval, an evaluation framework that decomposes chess commentary into atomic claims and routes them to engine-supported tools and expert-annotated gold references to assess factual correctness, conceptual coverage, and move-quality judgment. We release a benchmark of 325 position--move pairs spanning pedagogical, tournament, and critical positions, including 125 positions with expert-verified gold atoms and a five-class error taxonomy. Evaluating leading proprietary and open-weight models, we find that factual hallucinations remain pervasive in chess commentary: GPT-5.4 without tools produces incorrect sub-claims 22.0% of the time, while smaller open-weight models exceed 40%. Although tool augmentation substantially improves factual correctness and move-quality assessment, coverage of expert strategic and tactical ideas remains limited across all models. Human calibration shows that ACT-Eval's factual judgments fall within the observed range of inter-human agreement, while its coverage scores correlate strongly with human assessments of strategic completeness.

Figures

Figures reproduced from arXiv: 2608.04240 by the authors.

Figure 1
Figure 1. Comparison between GCC-Eval and ACT-Eval on generated chess commentary. The commentary in this [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the ACT-Eval pipeline. Reference commentary is filtered and decomposed into expert-verified gold atoms, while generated commentary is decomposed into candidate atoms. The judge verifies tool-checkable candidate claims using chess-specific board-state and engine tools, matches verified candidates to the gold atoms, and computes factual precision, atomic recall, and move-quality F1. These metrics operation… view at source ↗
Figure 3
Figure 3. Factual error breakdown by category across [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Screenshot of ACT-Eval Human Calibration Study [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Screenshot of Study 1 in Human Calibration [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Screenshot of Study 2 in Human Calibration [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

  1. [1]

    ONE claim per atom - split compound claims into separate atoms

    work page

  2. [2]

    which is a typical tactical theme in such positions

    After Qh4+ and 48.Kg1, Black can win the queen byRxg2+. High coverage (recall = 1.0). “Qh4+ is a forcing move that sets up a brilliant discovered attack to win the white queen. After the forcedKg1, Black will playRxg2+, sacrificing the rook with check. This clears the g5 square, opening the h4–e7 diagonal for the black queen to capture White’s undefended ...

    work page

  3. [3]

    Group logically connected setup+consequence ONLY if they 're inseparable

    work page

  4. [4]

    X would \ support Y

    Each atom must be self-contained – carry the FULL move sequence from the root \ AND any premises/antecedents from surrounding sentences. A conditional ("X would \ support Y" when X is illegal) must include its premise or be dropped; a consequence \ of a prior move sequence must restate the sequence

    work page

  5. [5]

    Be specific – include piece names, squares, and concrete assertions

    work page

  6. [6]

    good": author presents the move as decent, solid, strong, best, or otherwise \ acceptable. A

    Ignore generic chess philosophy not applied to this position, and all non-positional \ fluff (history, attribution, biography, opening popularity – see CRITICAL note above) Also determine the author's OVERALL stance toward the played move itself \ (ignore alternatives, threats, and side comments): - "good": author presents the move as decent, solid, stron...

    work page

  7. [7]

    O-O is a strong move

    ACTUAL MOVE QUALITY: - Quality claims ("O-O is a strong move"): use eval_move (PRE_fen, move) NOT post_fen

    work page

  8. [8]

    an attacking move

    Ignore stylistic flourishes, stance/tone claims ("an attacking move", "a \ defensive idea", "weakens/strengthens the position", " improves/worsens the \ position", "creates pressure"), and author/player mental- state claims ("White \ feared X", "Black intended Y") – these are not position facts. Even negated \ versions ("does not weaken the position") are...

    work page

Show all 18 references
  1. [9]

    after 5.d4 .. exd4 6.Rxe8+ wins Black's rook

    CROSS-ATOM CONTEXT: Earlier atoms in the same sentence often establish the tactical basis for later atoms. APPLY prior tool findings; do not re-verify in isolation and do not forget them when a later atom's idiom depends on another atom's tactics. Example: atom 3 "after 5.d4 ....

    2003

  2. [10]

    Verify ALL atoms before outputting

    MANDATORY TOOL USE: You MUST call at least one tool. Verify ALL atoms before outputting

  3. [11]

    Bg5 pins the knight

    FEN SELECTION: - Move CREATES/DOES something ("Bg5 pins the knight"): POST-move FEN - Position BEFORE move ("knight was on g8"): PRE-move FEN - Alternative by same side: PRE-move FEN - Opponent's response to the move: POST-move FEN - When unclear: try both FENs

  4. [12]

    Black plans ...g5 then ...g4

    SAME-SIDE SEQUENCES (e.g., "Black plans ...g5 then ...g4 "): - PREFER explicit form: try_variation(fen, [".", "g5", ".", "g4"]). "." is an opponent-move placeholder; the tool picks a random legal reply and samples 3 branches. Use this when the claim's move is also legal for th...

  5. [13]

    PIECE COLORS: - {moving_color} pieces: {moving_color}, Opponent: { opponent_color} - PROTECTION: Only SAME-color pieces protect each other - ATTACKS: OPPOSITE-color pieces attack - Always verify color before claiming protect/attack

  6. [14]

    CAPTURES (only if this move was a capture): {capture_explanation}

  7. [15]

    After O-O, White can play Re1

    CONTINUATION MOVES: - For "After O-O, White can play Re1": use get_legal_moves(post_fen, "white") - ALWAYS specify color parameter when checking continuation moves

  8. [17]

    SUPPORTS

    "SUPPORTS"/"ENABLES"/"ALLOWS" CLAIMS: - "Supports" means "makes possible" or "doesn't interfere with", NOT "makes it happen immediately"

  9. [2022]

    arXiv preprint arXiv:2212.08195

    Improving chess commentaries by combining language models with symbolic reasoning engines. arXiv preprint arXiv:2212.08195. Leela Chess Zero. 2026. Leela chess zero: Open source neural network based chess engine. https: //lczero.org/. Accessed: 2026-05-25. Lichess. 2026. Liche...

    2026 arXiv

  10. [2024]

    not the best

    Maia-2: A unified model for human-ai align- ment in chess.Advances in Neural Information Pro- cessing Systems, 37:20919–20944. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elic- i...

    2022

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.