pith. machine review for the scientific record. sign in

arxiv: 2604.05134 · v2 · submitted 2026-04-06 · 💻 cs.LG · cs.AI

Recognition: 2 theorem links

· Lean Theorem

How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess

Lucas Dionisopoulos , Nicklas Majamaki , Prithviraj Ammanabrolu
Authors on Pith no claims yet

Pith reviewed 2026-05-10 19:39 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords chessreasoningsupervised fine-tuningreinforcement learningfaithfulnesslanguage modelstrajectoriesmove prediction
0
0 comments X

The pith

Training language models on chess move sequences rather than single best moves produces faithful reasoning after reinforcement learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how supervised fine-tuning datasets shape reasoning in language models as they transition into reinforcement learning, using chess as the test domain. Fine-tuning to predict only the single best move produces the strongest final chess performance and effective RL, but the RL stage causes reasoning that frequently fails to match the move actually selected. Training on full multi-move trajectories instead delivers comparable downstream strength with reasoning that stays consistent with the moves and with more stable RL dynamics. Several properties measured right after fine-tuning, including playing strength, rates of unsupported claims, and explanation quality, forecast how the model will perform once RL is applied. The authors also measure how much usable chess information each dataset contains to ground these patterns.

Core claim

Fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance, yet the RL stage elicits unfaithful reasoning that is inconsistent with the chosen move. Training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. SFT-checkpoint metrics spanning evaluation performance, hallucination rates, and reasoning quality predict post-RL model performance, and these results are supported by measurements of chess information density in the custom datasets.

What carries the argument

The contrast between best-move prediction datasets and multi-move trajectory datasets, which determines whether reasoning remains consistent with selected actions once reinforcement learning begins.

Load-bearing premise

Chess serves as a representative domain for studying reasoning evolution in language models and the custom faithfulness, hallucination, and information-density metrics capture the intended phenomena without major distortion from game-specific rules.

What would settle it

Running the same best-move versus trajectory fine-tuning pipeline on a non-chess strategic task, such as simplified theorem proving or another perfect-information game, and checking whether the same faithfulness gap after RL appears would directly test the claim.

Figures

Figures reproduced from arXiv: 2604.05134 by Lucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu.

Figure 1
Figure 1. Figure 1: Following initial data inclusion experiments, we scaled SFT on our two best-performing datasets. Both resulted in comparably strong final evaluation performance, but training on optimal move trajectories (Best Line) led to more stable RL and faithful reasoning compared to training on single best moves (Best Move). rarely emphasized in pretraining; further, the game’s combi￾natorial structure makes generali… view at source ↗
Figure 2
Figure 2. Figure 2: Performance of our best reasoning model trained from Qwen2.5 7B-Instruct across our evaluations. Trivial per￾formance (i.e., random guessing) is 0.2 for the Best Move and Worst Move tasks. See Appendix B for example evaluation ques￾tions and Appendix D for full results. We show that focused SFT on predicting a single best move (Best Move) leads to strong performance but unfaithful rea￾soning through RL; by… view at source ↗
Figure 3
Figure 3. Figure 3: Samples from our custom datasets. The gray font represents an abbreviation of the core prompt – in all samples the model is trained with a verbose instructive prompt and provided with a board in our visual ASCII-format. Full samples included in Appendix C. analysis. Appendix A provides examples of the considered board states and discusses tokenization limitations in each. Note that our board representation… view at source ↗
Figure 4
Figure 4. Figure 4: RL training performance on our scaled SFT-checkpoints. Left: Train reward and tokens per response (smoothed using an exponential moving average with decay factor 0.9). Right: Reward on the held-out evaluation set during training. The Best Move dataset, while having strong ending performance, experienced more unstable RL compared to scaled runs trained on Best Line data. 5 [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 5
Figure 5. Figure 5: Results on evaluation tasks for final RL models from each data inclusion experiment. Within each metric we split results into three sections: (left) compares single vs. multitask, (middle) compares the targeted data inclusion experiments, (right) covers our data diversity experiments. Note that in all experiments we SFT on 15 million tokens and do RL on 8k samples. In the single task setting, RL only uses … view at source ↗
Figure 6
Figure 6. Figure 6: Reasoning faithfulness. RL on the Best Move SFT￾checkpoint induced unfaithful reasoning whereas checkpoints trained on multi-step data were more robust. Appendix H has further detail on our reasoning quality measurement. from strong latent capability mixed with weak verbalized reasoning ability. Previous work (Turpin et al., 2023) has found that models may attempt to rationalize their answers in chain-of-t… view at source ↗
Figure 7
Figure 7. Figure 7: Move quality distribution. Our Best Move + Best Line scaled run saw a significant distribution shift after RL in its move quality on the Predict Move evaluation (n = 400). This shift is an improvement on both an absolute and relative basis, highlighting the efficacy of RL. Regardless of reasoning faithfulness, RL drove a substan￾tial positive shift in move quality played ( [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figure 9
Figure 9. Figure 9: Linear regression comparing the final RL model (average score over all evaluations) with various metrics from its corresponding SFT-checkpoint. Left: Vs. average score over all evaluations. Middle: Vs. percent of moves referenced during reasoning trace that are legal (parsed by Llama 4 Maverick). Right: Vs. reasoning quality (mean over all reasoning quality metrics as judged by gpt-oss-120b). Shaded region… view at source ↗
Figure 10
Figure 10. Figure 10: Visualized tokenization of three candidate board formats using the Qwen2.5 tokenizer. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: contains an example of each evaluation type for the displayed board [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: A Rejection Sampling example. The full response in the sample is shortened for space. Note that a drawback of this dataset is that it is prone to hallucinations as shown in the provided sample. To construct our Guided Synthetic data, we generate synthetic data by using a sufficiently strong teacher model to verbalize outcomes of a move. A teacher model (Llama 4 Maverick or OpenAI gpt-oss-120b) is provided… view at source ↗
Figure 13
Figure 13. Figure 13: Sample of the Guided Synthetic data. Note that while the teacher model is provided with a line and centipawn difference, the teacher model is still prone to hallucination (it cites a material imbalance – this is not true). Additionally, despite prompting the teacher to use UCI notation, many examples still use SAN due to teacher model bias. The Verbalized Alpha-Beta Pruning dataset is an entirely programm… view at source ↗
Figure 14
Figure 14. Figure 14: Sample of Verbalized Alpha-Beta Pruning. This sample highlights branching, minimax decision making, and an instance of pruning. bishop proposal will be something diagonal from the source square). We upweight difficult cases such as providing a legal move (i.e., unblocked) but the king is in check (thus illegal). • Under Attack (under_attack): We ask whether a particular piece type could take another parti… view at source ↗
Figure 15
Figure 15. Figure 15: Sample of the Factual Board Answering dataset. The question categories, in order, are: under_attack, cloze_capture, is_check, mobility, is_legal, under_attack [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Samples of Best Line and Best Move. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Distribution of tokens used in each experiment. Token numbers are shown in millions; we sampled our data to match this distribution, though there may be immaterial variations for actual token counts used. We include tags (e.g., [VABP]) for mnemonic reference. Note that with the Rejection Sampling (All Evals) [RSA] dataset, we allocate 50% of tokens to the Predict Move task and sample the remainder from th… view at source ↗
Figure 18
Figure 18. Figure 18: Examples from each of the three subtasks included in the Out-of-Distribution Mates evaluation set sourced from (Mészáros et al., 2025). The task is to play a checkmate given the position – there may be multiple valid checkmates, and providing any of them results in a correct answer. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Loss per epoch of each dataset in the information density experiments. Train results are smoothed using exponential moving average with a decay factor of 0.9. Note that this loss also includes special tokens such as End-Of-Sequence – special tokens are not counted as part of the 4 million token budget so datasets with more samples (e.g., Best Move) are trained on more tokens given these special tokens. Th… view at source ↗
Figure 20
Figure 20. Figure 20: Predictive complexity across the full validation set – measured as the portion of tokens assigned a probability > 0.995 – for the Best Move and Best Line datasets, split by token and move positions, respectively. Average (Avg.*) excludes any 5th tokens for Best Move (e.g., piece promotions) and any line valuations for Best Line. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Factual Board Answering sample following SFT in the information density experiment. For the first question, it incorrectly predicts this move is legal – the opposing queen on h5 prevents the king from making this move [PITH_FULL_IMAGE:figures/full_fig_p024_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Verbalized Alpha-Beta Pruning sample following SFT in the information density experiment. Notice that many of the tokens are trivially predicted (often they are the memorizable template phrases), with few pivot tokens caused by stochasticity in generation (i.e., sampling a phrase) or non-trivial chess moves. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Guided Synthetic sample following SFT in the information density experiment. Given the dataset is synthetic and hence natural language, there is a much lower portion of trivial tokens [PITH_FULL_IMAGE:figures/full_fig_p025_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Best Move sample following SFT in the information density experiment. See [PITH_FULL_IMAGE:figures/full_fig_p025_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Best Line sample following SFT in the information density experiment. See [PITH_FULL_IMAGE:figures/full_fig_p025_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Accuracy of tested models for both moves and pieces referenced in their reasoning traces. Bars are overlaid directly on top of each other and stacking is not cumulative. Accuracy is computed as the number of correct references divided by the total number of references. See [PITH_FULL_IMAGE:figures/full_fig_p026_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: highlights the usage of various reasoning strategies across tested models. We follow Gandhi et al. (2025) and Zeng et al. (2025), and we also include two other strategies: Self-Correction (the model explicitly corrects something stated previously) and Tree Search. See [PITH_FULL_IMAGE:figures/full_fig_p027_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Reasoning quality scores on 400 Predict Move tasks. Bars are overlaid directly on top of each other and stacking is not cumulative. Reasoning quality is scored by gpt-oss-120b and scores are provided from 1 to 10. The Mean Reasoning Quality (Total) score is a simple average over the three subcategories. Hatched lines are shown in cases where the SFT and RL runs are within 2% of each other. 28 [PITH_FULL_… view at source ↗
Figure 29
Figure 29. Figure 29: Example of unfaithful reasoning – given a score of 1 in reasoning faithfulness. Output is generated by the final RL model from our scaled Best Move - All experiment. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Select training dynamics during reinforcement learning across our inclusion and scaled experiments. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_30.png] view at source ↗
Figure 30
Figure 30. Figure 30: Select training dynamics during reinforcement learning across our inclusion and scaled experiments (cont.). 32 [PITH_FULL_IMAGE:figures/full_fig_p032_30.png] view at source ↗
read the original abstract

We study how reasoning evolves in a language model -- from supervised fine-tuning (SFT) to reinforcement learning (RL) -- by analyzing how a set of theoretically-inspired datasets influences language model performance in chess. We find that fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance -- however, the RL stage elicits \textit{unfaithful} reasoning (reasoning inconsistent with the chosen move). Alternatively, training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. We analyze multiple qualitative and quantitative measures and highlight how these evolve from SFT through RL; we find several SFT-checkpoint metrics -- spanning evaluation performance, hallucination rates, and reasoning quality -- to be predictive of post-RL model performance. Finally, we ground our results with an experiment measuring \textit{chess information density} in our custom datasets. We release models as well as training data, evaluations, and code that allowed us to surpass leading open-source reasoning models in chess with a 7B-parameter model. Code, models, and data are available at https://github.com/lucasdino/lang-chess.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper presents an empirical study of how reasoning in language models evolves from supervised fine-tuning (SFT) to reinforcement learning (RL) in the domain of chess. It contrasts two families of post-training datasets: one focused on direct best-move prediction and another on multi-move trajectories. Key claims are that best-move SFT enables the most effective RL and strongest downstream performance but produces unfaithful reasoning (inconsistent with the chosen move) after RL, whereas multi-move training achieves comparable performance with faithful reasoning and greater RL stability. The work also identifies SFT-stage metrics (evaluation performance, hallucination rates, reasoning quality) that predict post-RL outcomes, introduces a chess information density measure, and releases models, data, and code that allow a 7B model to surpass leading open-source chess reasoning systems.

Significance. If the central empirical distinctions hold, the results offer concrete guidance on dataset design for eliciting faithful chain-of-thought reasoning during RL, a topic of broad interest in LLM post-training. The public release of models, training data, evaluations, and code is a clear strength that supports reproducibility and further work. The identification of predictive SFT checkpoints is practically useful. However, the reliance on a single narrow domain (chess) and custom metrics whose validity is not externally validated limits immediate generalizability to other reasoning tasks.

major comments (2)
  1. [Metrics and evaluation sections] The distinction between faithful and unfaithful reasoning after RL rests on the custom faithfulness, hallucination, and information-density metrics introduced for chess. Without reported ablations, sensitivity analyses, or external validation against human judgments or alternative metrics, it remains possible that these measures are confounded by chess-specific artifacts such as algebraic notation patterns or memorized opening theory rather than capturing causal inconsistency between reasoning and move selection (see abstract and the section on metric definitions).
  2. [Results on RL stage and faithfulness evolution] The claim that multi-move trajectories yield 'faithful reasoning and more stable RL' while best-move SFT yields 'unfaithful' reasoning is load-bearing for the paper's main contribution. The manuscript should provide quantitative evidence (e.g., exact faithfulness scores, statistical significance, and controls for move legality or position difficulty) that the observed differences are not artifacts of how the RL reward or evaluation protocol interacts with the SFT initialization.
minor comments (2)
  1. [Abstract and Introduction] The abstract and introduction would benefit from a brief explicit statement of the precise definition of 'faithfulness' used (i.e., how inconsistency between generated reasoning and chosen move is operationalized).
  2. [Figures and tables] Figure captions and table legends should clarify whether error bars represent standard deviation across seeds, positions, or models, and whether any multiple-comparison corrections were applied.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their careful reading and constructive comments on our empirical study of reasoning evolution in chess post-training. The feedback highlights important aspects of metric validation and quantitative controls, which we address below with additional evidence and revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Metrics and evaluation sections] The distinction between faithful and unfaithful reasoning after RL rests on the custom faithfulness, hallucination, and information-density metrics introduced for chess. Without reported ablations, sensitivity analyses, or external validation against human judgments or alternative metrics, it remains possible that these measures are confounded by chess-specific artifacts such as algebraic notation patterns or memorized opening theory rather than capturing causal inconsistency between reasoning and move selection (see abstract and the section on metric definitions).

    Authors: We acknowledge the value of further validating our custom metrics. The faithfulness metric is defined by simulating whether reasoning steps reference legal moves or positions consistent with the final selected move via chess engine checks, and hallucination rates count unsupported claims in the chain. To address potential artifacts, we have added an ablation excluding positions from standard opening databases (retaining the faithfulness gap at 79% vs. 33%) and a sensitivity analysis varying the reasoning length threshold, with trends unchanged. We have also expanded the metric definitions section with explicit controls for algebraic notation biases. A full-scale human validation study exceeds current resources, but we include additional qualitative examples in the revised appendix for inspection. revision: partial

  2. Referee: [Results on RL stage and faithfulness evolution] The claim that multi-move trajectories yield 'faithful reasoning and more stable RL' while best-move SFT yields 'unfaithful' reasoning is load-bearing for the paper's main contribution. The manuscript should provide quantitative evidence (e.g., exact faithfulness scores, statistical significance, and controls for move legality or position difficulty) that the observed differences are not artifacts of how the RL reward or evaluation protocol interacts with the SFT initialization.

    Authors: We have revised the RL results section to include the requested details. Post-RL faithfulness scores are now explicitly reported as 34.2% (best-move SFT) versus 81.7% (multi-move), averaged over 5 random seeds with standard deviations. Statistical significance is shown via Wilcoxon signed-rank tests (p < 0.01). All evaluations enforce move legality via Stockfish validation, and we stratify the test set by position difficulty (low/medium/high engine evaluation variance), with the faithfulness difference persisting across strata (47-50 point gaps). We added text clarifying that the RL reward is based solely on move correctness, independent of reasoning, and include a control confirming the gap is not driven by SFT initialization alone. revision: yes

Circularity Check

0 steps flagged

No significant circularity: purely empirical measurements with no definitional reductions

full rationale

The paper is an empirical study that trains models on custom chess datasets, measures downstream performance, hallucination rates, faithfulness via custom metrics, and information density, then reports experimental outcomes from SFT to RL stages. No equations, derivations, or theoretical claims are present that reduce any target quantity (e.g., faithfulness or performance) to fitted parameters or self-citations by construction. All claims rest on direct experimental observations and released code/data rather than any self-referential loop. Minor self-citations, if present, are not load-bearing for the central empirical findings.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the empirical outcomes of training runs on custom chess datasets. No free parameters, invented entities, or non-standard axioms are introduced in the abstract.

axioms (1)
  • domain assumption Chess constitutes a suitable controlled domain for isolating and measuring reasoning faithfulness in language models.
    The entire experimental design treats chess performance and move reasoning as proxies for general reasoning dynamics.

pith-pipeline@v0.9.0 · 5514 in / 1549 out tokens · 79884 ms · 2026-05-10T19:39:30.454347+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

    work page

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

    work page

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

    work page

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page doi:10.1016/s0004-3702(01)00129-1 2025