Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

This paper claims that looped-transformer iteration states encode human preference primarily as a relational comparison between responses, and that after correcting two evaluation errors the effect is real but small — about 2.3 percentage p

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Relational readouts of looped-transformer states beat pointwise readouts by just 2.3 points after correction; the paper's headline accuracies were evaluation artifacts acknowledged in its own erratum.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Self-refuting as submitted: the v2 erratum retracts the headline numbers while the v1 body keeps asserting them. The salvageable core is the methodological analysis of pairwise-evaluator pitfalls, not the empirical claim. the 3 major comments →

arxiv 2604.09870 v2 pith:7H7ZXYMA submitted 2026-04-10 cs.LG cs.AI

Relational Preference Encoding in Looped Transformer Internal States

Jan Kirin This is my paper
classification cs.LG cs.AI
keywords looped transformerrelational preferenceRLHFinternal statesprobingpairwise evaluatorflip testevaluation artifacts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper originally reported that a lightweight pairwise evaluator reading the internal loop states of a frozen 2.6B-parameter looped transformer achieved 95.2% accuracy on the HH-RLHF test set, far above reward-model baselines. An appended erratum, however, shows that figure was inflated by a canonical-ordering artifact (the evaluator learned to prefer the first-presented argument) and by data leaks in the probe baselines. The surviving claim is that preference is predominantly accessible via relational comparisons between response representations rather than as an absolute property of individual responses, but the corrected effect is modest: pairwise decoding beats pointwise decoding by about 2.3 percentage points (95% CI +1.3 to +3.3), and no corrected readout rivals end-to-end reward models. The paper's lasting contributions are methodological: it demonstrates that swap-based training protocols deflate pairwise training metrics, that standard accuracy metrics can hide constant-output degenerate evaluators, and that a simple flip test exposes such failures.

Core claim

We show that in the frozen looped transformer Ouro-2.6B, preference is predominantly accessible through relational comparisons between the representations of chosen and rejected responses, rather than as an absolute property of a single response. A lightweight pairwise evaluator reading per-iteration states achieves an antisymmetrized accuracy of 63.9% on the full test set, and beats pointwise decoding by +2.3 points (95% CI [+1.3, +3.3]) after correcting the two evaluation errors that inflated the original numbers. The corrected pair-disjoint probes remain above chance (56.5% pairwise, 54.2% pointwise), but no corrected readout rivals end-to-end reward models. We also identify systematic pi

What carries the argument

The central object is the set of per-iteration hidden states of Ouro-2.6B-Thinking, captured by a forward hook as a list of four tensors. The argument is carried by the pairwise difference representation at each loop step — the difference between attention-pooled chosen and rejected states — passed through bias-free layer normalization (to preserve antisymmetry), a GRU over the four steps, and a scorer. The flip test — swapping input order and checking whether scores change sign — is the diagnostic mechanism that distinguishes genuine order sensitivity from a constant function, and antisymmetry correlation is identified as the robust metric because sign-flip rate is distorted by scorer bias.

Load-bearing premise

The central claim presupposes that the four tensors captured by the forward hook really are per-iteration loop states, and that the corrected measurements from the unpublished audit are accurate; if either fails, the relational-encoding result collapses.

What would settle it

Train the same pairwise and pointwise evaluators on the same frozen model with the corrected protocol — antisymmetrized evaluation and strictly pair-disjoint splits — on a fresh test set; if the pairwise advantage falls outside the reported 95% confidence interval of +1.3 to +3.3 points, the relational-encoding claim is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the corrected claim holds, lightweight evaluators can read a genuine but modest relational preference signal from frozen loop states, without modifying the base model's weights.
  • Swap-based training protocols for pairwise preference models systematically deflate reported training accuracy; fixed-ordering evaluation on held-out data is necessary for accurate performance estimates.
  • The flip test (correlation, sign-flip rate, mean sum) should be a mandatory validation step for any pairwise preference evaluator, because standard accuracy metrics can fail to detect constant-output degenerate solutions.
  • Antisymmetry correlation, not strict sign-flip rate, is the reliable measure of order sensitivity when a learned bias offset is present.
  • No corrected readout from loop states rivals end-to-end reward models on the HH-RLHF dataset, so loop-state monitoring is not a substitute for reward modeling at this stage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the corrected small effect size suggests that the 'relational versus absolute' framing may be overstated; the +2.3-point edge could partly reflect the trivial advantage of seeing two samples to compare, rather than a special property of looped transformer states.
  • Testable extension: run the same pairwise-versus-pointwise protocol on a non-looped transformer of similar scale; if the relational edge persists, it is not specific to looped iteration states.
  • The flip test and swap-deflation lessons generalize well beyond looped transformers and apply to any pairwise preference learning setup, including reward-model training.
  • Because the corrected numbers depend on an audit described as pending in an unpublished follow-up, the surviving effect sizes should be treated as provisional until that audit is publicly available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript (arXiv:2604.09870v2) investigates whether the internal loop-iteration states of the frozen looped transformer Ouro-2.6B encode human preference. The original body reports 95.2% pairwise test accuracy on HH-RLHF, an 84.5% pairwise linear probe, and a 21.75% pointwise probe below chance, concluding that preference is encoded predominantly relationally. A prepended v2 erratum states that a post-publication audit found the headline numbers inflated by two independent evaluation errors: the 95.2% is a canonical-ordering artifact (63.9% strict antisymmetrized accuracy), and the pairwise/pointwise probes contained source-item leaks (corrected 56.5%/54.2%). The erratum withdraws the 'inverted polarity' finding and states that no corrected readout rivals end-to-end reward models. The surviving claims are a +2.3-point relational-over-pointwise edge (95% CI [+1.3, +3.3]) and that the antisymmetrized evaluator still beats the linear probe. The methodological findings (constant-output degeneracy, flip test, swap-protocol metric deflation) are said to stand, with antisymmetrized accuracy replacing antisymmetry correlation as the certification metric.

Significance. If the corrected +2.3-point effect and the antisymmetrized-evaluator-over-probe result are reliable, the paper makes a modest empirical contribution about relational preference decoding in looped-transformer states, and its methodological findings — swap-protocol training-metric deflation and the flip test for pairwise evaluators — are useful and deserve credit. However, the manuscript as submitted is not in a publishable state: the body's headline claims are explicitly retracted by the paper's own erratum, the corrected figures are deferred to an unpublished in-preparation follow-up, and the identification of the 'loop iteration states' is asserted rather than demonstrated. The scientific content that remains after correction is both much weaker and unverifiable within this document.

major comments (3)
  1. [§3.1] The paper's object of study is the four tensors in forward-hook output[1]. The only support for the claim that these are 'genuine loop iteration states, not layer outputs or attention weights' is 'structural analysis' — shared dimensionality and values 'consistent with iterative refinement.' No code, no comparison with Ouro's internal loop variables, no ablation, and, as §2.1 concedes, no second publicly available looped transformer for cross-architecture corroboration. Every result, including the surviving +2.3-point relational effect, presupposes this identification. This is load-bearing. The author must provide direct evidence (e.g., comparing hook output with Ouro's loop-storage variables, demonstrating early-exit dynamics per tensor, or using controlled inputs with known iteration counts). As it stands, the loop-state ontology is unsupported.
  2. [Abstract / v2 erratum] The body's central claims are contradicted by the paper's own erratum. The abstract, §1 contributions, §4.1 Table 1, §4.2 Table 2, §5.2, and §6 all report the 95.2% accuracy and the 84.5%/21.75% probe values as valid. The erratum states these are artifacts. The corrected values are not derivable from the manuscript; they are summarized in the erratum and deferred to 'Kirin, 2026, in preparation,' so no reader can verify the pair-disjoint split, the CI computation, or the antisymmetrized-evaluator comparison. A manuscript whose headline results are retracted within its own covers is internally inconsistent. It must be rewritten so that corrected claims and their supporting evidence appear in the body.
  3. [§4.2 / erratum] Even taking the erratum's corrected figures at face value, the interpretive claim that 'preference is predominantly accessible via relational comparisons' is not supported at the corrected magnitude. The corrected pairwise probe is 56.5% and the pointwise probe 54.2% — both barely above chance and separated by 2.3 points. The original evidence for 'predominantly relational' included the below-chance 21.75% pointwise result and a 19–30 point gap; those are withdrawn. A 2.3-point edge, while statistically significant by the reported CI, does not by itself establish 'predominantly' unless accompanied by additional analysis (e.g., fraction of variance explained, stability across splits/seeds, or a pre-specified relational-vs-pointwise comparison). The manuscript's conclusion as written is therefore not supported by the surviving numbers.
minor comments (5)
  1. [§3.1 / §3.2] The hook description is under-specified: 'output[1]' is not defined independently of Ouro's implementation, and the claim that output[0] is a standard BaseModelOutputWithPast is not enough to rule out an unpacking artifact. Please include the patched model code or an explicit trace of the forward-pass tuple.
  2. [§4.1] The error analysis is based on 'manual inspection of 30 examples' with no sampling protocol, no inter-annotator agreement, and no quantitative breakdown beyond approximate counts. This is anecdotal evidence; it should be labeled as such.
  3. [§4.4] The 'LR dead zone' explanation is internally inconsistent: it says epoch 2 evaluates 'approximately the same weights as epoch 1,' yet test accuracy rises from 83.3% to 95.2%. If the weights are nearly unchanged, the accuracy increase requires explanation. Also, 'too high for fine-tuning but too low to escape the current loss basin' is not a dead zone by the standard definition.
  4. [§5.5] The 'amygdala'/'basal ganglia' steering modules and the Vesper 'Dusk/Twilight/Dawn' connectome-derived architecture are speculative, are not part of any experiment in this paper, and should be removed or moved to an appendix clearly separated from the scientific contribution. Their presence distracts from the already weakened central claim.
  5. [Global] Several passages use 'ceiling' inconsistently: the abstract calls the L-BFGS probe a 'ceiling' while §4.7 correctly recharacterizes it as a lower bound on linearly extractable signal. Please harmonize the terminology throughout, and ensure the erratum's corrected values are reflected in all tables and figures.

Circularity Check

1 steps flagged

No derivation-level circularity: the surviving relational effect is a held-out comparison, but its corrected numbers are load-bearing on a self-cited unpublished audit.

specific steps
  1. self citation load bearing [Erratum (v2 prepended to Abstract)]
    "The central finding survives at much smaller magnitude: preference is decoded more accurately relationally than pointwise (paired +2.3 points, 95% CI [+1.3, +3.3]), and the antisymmetrized evaluator still beats the linear probe... Full audit in the follow-up work (Kirin, 2026, in preparation)."

    The surviving headline is not derived or reproducible in this document; the erratum retracts the original statistics and substitutes numbers attributed entirely to an unpublished follow-up by the same author. The +2.3-point relational edge, the 63.9% antisymmetrized accuracy, and the pair-disjoint probe values therefore rest solely on the self-citation 'Kirin, 2026, in preparation'. Since the original evidence is withdrawn, the paper's remaining positive claim reduces to an uncheckable self-citation rather than to its own derivations, code, or released data.

full rationale

The original 95.2% vs 84.5% vs 21.75% results were inflated by evaluation errors, not by circular construction; the corrected comparison is a held-out measurement and is not equal to a fitted parameter by construction. The main derivation-level risk is elsewhere: Section 3.1 identifies output[1] as 'genuine loop iteration states' solely by shared dimensionality and values 'consistent with iterative refinement', with no independent definition or cross-check. That is a validity/object-of-study risk, not a circular reduction, because no equation or self-citation forces the hook output to equal loop states. The concrete circularity present is narrower: the erratum's corrected numbers, which now carry the central claim, are deferred to an unpublished self-cited audit ('Kirin, 2026, in preparation'), so the surviving empirical claim is load-bearing on a self-citation that cannot be verified inside the paper. That keeps the score below the 6+ range (no fitted parameter is renamed as a prediction, and no equivalence of equations is shown), but above a clean 0-2.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on (1) a single model whose hook output is asserted to be per-iteration loop states on weak structural evidence, (2) HH-RLHF labels with an acknowledged 25–30% noise floor, and (3) corrected numbers from an unpublished self-cited audit. No parameter is fitted to the target result in the style of a physical constant; the fitted/selected content is concentrated in operating points — epoch-2 checkpoint selection (post hoc), the 50k example subset (hardware ceiling), the 0.87 early-exit threshold — and in the architectural search reported in Appendix B. The most consequential selection is the epoch-2 checkpoint: the body reports it as a point estimate with no adjustment for choosing the best of 5 checkpoints, and the erratum's confidence interval does not state that it accounts for this selection.

free parameters (3)
  • checkpoint selection epoch = epoch 2
    The headline model was chosen after observing test accuracy across epochs (83.3→95.2→89.5→67.2→62.4); body reports the peak as a point estimate with no selection correction; the erratum's CI is not stated to adjust for checkpoint selection.
  • training subset size = 50,000 of 161k HH-RLHF pairs
    Paper frames it as a hardware ceiling (360GB features, 15h extraction); subset composition unspecified; §4.7 uses data quantity as part of the explanation of the probe/evaluator gap, so scale is load-bearing.
  • early-exit threshold = 0.87
    Hand-set operating point for Ouro's adaptive early exit; determines which states exist to be captured, so every downstream feature depends on it; effects are not ablated.
axioms (4)
  • domain assumption HH-RLHF chosen/rejected annotations are treated as usable ground truth for human preference despite an acknowledged 25–30% annotator disagreement floor.
    The body reports 95.2% agreement as meaningful while conceding half of the 4.8% 'errors' are safety-prompt disagreements, and the surviving +2.3-point claim inherits this label-noise floor; the erratum does not re-estimate the noise ceiling.
  • domain assumption The four tensors in forward-hook output[1] are genuine per-iteration loop states, identified only by shared dimensionality and value differences 'consistent with iterative refinement'.
    §3.1 'Feature Extraction': 'We verified through structural analysis that these are genuine loop iteration states, not layer outputs or attention weights: all 4 tensors share the same dimensionality, and their values differ across iterations in ways consistent with iterative refinement.' No causal or weight-tying check is described.
  • domain assumption Ouro-2.6B-Thinking exists as described: a publicly released looped transformer with open weights, hub files, and accessible per-iteration hidden states; no model-card artifact is cited.
    §2.1 states Ouro is 'the only publicly released looped transformer with documented architecture, open weights, and accessible per-iteration hidden states'; the citation given (Zhu et al. 2025) is the looped-LM line, not a release artifact for this specific model. All results are single-model.
  • domain assumption The corrected audit numbers cited in the erratum (pair-disjoint splits, antisymmetrized accuracy, CI) are correctly computed, even though the audit is deferred to an unpublished self-cited follow-up.
    The v2 erratum states the corrected figures without protocol details: 'Full audit in the follow-up work (Kirin, 2026, in preparation)'. Since two prior artifacts were each invisible to one check, the corrected claims stand or fall on this external, currently unavailable audit.
invented entities (1)
  • 'amygdala'/'basal ganglia' steering modules and the Vesper 'Dusk/Twilight/Dawn' connectome-derived recurrent architecture no independent evidence
    purpose: Planned future components: feeding evaluator preference scores back into generation (gating/steering/early-exit) and supporting epistemic reasoning on ARC-AGI-3; not used in any experiment here
    §5.5 labels these 'entirely speculative and empirically unvalidated'; they bear no weight on the reported results, so they are listed for completeness rather than as an explanatory crutch.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Relational Preference Encoding in Looped Transformer Internal States." pith.science (2026). https://pith.science/paper/7H7ZXYMA

@misc{pith2026260409870,
  author       = {Pith},
  title        = {Pith review of: Relational Preference Encoding in Looped Transformer Internal States},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7H7ZXYMA}},
  note         = {Machine review of arXiv:2604.09870}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We investigate how looped transformers encode human preference, training lightweight evaluator heads on frozen Ouro-2.6B loop-iteration states on Anthropic HH-RLHF. v2: an erratum is prepended; the original manuscript is unchanged. A post-publication audit found the three headline results inflated by two independent evaluation errors. The 95.2% pairwise evaluator accuracy is a canonical-ordering artifact: the data were correctly split, but the evaluator learned to prefer the first-presented argument; its strict antisymmetrized accuracy on the full 8,552-pair test set is 63.9%. The 84.5% pairwise probe and the below-chance 21.75% pointwise probe were source-item leaks (orientation rows and pair partners crossing the train/test split); corrected pair-disjoint values are 56.5% and 54.2% -- above chance, so the "inverted polarity" finding is withdrawn. The central finding survives at much smaller magnitude: preference is decoded more accurately relationally than pointwise (paired +2.3 points, 95% CI [+1.3, +3.3]), and the antisymmetrized evaluator still beats the linear probe, but no corrected readout rivals end-to-end reward models. The methodological findings stand (constant-output degeneracy, flip test, swap-protocol metric deflation), with one correction: antisymmetrized accuracy, not antisymmetry correlation, certifies relational discrimination. The two errors are mutually invisible -- a split audit cannot see an ordering prior, antisymmetrization cannot see a leak -- so both checks are required. Full audit in the follow-up work (Kirin, 2026, in preparation).

Figures

Figures reproduced from arXiv: 2604.09870 by Jan Kirin.

Figure 1
Figure 1. Figure 1: Preference prediction accuracy by access pattern and method. The gap between pairwise and [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cross-epoch flip test analysis. Left: Antisymmetry correlation (ρ) is stable at −0.92 to −0.97 across all five epochs. Centre: Strict sign flip rate ranges from 25% (epoch 2, peak accuracy) to 96% (epochs 4–5, overfit) — inversely tracking accuracy. Right: Scorer bias (mean sum of normal and flipped scores) peaks at +2.51 at epoch 2 and dissipates to −0.37 by epoch 5, confirming that sign flip rate measure… view at source ↗
Figure 3
Figure 3. Figure 3: Swap-protocol metric inversion across five epochs. The deflated training metric (blue) climbs [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary

    cs.LG 2026-07 conditional novelty 7.0

    Strictly pre-answer hidden states of a looped transformer add significant AUROC over surface shortcuts for predicting correctness, and the readout yields decision-level gains but no generative control.

Reference graph

Works this paper leans on

10 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Emotion concepts and their function in a large language model

    Anthropic. Emotion concepts and their function in a large language model. Transformer Circuits Thread, 2026. https://transformer-circuits.pub/2025/emotion-features/index.html

    2026

  2. [2]

    Z., and Koltun, V

    Bai, S., Kolter, J. Z., and Koltun, V. Deep Equilibrium Models. In NeurIPS, 2019

    2019

  3. [3]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Bai, Y., et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    Deep Reinforcement Learning from Human Preferences

    Christiano, P., et al. Deep Reinforcement Learning from Human Preferences. In NeurIPS, 2017

    2017

  5. [5]

    Universal Transformers

    Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, . Universal Transformers. In ICLR, 2019

    2019

  6. [6]

    Adaptive Computation Time for Recurrent Neural Networks

    Graves, A. Adaptive Computation Time for Recurrent Neural Networks. arXiv:1603.08983, 2016

    Pith/arXiv arXiv 2016

  7. [7]

    and Desarkar, M

    Maheswaran, A. and Desarkar, M. S. A Unified View on Emotion Representation in Large Language Models. In EACL, 2026

    2026

  8. [8]

    Training language models to follow instructions with human feedback

    Ouyang, L., et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022

    2022

  9. [9]

    Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

    Templeton, A., et al. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, 2024

    2024

  10. [10]

    Scaling Latent Reasoning via Looped Language Models

    Zhu, R., Wang, Z., Hua, K., Zhang, T., et al. Scaling Latent Reasoning via Looped Language Models. arXiv:2510.25741, 2025

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.