Pith. sign in

REVIEW 5 major objections 6 minor 27 references

A training method that labels each token good or bad beats next-token prediction for out-of-distribution factuality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:42 UTC pith:ZD56OO7J

load-bearing objection New post-training recipe with consistent small OOD gains, but the single automatic judge leaves the headline claim under-verified. the 5 major comments →

arxiv 2607.17524 v1 pith:ZD56OO7J submitted 2026-07-20 cs.CL cs.LG

Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

Zitong Huang , Gustavo Lucas Carvalho , Deqing Fu , Robin Jia This is my paper
classification cs.CL cs.LG
keywords token-level supervisionoff-policy learningfactualityout-of-distribution generalizationLoRAconditional steeringsummarizationmachine translation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Token-Level Off-Policy Labeling (TOPL), a post-training method that replaces next-token prediction with a token-level binary correctness classification: given a document and a response, the model predicts for each token whether the token is faithful to the document. Training this way, then merging the learned LoRA update back into the generator, is claimed to improve factual consistency under distribution shift more than standard supervised fine-tuning, preference optimization, or other token-level objectives, on 11 summarization datasets, and to transfer to machine translation. The paper also argues that the LoRA matrices learned by TOPL form an interpretable mechanism: one matrix separates factual from non-factual token representations, and the other acts as a steering direction. A sympathetic reader would care because it suggests a simple off-policy training recipe that gives robustness without online rollouts.

Core claim

The central claim is that training a language model to predict, per token, whether the token is factually consistent with its input is a stronger post-training objective for out-of-distribution faithfulness than training it to generate reference tokens or prefer whole sequences. TOPL attaches a scalar reward head to the hidden states at an intermediate layer, trains it with binary cross-entropy on tokens from perturbed summaries labeled good and bad, then discards the head and merges the LoRA updates into the generator. On 11 AggreFact summarization datasets, TOPL ranks first on average across three backbone models, and on machine translation it beats SFT, TLDR, and unlikelihood training on

What carries the argument

The load-bearing object is a LoRA adapter trained by a binary cross-entropy reward head placed after the final RMSNorm at a chosen intermediate layer, on token-level correctness labels derived from perturbed summaries. The paper shows that the learned LoRA-A matrix acts as a low-rank classifier or conditioning projection that separates factual and non-factual hidden states, while LoRA-B supplies steering directions; the TOPL update is equivalent to a rank-r conditional steering transformation h' = W h + sum_i v_i (c_i^T h). This lets the method avoid optimizing a generation objective directly and instead reshape internal representations, which the authors argue is why it generalizes.

Load-bearing premise

The load-bearing premise is that the automated factuality judge Bespoke-MiniCheck-7B, chosen in Section 4.1 as the evaluation metric, scores faithfulness accurately enough that a few-hundredths average difference reflects real quality; the paper does not validate this judge against human ratings in this setup.

What would settle it

Re-run the comparison scoring generated summaries with a second independent judge or human annotators; if TOPL no longer ranks first on average across the 11 AggreFact datasets, the central OOD claim fails. A second check: on a fresh backbone, measure LoRA-A AUROC; if high separability comes without OOD improvement, the proposed mechanism is called into question.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If TOPL is right, off-policy post-training can deliver out-of-distribution factuality gains without the complexity of online rollouts, making the recipe cheap to apply.
  • The same adapter that improves generation can be read out as a token-level factuality detector and as steering directions, so interpretability tools come for free.
  • Because the method transfers from summarization to machine translation, other faithful-generation tasks with token-level labels (long-form QA, controlled rewriting) are natural next targets.
  • The label-sensitivity experiments imply that contrastive supervision matters more than label balance; practical data pipelines should ensure both good and bad tokens appear together.
  • LoRA-A separability could be used as a training-time diagnostic: high AUROC on held-out tokens would predict better OOD behavior before evaluating on downstream benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: TOPL could be run as a general correctness pretraining step on any corpus with token-level error annotations, then merged, without needing preference pairs or reference outputs.
  • My inference: if LoRA-A separability is a genuine proxy for OOD robustness, one could select LoRA insertion layers or early-stop using AUROC, avoiding expensive benchmark evaluation.
  • My inference: the paper's OOD rankings depend on a single automated judge; confirming the top-line result with a second judge or human ratings would strengthen the claim, since the authors validate only the sentence-level vs full-summary protocol (Spearman 0.79).
  • My inference: the finding that final layers should be left untouched suggests adapter placement rules that may transfer to other token-level classification objectives beyond factuality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces TOPL (Token-Level Off-Policy Labeling), a post-training method that replaces next-token prediction with a token-level binary classification objective. Given a document and a response containing factual and perturbed tokens (derived from the FAVA dataset), TOPL attaches a reward head to an intermediate layer and trains LoRA adapters to predict per-token correctness under binary cross-entropy. At generation time the reward head is discarded and the merged LoRA update is used. The method is evaluated on document summarization with three backbones (Qwen3-8B, Llama-3.1-8B, Gemma-3-4B), compared against SFT, DPO, TDPO, TLDR, Unlikelihood, and a sequence-level analogue (SOPL). The central claim is strong out-of-distribution factuality on 11 AggreFact datasets, plus a transfer study to machine translation. The paper also analyzes the LoRA adapters: LoRA-A is claimed to form a token-level correctness classifier, and LoRA-B is claimed to act as a steering vector; a post-hoc steering experiment and an AUROC separability analysis are provided.

Significance. If the empirical claims hold, TOPL is a simple, off-policy alternative to preference optimization that improves OOD faithfulness without online rollouts. The multi-backbone and multi-baseline setup is a strength, and the label-sensitivity ablations probe the learning signal in a useful way. The LoRA-A/LoRA-B analysis is suggestive, though it is framed as a hypothesis rather than a causal proof. The main limitation is that every OOD conclusion rests on a single automated judge (Bespoke-MiniCheck-7B) whose criterion validity is not established for these outputs; the reported gains are small (a few points) and could reflect stylistic biases. The paper is worth serious consideration, but the load-bearing evidence needs reinforcement before publication.

major comments (5)
  1. [Section 4.1, Appendix A.1] The OOD claim relies exclusively on Bespoke-MiniCheck-7B. The Spearman=0.79 check in Appendix A.1 compares the same judge under two aggregation protocols; it is a consistency check, not a validity check against human judgments. The headline differences are small (OOD Qwen 0.8726 vs TLDR 0.8533; OOD Gemma 0.8398 vs Unlikelihood 0.8300; MT OOD 0.9300 vs TLDR 0.9226). A judge with a stylistic prior could flip these rankings. Please report human agreement on a sample, or corroborate with a second independent metric (e.g., AlignScore, TRUE, or expert annotations), at least on the main OOD comparisons.
  2. [Section 4.3 / Tables 10-11] Per-dataset OOD results are not reported anywhere; Tables 10-11 give only averages over 11 AggreFact datasets. The claim of 'strong out-of-distribution generalization across 11 datasets' is therefore not verifiable. Also, the sliding-window layer search in Section 4.3 is not stated as being performed on a validation split or on the OOD test set. If the search used the OOD test data, the chosen layer ranges (0-29 for Qwen, 0-27 for Llama/Gemma) may be overfit to AggreFact, which would inflate the main result. Please report per-dataset scores and clarify the selection protocol.
  3. [Section 4.1 / Tables 3-9] TOPL uses LoRA layer ranges (0-29 or 0-27) that differ from all baselines (0-35, 0-31, or 0-33). Section 4.3 shows that the layer range has a substantial effect on factuality, yet no layer-range tuning is reported for SFT, DPO, SOPL, TDPO, TLDR, or Unlikelihood. This is a confound: TOPL's advantage could come from avoiding disruptive final layers rather than from the token-level objective. Baselines should either be tuned over the same layer-range choices or use the same range as TOPL.
  4. [Section 4.4 / Figure 4] Random labels improve Qwen3-8B over the base model. This is reported as a curiosity and attributed to prior findings, but it undercuts the central mechanistic claim that token-level correctness information drives the gains. If a model improves with random labels, the binary classification objective may act as a regularizer or LoRA training dynamics may dominate. The paper should address this directly, e.g., by testing random labels across multiple seeds, reporting whether the effect is stable, and tempering the 'contrastive supervision is essential' conclusion.
  5. [Section 5.1 / Eq. (5)] Equation (5) restates the definition of LoRA (h' = Wh + (alpha/r) B A h) with c_i and v_i as rows of A and columns of B. The 'conditional steering' interpretation is therefore an algebraic identity, not empirical evidence. The paper does frame this as a hypothesis, which is appropriate, but the reader should not be left with the impression that the mechanism has been proven. The evidence in Sections 5.2-5.3 is correlational; please keep the wording as 'interpretation/hypothesis' and make the algebraic nature explicit.
minor comments (6)
  1. [Tables 10-11] DPO and SOPL rows report no standard deviation, while other methods do. Please provide metrics or state that these are single runs.
  2. [Section 4.5 / Table 13] The machine-translation transfer study uses only Qwen3-8B. One backbone is enough for a pilot, but the cross-task generality claim would be stronger with at least one additional model.
  3. [Section 4.2] The phrase 'strongest overall results' is based on the OOD average; without per-dataset breakdowns, the reader cannot see whether TOPL wins by large margins on a few datasets and loses on others. Please include a per-dataset table or a scatter plot.
  4. [Section 5.3] The steering intervention computes mean representations of good and bad tokens; it is not stated whether these means are computed on the training split, validation split, or test split. Please clarify to avoid any possible leakage.
  5. [Appendix A.1] The sentence-level vs full-summary correlation is computed on 300 samples, but the sample source (which model, which datasets) is unspecified. Please describe the sample.
  6. [Global] Typos and minor wording issues: 'the the LoRA- A components acts' in the introduction; 'i.e. a ”good” token' has stray quotation marks; Eq. (4) notation 'τ α' could be cleaner.

Circularity Check

0 steps flagged

No load-bearing circularity; central OOD claims rest on external benchmarks, not on fitted parameters or self-cited theorems.

full rationale

The main claim—that TOPL improves out-of-distribution factuality—is an empirical benchmark result. Training labels come from the external FAVA dataset; OOD evaluation uses an external judge (Bespoke-MiniCheck-7B) on 11 AggreFact datasets; the machine-translation result uses external XCOMET on FLORES+. No parameter is fitted to the evaluation data, and no reported 'prediction' is a re-labeled fit. The conditional-steering analysis in Sec. 5.1 is an algebraic identity: Eq. (5) is just the LoRA update rewritten as a sum of rank-1 terms, so saying that LoRA-A encodes concept projections and LoRA-B defines steering directions is a re-expression of the update, not a derivation whose output is secretly its input. The paper frames it as a hypothesis and then gives external AUROC and steering-intervention evidence. LoRA-A separability is an empirical property of trained adapters, not a direct copy of the loss, though it is unsurprising given the classification objective. The only same-author citation used as more than a baseline is TLDR (Fu et al., 2025), which appears as related work and as a compared method, not as justification for TOPL's effectiveness; no uniqueness theorem or external support is imported from self-citations. The paper explicitly acknowledges the correlation/causation limitation in Sec. 5.4. The score reflects a minor non-load-bearing self-citation and the near-tautological interpretability framing, not circularity in the core empirical derivation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central result depends on standard training hyperparameters (free), the FAVA corruption proxy, the MiniCheck judge, and the assumption that merging a classification-trained LoRA improves decoding.

free parameters (3)
  • LoRA rank and scaling = r=4, α=8
    Chosen hyperparameters for all methods; standard but not derived.
  • LoRA insertion layer range = layers 0-29 (Qwen), 0-27 (Llama/Gemma)
    Selected via sliding-window search over factuality scores (Sec 4.3); the main results use ranges informed by the search.
  • Per-method learning rate and batch size = varying (e.g., 5e-5 to 1e-4)
    Tuned separately per baseline; could affect fairness of comparisons.
axioms (4)
  • domain assumption FAVA token-level corruptions are a valid proxy for factual errors.
    Training labels zk are derived from FAVA's simulated hallucinations; the method assumes these generalize to real hallucinations in AggreFact and translation.
  • domain assumption Merging a classification-trained LoRA into the model preserves decoding while improving factuality.
    TOPL discards the reward head and generates with the original LM head; the paper assumes the merged LoRA update does not disrupt decoding (noted in Sec 4.3 as a risk for final layers).
  • domain assumption Bespoke-MiniCheck-7B scores reflect factuality.
    All OOD comparisons use this evaluator's average score; no human validation is reported in this preprint.
  • standard math LoRA update algebra follows the standard linear form h' = Wh + sum_i v_i(c_i^T h).
    The identity in Eq. 5 is the definition of LoRA; no extra assumption beyond linearity.

pith-pipeline@v1.3.0-alltime-deepseek · 14338 in / 11712 out tokens · 98622 ms · 2026-08-01T17:42:22.401629+00:00 · methodology

0 comments
read the original abstract

We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors.

Figures

Figures reproduced from arXiv: 2607.17524 by Deqing Fu, Gustavo Lucas Carvalho, Robin Jia, Zitong Huang.

Figure 1
Figure 1. Figure 1: Overview of TOPL training and its representation-space interpretation. Left: Training pipeline. Given a document, a query, and a response containing both factual and non-factual tokens, the inputs are processed by a large language model (LLM) with LoRA adapters to obtain token-level hidden representations. Instead of performing standard next-token prediction with a language modeling head, TOPL replaces the… view at source ↗
Figure 2
Figure 2. Figure 2: OOD factuality scores across different backbone models. TOPL consistently ranks [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Factuality scores under different LoRA insertion ranges. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Factuality scores under different label settings. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Machine translation results on Qwen3-8B measured by XCOMET (higher is better). TOPL achieves the best performance among all methods on the out￾of-distribution (OOD) FLORES+ benchmark. To evaluate whether TOPL extends to other faithful generation tasks, we additionally apply it to machine translation. We train Qwen3-8B on four language pairs from MLQE-PE (Fomicheva et al., 2022) (en-de, en-zh, ro-en, and et… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of LoRA-based representations across training methods and parame [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison between sentence-level and full-summary evaluation scores on 300 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Factuality scores under full-summary evaluation across different backbone models [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: LoRA-A and LoRA-B analysis for DPO and SOPL. Top row: SOPL; bottom row: DPO. Left column: LoRA-A separability heatmaps. Right column: LoRA-B steering curves under different scaling factors λ. While both methods exhibit relatively weak LoRA-A separability compared to TOPL, they still demonstrate clear LoRA-B steering behavior. Compared to TOPL, both DPO and SOPL exhibit substantially weaker LoRA-A separabil… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 16 linked inside Pith

  1. [5]

    Survey of hallucination in natural language generation.ACM computing surveys, 55(12):1–38,

    11 Published as a conference paper at COLM 2026 Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM computing surveys, 55(12):1–38,

    2026

  2. [7]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Pith/arXiv arXiv

  3. [8]

    Gradient imbalance in direct preference optimization.arXiv preprint arXiv:2502.20847,

    Qinwei Ma, Jingzhe Shi, Can Jin, Jenq-Neng Hwang, Serge Belongie, and Lei Li. Gradient imbalance in direct preference optimization.arXiv preprint arXiv:2502.20847,

    Pith/arXiv arXiv

  4. [9]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.arXiv preprint arXiv:2310.06824,

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.arXiv preprint arXiv:2310.06824,

    Pith/arXiv arXiv

  5. [10]

    Fine-grained hallucination detection and editing for language models.arXiv preprint arXiv:2401.06855,

    Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models.arXiv preprint arXiv:2401.06855,

    Pith/arXiv arXiv

  6. [11]

    doi: 10.1038/s41586-024-07335-x

    ISSN 1476-4687. doi: 10.1038/s41586-024-07335-x. URLhttps://doi.org/10.1038/s41586-024-07335-x. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741,

    doi:10.1038/s41586-024-07335-x

  7. [12]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  8. [14]

    Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  9. [15]

    Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors

    12 Published as a conference paper at COLM 2026 Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Woj- ciech Kry´sci´nski, Justin Rousseau, and Greg Durrett. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. InProceedings of the 61st Annual Meeting of the Association for Computatio...

    2026

  10. [16]

    Minicheck: Efficient fact-checking of llms on grounding documents

    Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of llms on grounding documents. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8818–8847,

    2024

  11. [17]

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid

    URLhttps://goo.gle/Gemma3Report. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248,

    Pith/arXiv arXiv

  12. [18]

    Llava-critic-r1: Your critic model is secretly a strong policy model.arXiv preprint arXiv:2509.00676,

    Xiyao Wang, Chunyuan Li, Jianwei Yang, Kai Zhang, Bo Liu, Tianyi Xiong, and Furong Huang. Llava-critic-r1: Your critic model is secretly a strong policy model.arXiv preprint arXiv:2509.00676,

    Pith/arXiv arXiv

  13. [19]

    Neural text generation with unlikelihood training.arXiv preprint arXiv:1908.04319,

    Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training.arXiv preprint arXiv:1908.04319,

    Pith/arXiv arXiv 1908

  14. [21]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  15. [22]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  16. [23]

    Token-level direct preference optimization.arXiv preprint arXiv:2404.11999,

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization.arXiv preprint arXiv:2404.11999,

    Pith/arXiv arXiv

  17. [24]

    Fine-tuning language models from human prefer- ences.arXiv preprint arXiv:1909.08593,

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences.arXiv preprint arXiv:1909.08593,

    Pith/arXiv arXiv 1909

  18. [25]

    Specifically, instead of decomposing the summary into individual sentences, we directly evaluate the full summary using the Bespoke-MiniCheck-7B model

    13 Published as a conference paper at COLM 2026 A Appendix A.1 Full-Summary Evaluation In addition to the sentence-level evaluation used in the main paper, we consider an alter- native evaluation protocol that scores the entire generated summary as a single sequence. Specifically, instead of decomposing the summary into individual sentences, we directly e...

    2026

  19. [26]

    15 Published as a conference paper at COLM 2026 TOPL Training Hyperparameters Qwen3-8B Llama-3.1-8B Gemma-3-4B LoRA Rank (r) 4 4 4 LoRAα8 8 8 LoRA Dropout 0.1 0.1 0.1 GPU 1×RTX A6000 Batch Size (per device) 16 16 16 Gradient Accumulation 1 1 1 Learning Rate 5×10 −5 1×10 −4 1×10 −4 LoRA Start Layer 0 0 0 LoRA End Layer 29 27 27 Table 3: Training hyperparam...

    2026

  20. [27]

    We report the total number of supervised tokens and the proportion of positive (factual) and negative (hallucinated) labels for each configuration

    18 Published as a conference paper at COLM 2026 Setting #Tokens % Good % Bad Original 2.91M 67.2 32.8 Random 2.91M 49.8 50.2 Balanced 1.40M 49.5 50.5 Bad-skewed 1.43M 32.9 67.1 Good-skewed 1.38M 65.8 34.2 Good-only 1.95M 100.0 0.0 Dense Balanced 1.82M 50.0 50.0 Table 12: Token-level supervision statistics for label sensitivity settings. We report the tota...

    2026

  21. [2017]

    Spurious rewards: Rethinking training signals in rlvr.arXiv preprint arXiv:2506.10947,

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr.arXiv preprint arXiv:2506.10947,

    Pith/arXiv arXiv

  22. [2019]

    Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719,

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719,

    Pith/arXiv arXiv

  23. [2022]

    Tldr: Token-level detective reward model for large vision language models

    Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. Tldr: Token-level detective reward model for large vision language models. InInternational Conference on Learning Representations, volume 2025, pp. 45205– 45236,

    2025

  24. [2023]

    Programming refusal with conditional ac- tivation steering

    Bruce W Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional ac- tivation steering. InInternational conference on learning representations, volume 2025, pp. 90960–90985,

    2025

  25. [2024]

    doi: 10.18653/v1/2024

    Association for Computational Linguistics. doi: 10.18653/v1/2024. wmt-1.4. URLhttps://aclanthology.org/2024.wmt-1.4/. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30,

    doi:10.18653/v1/2024 2024

  26. [2025]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  27. [2026]

    Findings of the WMT 2024 shared task of the open language data initiative

    Laurie Burchell, Jean Maillard, Antonios Anastasopoulos, Christian Federmann, Philipp Koehn, and Skyler Wang. Findings of the WMT 2024 shared task of the open language data initiative. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 110–117, Miami, Florida, USA, November

    2024