Pith. sign in

REVIEW 5 cited by

Reverse Training to Nurse the Reversal Curse

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.13799 v3 pith:LBW3YIER submitted 2024-03-20 cs.CL cs.AI

Olga Golovneva , Zeyuan Allen-Zhu , Jason Weston , Sainbayar Sukhbaatar This is my paper
classification cs.CLcs.AI
keywords trainingmodelsreversalcursereverseevenfeatureissue
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Identity-bridge regularization, rephrased into an out-of-context reasoning form, yields ~40% reversal accuracy in a 1B LLM and provably fixes reversal in an idealized one-layer transformer.

  2. Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs

    cs.CL 2025-10 unverdicted novelty 6.0 of 10

    Masked fine-tuning enables autoregressive LLMs to inject new factual knowledge without paraphrases and with reversal-curse resistance, matching diffusion LLM advantages on QA tasks.

  3. Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A mask-and-reconstruct fine-tuning objective lets autoregressive LLMs inject new knowledge without paraphrases and largely removes the reversal curse.

  4. Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Adding identity supervision on bridge tokens enables out-of-distribution two-hop reasoning in simple transformers, with a nuclear-norm theory explaining the benefit.

  5. PlanE: Meta Planning of Data, Tuning, and Inference for Extractive-based LLMs

    cs.AI 2026-05 conditional novelty 4.0 of 10

    A quadratic meta-planner trained on a few model-dataset runs selects the optimal data-tuning-inference configuration for extractive LLMs, matching grid search on three IE tasks.

Pith tools