Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Improving Next Tokens via Second-to-Last Predictions with Generate and Refine

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a decoder-only model trained to predict the second-to-last token can serve as a refiner that improves the next-token predictions of standard GPT models.

desk verdict A neat decoding trick with a too-strong abstract; the simplewiki result is solid, but the 'consistent significant gains' claim rests on single training runs. read the letter →

arxiv 2411.15661 v2 pith:F2CZHFXK submitted 2024-11-23 cs.CL cs.LG

Johannes Schneider This is my paper · ORCID
classification cs.CLcs.LG
keywords largelanguagemodelsgenerate-then-refinesecond-to-lasttokenpredictionbidirectionaldecoder-onlymodelself-correctiondeterministicpermutationnext-token
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that an autoregressive language model's next-token predictions can be improved by a second, independently trained model that predicts the second-to-last token given the candidate next token. The authors train this refiner as a decoder-only transformer using a deterministic permutation of short token subsequences, which avoids the mask token and random masking of BERT-style training. At inference, the refiner re-ranks the top-k next-token candidates of GPT-2 by boosting candidates whose predicted second-to-last token matches the actual one. They report that second-to-last-token accuracy exceeds next-token accuracy by more than 15%, and that the generate-then-refine combination yields small (0.02 to 0.35 points) but consistent and statistically significant gains across GPT-2 variants and datasets. If true, this gives a low-cost path to bidirectional self-correction for unidirectional models.

What carries the argument

The mechanism is Algorithm AGR (Auto-encoding Generate-then-Refine), which couples a next-token model f_n with a second-to-last-token model f_s. For each of the top k candidates, f_s receives the candidate token and the preceding context, and if its argmax is the correct second-to-last token, the candidate's score is multiplied by (1+w). The refiner is trained by a deterministic permutation: each input sequence is split into subsequences of length l=4, and the fourth token is moved to the end (so (y0,y1,y2,y3) becomes (y0,y1,y2,y4)) with only the predictions at the moved positions contributing to the loss. This gives a decoder-only transformer bidirectional information without a mask token or random masking, and it keeps training close to the speed of ordinary causal next-token training.

What would settle it

On a constructed corpus where every target token is followed by a fixed or random token unrelated to the target, train f_s and f_n with the paper's setup and compare their accuracies; if f_s is not more accurate than f_n, or if AGR fails to improve next-token accuracy, the locality assumption is violated for that data.

Watch

Extended reading notes

Core claim

The central claim is that predicting the token just before a target token, using both earlier context and the target itself, is easier than predicting the target from preceding tokens only, and that a model trained solely for that auxiliary task can act as a verifier to improve the main model's choices. Concretely, for each of the top k next-token candidates from an autoregressive model, the auxiliary bidirectional model is prompted with that candidate as the final token; if the auxiliary model's most likely second-to-last token equals the true one, the candidate's probability is multiplied by (1+w). The paper shows that this re-ranking improves next-token accuracy on every tested GPT-2-scale configuration, and that the improvements are statistically significant for the default parameter setting even though they are small in absolute size. The paper also shows that the same decoder architecture, trained with a deterministic permutation of l-token subsequences, learns second-to-last prediction much faster and more accurately than it learns next-token prediction, indicating that the bidirectional task is genuinely easier for the model family.

Load-bearing premise

The load-bearing premise is that the token immediately following a target word is more useful than an earlier token when predicting that target, so a bidirectional predictor can beat a unidirectional one; if this locality assumption fails on a dataset, the refiner cannot improve next-token predictions.

Editorial extensions

If this is right

  • The refiner f_s achieves 15-20 percentage points higher accuracy than f_n on the same data, so bidirectional context is substantially easier to learn than next-token prediction for decoder-only transformers.
  • Algorithm AGR improves next-token accuracy on all three datasets with the default k=15, w=0.05; a t-test gives p<0.05 for each, so the gains are statistically significant.
  • Increasing the boost weight w helps only when f_s is reliable; at w=0.1 the method loses accuracy on openwebtext and Stories, where f_s scores only about 60%, so the refiner's trustworthiness should be measured before weighting it heavily.
  • The deterministic permutation scheme removes the mask token and raises training efficiency relative to BERT-style random masking, offering a faster route to bidirectional pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same re-ranking should apply to any autoregressive decoder, not just GPT-2, and to auxiliary targets other than the immediately preceding token (e.g., two tokens back), because the locality assumption is not specific to the GPT family.
  • Combining AGR with beam search or contrastive decoding seems natural: those methods change the candidate set, and the refiner would simply re-score the new candidates.
  • A testable extension would be to use the refiner during training, e.g., as an auxiliary loss head, rather than only at inference, which could make the next-token model itself learn to produce candidates that the refiner confirms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a generate-then-refine method (Algorithm 1, AGR) for improving next-token predictions of a standard autoregressive GPT model. A second, decoder-only model fs is trained to predict the second-to-last token given the preceding context and a candidate last token; fs is then used to re-rank the top-k next-token candidates produced by the base model fn by upweighting candidates for which fs correctly identifies the actual second-to-last token. Training of fs uses a deterministic permutation of input subsequences that avoids a mask token, yielding an efficiency gain relative to BERT-style masking. Experiments on three GPT-2-scale models and three datasets report that fs has substantially higher accuracy than fn, and that AGR yields small but claimed statistically significant gains over fn on held-out validation data, with the largest gains on SimpleWiki (about 0.2-0.35 percentage points) and smaller gains on Stories and OpenWebText (about 0.02-0.06 points).

Significance. If the central claim were fully supported, this would be a modest but useful contribution: it demonstrates a non-circular, inference-time refinement mechanism that uses a second-to-last-token predictor to improve next-token accuracy, and it introduces a mask-free deterministic masking scheme for decoder-only bidirectional training. The paper ships a public code repository, evaluates on standard datasets and model sizes, and reports results on held-out data, so the main result is not circular. However, the statistical foundation is currently too weak to establish the headline claim of 'consistent and statistically significant gains' across datasets, because each model is trained only once and the reported t-tests reflect evaluation-sampling noise rather than training-seed variability. Given that the gains outside SimpleWiki are on the order of a few hundredths of a percentage point, run-to-run training variance at this scale could easily account for the observed differences. The theoretical justification in Section 4 is also presented imprecisely.

major comments (3)
  1. [Section 3.1 and 3.2, Table 3] The paper states 'We trained each model in Table 2 once.' All reported means and standard deviations come from ten evaluation runs that re-sample 70,000 token predictions from the same fixed model. The t-test reported in Section 3.2 therefore only tests whether AGR beats fn for these particular trained instances, not whether the improvement is reproducible across training runs. This is load-bearing for the abstract's claim of 'consistent and significant gains.' For the two larger datasets, the vanilla configuration (k=15, w=0.05) yields gains of only 0.04±0.02 (Stories) and 0.06±0.04 (OpenWebText) percentage points, with standard deviations that are already large relative to the gains. Run-to-run training variance at GPT-2 scale is typically at least this large, so the reported t-test results cannot rule out the possibility that the gains are due to the particular trained instances. The authors should train multiple seeds (at least 3-5) per dataset and report the distribution of AGR minus fn across seeds, or otherwise provide evidence that training variance is negligible compared to the reported gains.
  2. [Section 4, Eqs. (5)-(6)] The formal statement of the locality assumption is not a well-defined inequality. Both sides of the inequality in Eq. (5) and Eq. (6) begin with the same term ||p*(yt|yi≠t) - p*(yt|y0, y1, ..., yt-1)||, so the inequality reduces to a comparison between distances from p*(yt|yi≠t) to two different conditional distributions, but the second distribution in Eq. (6) is written with a lowercase p(yt|y1, y2, ..., yt-1, yt+1) rather than p*(·), and no distance metric is specified. The prose then asserts that 'This holds true if knowing the next token yt+1 is more useful than knowing a token y0 that occurred much earlier,' but this assertion does not follow from the displayed equations and the cited references on distributed symmetry breaking and correlated initialization do not directly test the language-model setting. The authors should either rigorously reformulate this assumption (e.g., in terms of conditional entropy or Bayes error with a specified distance) or clearly label it as a heuristic conjecture; the empirical results should not be presented as a consequence of this assumption.
  3. [Section 3.2 and Table 3] The paper explores several (k, w) configurations and highlights the best one (e.g., w=0.1 for SimpleWiki with p<0.001) without any correction for multiple comparisons. The text mixes the pre-specified vanilla configuration (k=15, w=0.05) with post-hoc selected configurations when discussing the strength of the results. This inflates the apparent evidence for the method. The primary analysis should be the pre-specified vanilla configuration (or an explicit multi-seed comparison), and the exploratory results should be clearly labeled as such, with appropriate caveats about selection-on-validation.
minor comments (5)
  1. [Abstract] The abstract contains a typo: 'taskssuchaspredicting' should be 'tasks such as predicting'.
  2. [Section 3.2] The formula for the gain is garbled: it reads 'ΔAGR−fn := Accva AGR − Accva s' but the second term should be Accva n (the validation accuracy of the next-token model), not Accva s.
  3. [Table captions] The captions for Tables 1-3 show 'T able' with a space; this should be 'Table'.
  4. [Algorithm 1 and Section 2] The symbol y is reused both for a candidate next token and for the actual second-to-last token y_{t-1} in the indicator function on line 6. Please use distinct notation (e.g., y for the candidate and y*_{t-1} for the ground-truth second-to-last token) to avoid ambiguity.
  5. [Section 2.1 and Section 5] The claimed training-efficiency gain ('from 15% up to 25%') is never measured or compared against a concrete baseline (e.g., wall-clock time against a standard BERT-style masking implementation). A brief quantitative comparison would make the claim more precise.

Circularity Check

0 steps flagged · score 2.0 of 10

No by-construction circularity: the main empirical gain is measured on held-out validation tokens and can even be negative for some settings; only minor self-citations and validation-based hyperparameter selection keep the score above zero.

full rationale

The paper's central claim is that the generate-then-refine algorithm AGR improves next-token accuracy over the base model fn. This is an empirical comparison on a held-out validation split, with 70,000 randomly sampled token predictions per run and ten evaluation runs. Nothing in Algorithm 1 forces an improvement: the paper itself reports negative deltas for w=0.1 on Stories and openwebtext (Table 3), so the re-ranking can hurt. The second-to-last-token model fs is trained separately from fn, and its higher accuracy is measured, not assumed. The theoretical discussion in Section 4 (Eq. 5-6) is explicitly labeled an assumption, and although it cites the author's own prior work [2,20] to argue that locality is common, those citations are analogical rather than a derivation of the empirical result, so they are not load-bearing. Two mild caveats prevent a score of zero: the hyperparameters k and w are varied in Table 3 and the best simplewiki configuration (w=0.1) is highlighted, introducing a degree of validation-selection bias; and Section 3.1 states 'We trained each model in Table 2 once,' so the t-tests in Section 3.2 cover evaluation-run variation rather than training-seed variation. Neither caveat is a by-construction equivalence between inputs and outputs, but both warrant a low nonzero circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method depends on two free hyperparameters (k and w) and a block length l; gains vary with them. Two theoretical assumptions about task difficulty and inductive bias are asserted rather than proved. No new physical entity is introduced.

free parameters (3)
  • k = 2, 5, 15
    Top-k candidate count used in Algorithm 1; varied across runs, and the results depend on it.
  • w = 0.01, 0.05, 0.1
    Probability boost applied when the refiner agrees; chosen by hand, tuned on validation, and the sign of the effect changes for w=0.1 on two datasets.
  • l = 4
    Block length for the deterministic token-shifting scheme used to train f_s; a design choice that also drives the training-efficiency claim.
assumptions (3)
  • ad hoc to paper Bidirectional optimal predictor has lower error than unidirectional predictor under a locality assumption.
    Section 4, Eq 5-6; asserted, not derived, and the cited support is the author's work on distributed computing and neural initialization rather than language data.
  • ad hoc to paper Transformer inductive bias is better suited to second-to-last token prediction than to next-token prediction.
    Assumption (ii) in Section 4; supported only by an overfitting observation, with no theoretical or empirical proof.
  • domain assumption Top-k candidates from f_n contain the correct next token often enough for refinement to help.
    Algorithm 1 depends on this; the paper itself notes that in many openwebtext cases the correct token is not in the top-k, so the method cannot help those cases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Next Tokens via Second-to-Last Predictions with Generate and Refine." pith.science (2026). https://pith.science/paper/F2CZHFXK

@misc{pith2026241115661,
  author       = {Pith},
  title        = {Pith review of: Improving Next Tokens via Second-to-Last Predictions with Generate and Refine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F2CZHFXK}},
  note         = {Machine review of arXiv:2411.15661}
}
read the original abstract

Autoregressive language models like GPT aim to predict next tokens, while autoencoding models such as BERT are trained on tasks such as predicting masked tokens. We train a decoder-only architecture for predicting the second to last token for a sequence of tokens. Our approach yields higher computational training efficiency than BERT-style models by employing a structured deterministic approach to masking tokens. We use our model to improve the next token predictions of a standard GPT by combining both predictions in a ``generate-then-refine'' approach. We demonstrate on different variants of GPT-2 and different datasets that (not unexpectedly) second to last token predictions are much more accurate, i.e., more than 15\% higher accuracy than standard next token predictions. The ``generate-then-refine'' approach also demonstrates notable improvements in next-token predictions, yielding smaller yet consistent and significant gains.

Figures

Figures reproduced from arXiv: 2411.15661 by the authors.

Figure 1
Figure 1. Conceptual outline of our “generate-then-refine” approach using second-to-last token prediction formalized in Algorithm 1 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example of how input sequences are processed to obtain sequences for training the second to last token prediction model fs 3 Evaluation We evaluated Algorithm (1) AGR using three datasets and three variants of the GPT-2 model [17] differing mostly in terms of size, i.e., the number of pa￾rameters. Note that GPT-2 is very similar to newer models such as GPT-3 and GPT-4 (see [22] for a more detailed comparison). More … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 22 canonical work pages

  1. [1]

    In: International conference on machine learning

    Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Gao, J., Piao, S., Zhou, M., et al.: Unilmv2: Pseudo-masked language models for unified language model pre-training. In: International conference on machine learning. pp. 642–652. PMLR (2020)

    work page 2020

  2. [2]

    Journal of the ACM (JACM)63(3), 1–45 (2016)

    Barenboim, L., Elkin, M., Pettie, S., Schneider, J.: The locality of distributed symmetry breaking. Journal of the ACM (JACM)63(3), 1–45 (2016)

    work page 2016

  3. [3]

    Advances in neural information processing systems (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems (2020)

    work page 2020

  4. [4]

    Factual Error Correction for Abstractive Summarization Models

    Cao, M., Dong, Y., Wu, J., Cheung, J.C.K.: Factual error correction for abstractive summarization models. arXiv preprint arXiv:2010.08712 (2020)

    work page Pith review arXiv 2020

  5. [5]

    arXiv preprint arXiv:2104.09061 (2021)

    Chen, S., Zhang, F., Sone, K., Roth, D.: Improving faithfulness in abstractive summarization with contrast candidate generation and selection. arXiv preprint arXiv:2104.09061 (2021)

    arXiv 2021

  6. [6]

    arXiv:1810.04805 (2018)

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. arXiv:1810.04805 (2018)

    arXiv 2018

  7. [7]

    In: Proceedings of the Con- ference on Empirical Methods in Natural Language Processing (2021)

    Dziri, N., Madotto, A., Zaiane, O.R., Bose, A.J.: Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In: Proceedings of the Con- ference on Empirical Methods in Natural Language Processing (2021)

    work page 2021

  8. [8]

    ACM Computing Surveys 55(12), 1–38 (2023) Title Suppressed Due to Excessive Length 11

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Computing Surveys 55(12), 1–38 (2023) Title Suppressed Due to Excessive Length 11

    work page 2023

Show all 28 references
  1. [9]

    Transactions of the Association for Computational Linguistics12, 1417–1440 (2024)

    Kamoi, R., Zhang, Y., Zhang, N., Han, J., Zhang, R.: When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics12, 1417–1440 (2024)

    2024

  2. [10]

    Open AI3, 111–132 (2022)

    Lin, T., Wang, Y., Liu, X., Qiu, X.: A survey of transformers. Open AI3, 111–132 (2022)

    2022

  3. [11]

    arXiv:1907.11692 (2019)

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692 (2019)

    2019 arXiv

  4. [12]

    Advances in Neural Information Processing Systems36 (2024)

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al.: Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems36 (2024)

    2024

  5. [13]

    OpenAI: Gpt-4 technical report (2023)

    2023

  6. [14]

    OpenAI: Hello gpt-4o! https://openai.com/index/hello-gpt-4o/ (2024), ac- cessed: 2024-09-19

    2024

  7. [15]

    Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., Wang, W.Y.: Automatically correcting large language models: Surveying the landscape of diverse automated correctionstrategies.TransactionsoftheAssociationforComputationalLinguistics 12, 484–506 (2024)

    2024

  8. [16]

    Open AI (2018)

    Radford, A.: Improving language understanding by generative pre-training. Open AI (2018)

    2018

  9. [17]

    OpenAI blog (2019)

    Radford,A.,Wu,J.,Child,R.,Luan,D.,Amodei,D.,Sutskever,I.,etal.:Language models are unsupervised multitask learners. OpenAI blog (2019)

    2019

  10. [18]

    The Journal of Machine Learning Research (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research (2020)

    2020

  11. [19]

    arXiv:1910.01108 (2019)

    Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 (2019)

    2019 arXiv

  12. [20]

    Neural Processing Let- ters 54(3), 2249–2266 (2022)

    Schneider, J.: Correlated initialization for correlated data. Neural Processing Let- ters 54(3), 2249–2266 (2022)

    2022

  13. [21]

    Artificial Intelligence Review57(11), 289 (2024)

    Schneider, J.: Explainable generative ai (genxai): A survey, conceptualization, and research agenda. Artificial Intelligence Review57(11), 289 (2024)

    2024

  14. [22]

    arXiv preprint arXiv:2408.00386 (2024)

    Schneider, J.: What comes after transformers?–a selective survey connecting ideas in deep learning. arXiv preprint arXiv:2408.00386 (2024)

    2024 arXiv

  15. [23]

    Advances in neural information pro- cessing systems (2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems (2017)

    2017

  16. [24]

    arXiv preprint arXiv:2411.03687 (2024)

    Xiao, Z., Snoek, C.G.: Beyond model adaptation at test time: A survey. arXiv preprint arXiv:2411.03687 (2024)

    2024 arXiv

  17. [25]

    arXiv preprint arXiv:2410.09675 (2024)

    Xie, Y., Goyal, A., Wu, X., Yin, X., Xu, X., Kan, M.Y., Pan, L., Wang, W.Y.: Coral: Order-agnostic language modeling for efficient iterative refinement. arXiv preprint arXiv:2410.09675 (2024)

    2024 arXiv

  18. [26]

    In: Findings of the Association for Computational Linguistics (2024)

    Xu, W., Deutsch, D., Finkelstein, M., Juraska, J., Zhang, B., Liu, Z., Wang, W.Y., Li, L., Freitag, M.: Llmrefine: Pinpointing and refining large language models via fine-grained actionable feedback. In: Findings of the Association for Computational Linguistics (2024)

    2024

  19. [27]

    Advances in neural information processing systems (2019)

    Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems (2019)

    2019

  20. [28]

    In: Findings of the Association for Computational Linguistics (2021)

    Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., Ghazvinine- jad, M.: Detecting hallucinated content in conditional neural sequence generation. In: Findings of the Association for Computational Linguistics (2021)

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.