Pith. sign in

REVIEW 4 major objections 5 minor 50 references

Enabling Autoregressive Models to Fill In Masked Tokens

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MARIA shows that a single trained linear layer over the concatenated hidden states of a frozen autoregressive model and a frozen masked language model gives the AR model the ability to fill masked tokens, while keeping its fast KV-cached…

desk verdict MARIA is a simple, plausible hybrid for masked infilling; the positional-alignment ambiguity is the one thing I'd want pinned down before trusting the numbers. read the letter →

arxiv 2502.06901 v1 pith:U5W44EE3 submitted 2025-02-09 cs.LG cs.AIcs.CL

Daniel Israel , Aditya Grover , Guy Van den Broeck This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords maskedinfillingautoregressivelanguagemodelhiddenstatefusionKVcachingdiscretediffusionlineardecoderproductofexperts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a frozen autoregressive language model can be given masked-infilling ability, meaning it can predict tokens buried between past and future context, by training just one linear layer on the concatenated hidden states of the AR model and a frozen masked language model. The resulting system, MARIA, keeps the autoregressive model's efficient decoding, which reuses previously computed attention keys and values (the KV cache), and needs only one forward pass through the MLM to provide future context. The authors argue this is a practical way to combine both paradigms without retraining the base models. They report lower perplexity than ModernBERT, SMDM, and DiffuLlama across masking rates from 10 percent to 90 percent, along with higher LLM-judged sample quality. If the claim holds, infilling capability can be added cheaply on top of existing open autoregressive models, avoiding the slow iterative decoding of discrete diffusion.

What carries the argument

The load-bearing object is the linear decoder $W_3 \in \mathbb{R}^{(d_1+d_2)\times v}$ applied to the sequence-aligned concatenation of the AR hidden states $h_1(x)$, computed from the clean input by the causal model, and the MLM hidden states $h_2(M)$, computed from the masked input by the bidirectional model. Truncation and shifting operations align the two hidden-state streams so that both model the same token positions. The training objective makes the decoder an autoregressive distribution conditioned on unmasked tokens, so the loss can be parallelized across the whole sequence like a standard causal Transformer. The product-of-experts initialization $W_3 \leftarrow [W_1/2;\, W_2/2]$ starts the decoder at the average of the two pretrained models' logits, which the paper shows converges much faster and better than random initialization. This single linear layer is what lets the frozen AR model infill without losing its KV-cached inference.

What would settle it

A decisive test is to re-train the MARIA linear decoder after permuting the positions of the masked future tokens in the bidirectional hidden-state stream before the concatenation. If the model's infilling perplexity on a held-out corpus stays about the same, the future-context signal is not doing the work; if performance collapses, the fusion is genuinely using bidirectional information. A second check is replacing the bidirectional stream with random vectors of the same dimension, which should destroy the gap over the AR baseline.

Watch

Extended reading notes

Core claim

The paper's central claim is that the masked-infilling gap of autoregressive models is not an architectural limitation that requires retraining, but a decodable one: the information needed to condition on future tokens is already present in a pretrained bidirectional encoder, and a linear map over the concatenation of the two frozen models' hidden states suffices to combine it with the causal stream. Formally, MARIA optimizes $L_{MARIA} = -\mathbb{E}_{x\sim D,\,m\sim M}\left[\sum_{i\in m} \log \pi_{MARIA}(x_i \mid c(i,m))\right]$, where $c(i,m) = \{x_{<i},\, x_{>i} \cap \bar{m}\}$ is the past context plus all unmasked future tokens, and the distribution is decoded as $\sigma(W_3[h_1(x); h_2(M)])$. With product initialization $W_3 \leftarrow [W_1/2;\, W_2/2]$ and training on 7 to 30 billion tokens, MARIA reports lower infilling perplexity than ModernBERT, SMDM, and DiffuLlama at every tested masking rate, higher ELO ratings under an LLM judge, and KV-cached greedy decoding. Scaling the frozen AR half from 1B to 7B parameters improves every reported infilling number, which the paper presents as a direct path to scaling masked infilling.

Load-bearing premise

The method rests on the assumption that the hidden states of the frozen causal model and the frozen bidirectional model can be aligned positionally and are complementary enough that a single linear layer can combine them; if those representation spaces are too different, the linear fusion cannot deliver the reported infilling gains.

Editorial extensions

If this is right

  • Infilling becomes a cheap add-on: any frozen AR model with a tokenizer-compatible MLM can be upgraded by training one linear layer, with no unfreezing of base weights.
  • MARIA's decoding stays near AR cost: one MLM forward pass to obtain future context, then standard KV-cached AR decoding, so long-context infilling throughput is much higher than iterative discrete diffusion.
  • Scaling works: replacing the 1B OLMo in MARIA with the 7B OLMo improves infilling perplexity at every tested masking rate, from 0.1 to 0.9.
  • The fused representations carry more usable information than the MLM's alone: a linear readout for part-of-speech tagging reaches higher accuracy on MARIA hidden states than on ModernBERT hidden states.
  • Quality and efficiency beat discrete diffusion: at 50 percent word masking, MARIA samples score higher under an LLM judge than SMDM and DiffuLlama, while greedy decoding replaces 256 denoising steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that nothing in the recipe is specific to OLMo or ModernBERT; the same linear-fusion scheme should transfer to other AR/MLM pairs that share a tokenizer, and the cheapest test is swapping in a different AR model and re-training only $W_3$.
  • The simulated-annealing sampler used for unconditional generation points to a general test-time-compute recipe: resampling and temperature annealing could be applied to any model that can condition on both sides, and it should be benchmarked against diffusion at equal wall-clock time.
  • The same-tokenizer constraint is a practical boundary; a learned alignment layer between the two hidden spaces would extend the idea to arbitrary AR/MLM pairs, a direction the paper does not attempt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MARIA, a masked-infilling method that combines a frozen pretrained autoregressive model (OLMo) and a frozen masked language model (ModernBERT) by training a single linear decoder over their concatenated hidden states. The authors argue that this minimal addition gives AR models the ability to condition on future unmasked tokens while preserving KV-cached inference. They report downstream perplexity at several masking rates, throughput measurements, LLM-judged ELO sample quality, a simulated-annealing test-time scaling procedure, and a POS-tagging representation probe, concluding that MARIA significantly outperforms discrete diffusion baselines (SMDM and DiffuLlama).

Significance. If the central mechanism is sound, the contribution is practically interesting: it trains only a small linear layer, reuses two frozen open-source models, and retains KV-cached decoding. The product-initialization idea is a genuine and useful detail, and the controlled POS probing experiment is a sensible sanity check. However, the paper's central empirical claims are currently weakened by three issues: the exact-vs-NELBO perplexity comparison with diffusion baselines, modest and statistically unsupported ELO margins, and, most importantly, an under-specified positional alignment between the AR and MLM hidden-state streams that is load-bearing for the method's validity. The manuscript is therefore promising but not yet in a publishable state.

major comments (4)
  1. [3, Training Procedure; Figure 1; Algorithm 1] The paper never specifies the exact indexing of the AR hidden state used for a masked position i. In a causal decoder, h1(X)_i attends to x_i itself, so decoding π(x_i | x_<i, future unmasked) from h1(X)_i would let the target token leak through the AR stream. The correct construction must use the AR hidden state at position i-1 (or an equivalent shift) together with the MLM hidden state at position i. Figure 1 says only that 'truncating and shifting operations' are performed, and Algorithm 1's line 'ar hidden state <- ar output.hidden states' is ambiguous about whether the full tensor or its last position is passed to the MARIA linear layer. If the unshifted h1(X)_i is used, the reported masked perplexity improvements would be an artifact of copying the target from the AR representation rather than evidence of infilling. Please state the exact index map for both training and inference, and provide a no-leakage sanity check, for example by training with a deliberately corrupted positional offset or by masking the target token from the AR input.
  2. [4, Table 2] The headline perplexity comparison is an exact-vs-bound comparison: for SMDM and DiffuLlama the table reports NELBO upper bounds (entries marked with '≤'), whereas MARIA and ModernBERT entries are exact perplexities. Because a NELBO is an upper bound on the true negative log-likelihood, the reported '≤' values are not directly comparable to exact MARIA values, and the statement that MARIA 'achieves lower perplexity' is not established by this table alone. Please report exact likelihoods for the diffusion baselines, or at minimum discuss the tightness of the bounds on these particular datasets and masking rates. The absence of error bars on every perplexity row is also problematic, since some differences at low mask rates are small (e.g., 2.82 vs 2.92).
  3. [4, Sample Quality; Figure 4] The abstract's claim of 'significantly outperforms' rests partly on ELO win rates of 53.1% and 57.4% against SMDM and DiffuLlama. These are modest margins, and the paper reports no confidence intervals, no number of comparisons per pair, and no judge-agreement statistics. The fact that the same GPT-4o-mini judge rated four models above ground truth also suggests a systematic judge bias that should be discussed. Either soften the 'significantly' language or provide statistical support for the superiority claim.
  4. [3, Eq. (8)] Equation (8) is mathematically incorrect: σ((W1 h1 + W2 h2)/2) is the softmax of the average logits, not σ((π_AR + π_MLM)/2), because π_AR and π_MLM are probability vectors, not logits. The intended product-of-experts interpretation should be written in terms of log-probabilities, including the partition functions, and the absence of a bias term in the initialization should be addressed if the authors claim that the initialization implements an exact ensemble.
minor comments (5)
  1. [Figure 1 caption] There are two typos in the caption: 'pretained' should be 'pretrained' and 'recieves' should be 'receives'.
  2. [4, Representations, Table 3] The POS-accuracy gains are attributed partly to the much larger hidden dimension of the MARIA representations (1024 for ModernBERT vs 3072 and 5120 for MARIA 1B and 7B). A dimension-matched control, or at least an explicit caveat, would make the 'better representations' claim more convincing.
  3. [Algorithm 1] Algorithm 1 uses Python-slice notation 'input ids[prev idx:curr idx]' without defining whether the stop index is exclusive; if this is intended as pseudocode, the slicing convention should be stated.
  4. [4, Test Time Scaling, Figure 5] The simulated-annealing hyperparameters (30% remasking, linear temperature schedule) are given once and used without sensitivity analysis; a small ablation over these choices would strengthen the test-time-scaling claim.
  5. [4, Throughput, Figure 3] The throughput figure is described as an average over 10 runs with 2 warm-ups, but no variance or error bars are shown; reporting the spread would help assess the reliability of the throughput ranking.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: MARIA's trained linear decoder is evaluated against held-out external data and external baselines, and the only self-citations are not load-bearing.

full rationale

MARIA's derivation chain is self-contained. The trainable component is a single linear decoder W3 trained to minimize LMARIA, the conditional negative log likelihood of masked tokens given past tokens and unmasked future tokens. The reported downstream perplexity is computed on held-out examples from five external corpora, not on the training set, and the comparisons are against external models (ModernBERT, SMDM, DiffuLlama) plus an external LLM judge (GPT-4o-mini). The product initialization W3 = [W1/2; W2/2] is an algebraic initialization derived from existing frozen weights; it is not a fit to the evaluation outcome, and Figure 2 merely compares it with random initialization. The only potential circularity concern is target leakage through the AR stream: if the unshifted h1(X)_i were paired with mask i, the model would see the target token. But the paper's Figure 1 states that 'truncating and shifting operations' align the two streams, and Algorithm 1 feeds the AR model only input ids[prev idx:curr idx] before the current mask while taking the MLM state at the masked index, which is the non-leaking alignment. The text is terse but does not exhibit an explicit reduction of the prediction to the input target. The self-citations (Zhao et al. 2024, Prepacking; Liu et al. 2024a, Discrete Copula Diffusion) appear in related-work enumerations and are not load-bearing for MARIA's infilling claim. Therefore no step of the paper's argument reduces its reported masked-infilling results to its own fitted inputs by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities; its free choices are hyperparameters and evaluation settings. The key assumptions are representational compatibility between the frozen models and the validity of the evaluation baselines.

free parameters (3)
  • Masking rate distribution = Beta(2.5, 2.5)
    Chosen based on prior work (Shen et al., 2023) rather than derived; affects the training distribution over mask ratios.
  • Simulated annealing schedule = 30% tokens remasked, linear temperature schedule from 1 to 0
    Heuristic for unconditional generation; not derived from first principles.
  • Evaluation masking ratio for sample quality = 50% of words masked
    A single operating point chosen for ELO evaluation; results may vary with mask ratio.
assumptions (5)
  • domain assumption ModernBERT and OLMo share a tokenizer and their hidden states can be aligned and concatenated by position after truncation and shifting operations.
    Required by the MARIA objective and Algorithm 1; the truncation and shifting operations are only described in the Figure 1 caption.
  • domain assumption A linear layer over concatenated hidden states is sufficient to combine the two models' predictive abilities.
    Core architectural hypothesis; only empirical validation is provided.
  • domain assumption MLM hidden states computed once on the original masked input remain valid as masked tokens are filled left-to-right, with the AR model supplying generated left context.
    Algorithm 1 uses a single MLM forward pass; this is a deliberate approximation of the training objective.
  • domain assumption The NELBO computed by sampling is a tight upper bound on true perplexity for discrete diffusion baselines, making exact-versus-bound comparisons meaningful.
    Section 4, Downstream Perplexity; relies on prior evidence (Kingma et al., 2023) that the bounds are tight.
  • domain assumption GPT-4o mini provides an unbiased judge of infilling quality for ELO ratings.
    Section 4, Sample Quality; no human validation or bias analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enabling Autoregressive Models to Fill In Masked Tokens." pith.science (2026). https://pith.science/paper/U5W44EE3

@misc{pith2026250206901,
  author       = {Pith},
  title        = {Pith review of: Enabling Autoregressive Models to Fill In Masked Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5W44EE3}},
  note         = {Machine review of arXiv:2502.06901}
}
read the original abstract

Historically, LLMs have been trained using either autoregressive (AR) or masked language modeling (MLM) objectives, with AR models gaining dominance in recent years. However, AR models are inherently incapable of masked infilling, which is the ability to predict masked tokens between past and future context. In contrast, MLM models suffer from intrinsic computational inefficiencies during both training and inference that hinder their scalability. This work introduces MARIA (Masked and Autoregressive Infilling Architecture), a novel approach that leverages the strengths of both paradigms to achieve state-of-the-art masked infilling performance. MARIA combines a pre-trained MLM and AR model by training a linear decoder that takes their concatenated hidden states as input. This minimal modification enables the AR model to perform infilling while retaining its inherent advantages in terms of faster inference with KV caching. Our results demonstrate that MARIA significantly outperforms existing methods, namely discrete diffusion models, on masked infilling tasks.

Figures

Figures reproduced from arXiv: 2502.06901 by the authors.

Figure 1
Figure 1. MARIA architecture and training pipeline. MARIA takes two frozen pretained models: one MLM and one AR. As input, the MLM recieves the masked inputs and the AR model recieves the denoised inputs. We compute the hidden states under each model and perform truncating and shifting operations to ensure both hidden states model the same tokens. MARIA trains a linear layer to predict the logits of each masked input on the c… view at source ↗
Figure 2
Figure 2. Comparing evaluation loss curves for two different weight initializations. Product initialization (ours) is a far better weight initialization than random weight initialization, leading to faster training and better convergence. receives the masked inputs such that we compute the hid￾den state [h1(X); h2(M)]1:n concatenated on the sequence dimension. These are then decoded with W3 to next token logits. Thus, the aut… view at source ↗
Figure 4
Figure 4. ELO scores for masked infilling. We perform infill￾ing on downstream data with words masked 50 percent. Using GPT4o-mini as a judge we compute the ELO scores for each model respectively. MARIA 7B and 1B have the highest rating ELO rat￾ing under the Bradley-Terry model. use 256 denoising steps. For ModernBERT and MARIA models, we decode greedily. We assess sample quality using an ELO system judged by GPT-4o mini (Ach… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Scaling test time compute for unconditional genera￾tion. We compare our simulated annealing inference approach for MARIA to our baseline discrete diffusion methods. MARIA 1B using simulated annealing effectively trades-off quality (as mea￾sured by generative perplexity…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 9 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    and Tsitsiklis, J

    Bertsimas, D. and Tsitsiklis, J. Simulated annealing. Statistical science, 8 0 (1): 0 10--15, 1993

    work page 1993

  4. [4]

    One billion word benchmark for measuring progress in statistical language modeling, 2014

    Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. One billion word benchmark for measuring progress in statistical language modeling, 2014

    work page 2014

  5. [5]

    N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J

    Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/abs/2403.04132

    arXiv 2024

  6. [6]

    B., Bierbaum, M., O'Keeffe, K

    Clement, C. B., Bierbaum, M., O'Keeffe, K. P., and Alemi, A. A. On the use of arxiv as a dataset, 2019

    2019

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    arXiv 2019

  8. [8]

    Enabling language models to fill in the blanks

    Donahue, C., Lee, M., and Liang, P. Enabling language models to fill in the blanks. arXiv preprint arXiv:2005.05339, 2020

    arXiv 2005

Show all 50 references
  1. [9]

    Glm: General language model pretraining with autoregressive blank infilling, 2022

    Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. Glm: General language model pretraining with autoregressive blank infilling, 2022. URL https://arxiv.org/abs/2103.10360

    2022 arXiv

  2. [10]

    Incoder: A generative model for code infilling and synthesis, 2023

    Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., tau Yih, W., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis, 2023. URL https://arxiv.org/abs/2204.05999

    2023 arXiv

  3. [11]

    Scaling diffusion language models via adaptation from autoregressive models

    Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024

    2024 arXiv

  4. [12]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  5. [13]

    H., Ivison, H., Magnusson, I., Wang, Y., et al

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  6. [14]

    and Hashimoto, T

    Gulrajani, I. and Hashimoto, T. B. Likelihood-based diffusion language models, 2023. URL https://arxiv.org/abs/2305.18619

    2023 arXiv

  7. [15]

    Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural computation, 14 0 (8): 0 1771--1800, 2002

    2002

  8. [16]

    Denoising diffusion probabilistic models, 2020

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models, 2020. URL https://arxiv.org/abs/2006.11239

    2020 arXiv

  9. [17]

    The curious case of neural text degeneration, 2020

    Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration, 2020. URL https://arxiv.org/abs/1904.09751

    2020 arXiv

  10. [18]

    A., Bastings, J., Poole, B., van den Berg, R., and Salimans, T

    Hoogeboom, E., Gritsenko, A. A., Bastings, J., Poole, B., van den Berg, R., and Salimans, T. Autoregressive diffusion models. CoRR, abs/2110.02037, 2021. URL https://arxiv.org/abs/2110.02037

    2021 arXiv

  11. [19]

    P., Salimans, T., Poole, B., and Ho, J

    Kingma, D. P., Salimans, T., Poole, B., and Ho, J. Variational diffusion models, 2023. URL https://arxiv.org/abs/2107.00630

    2023 arXiv

  12. [20]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023

    2023

  13. [21]

    Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities

    Lee, M., Liang, P., and Yang, Q. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI conference on human factors in computing systems, pp.\ 1--19, 2022

    2022

  14. [22]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Lewis, M. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019

    1910 arXiv

  15. [23]

    A., and Choi, Y

    Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.03023, 2021

    2021 arXiv

  16. [24]

    Liu, A., Broadrick, O., Niepert, M., and Broeck, G. V. d. Discrete copula diffusion. arXiv preprint arXiv:2410.01949, 2024 a

    2024 arXiv

  17. [25]

    Multi-task learning based pre-trained language model for code completion

    Liu, F., Li, G., Zhao, Y., and Jin, Z. Multi-task learning based pre-trained language model for code completion. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp.\ 473--485, 2020

    2020

  18. [26]

    Think while you generate: Discrete diffusion with planned denoising, 2024 b

    Liu, S., Nam, J., Campbell, A., Stärk, H., Xu, Y., Jaakkola, T., and Gómez-Bombarelli, R. Think while you generate: Discrete diffusion with planned denoising, 2024 b . URL https://arxiv.org/abs/2410.06264

    2024 arXiv

  19. [27]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

    Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024 c

    2024

  20. [28]

    Discrete diffusion language modeling by estimating the ratios of the data distribution

    Lou, A., Meng, C., and Ermon, S. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023

    2023

  21. [29]

    Manning, C. D. Part-of-speech tagging from 97\ In International conference on intelligent text processing and computational linguistics, pp.\ 171--189. Springer, 2011

    2011

  22. [30]

    Pointer sentinel mixture models, 2016

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016

    2016

  23. [31]

    Meet in the middle: A new pre-training paradigm, 2023

    Nguyen, A., Karampatziakis, N., and Chen, W. Meet in the middle: A new pre-training paradigm, 2023. URL https://arxiv.org/abs/2303.07295

    2023 arXiv

  24. [32]

    Scaling up masked diffusion models on text, 2024

    Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text, 2024. URL https://arxiv.org/abs/2410.18514

    2024 arXiv

  25. [33]

    Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2024

    Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2024. URL https://arxiv.org/abs/2406.03736

    2024 arXiv

  26. [34]

    N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context, 2016. URL https://arxiv.org/abs/1606.06031

    2016 arXiv

  27. [35]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  28. [36]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  29. [37]

    S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J

    Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models, 2024. URL https://arxiv.org/abs/2406.07524

    2024 arXiv

  30. [38]

    A., Faghri, F., Cho, M., Nabi, M., Naik, D., and Farajtabar, M

    Samragh, M., Mirzadeh, I., Vahid, K. A., Faghri, F., Cho, M., Nabi, M., Naik, D., and Farajtabar, M. Scaling smart: Accelerating large language model pre-training with small model initialization. arXiv preprint arXiv:2409.12903, 2024

    2024 arXiv

  31. [39]

    Berts are generative in-context learners, 2024

    Samuel, D. Berts are generative in-context learners, 2024. URL https://arxiv.org/abs/2406.04823

    2024 arXiv

  32. [40]

    Sang, E. F. T. K. and Meulder, F. D. Introduction to the conll-2003 shared task: Language-independent named entity recognition, 2003. URL https://arxiv.org/abs/cs/0306050

    2003 arXiv

  33. [41]

    Film: Fill-in language models for any-order generation, 2023

    Shen, T., Peng, H., Shen, R., Fu, Y., Harchaoui, Z., and Choi, Y. Film: Fill-in language models for any-order generation, 2023. URL https://arxiv.org/abs/2310.09930

    2023 arXiv

  34. [42]

    Long horizon temperature scaling

    Shih, A., Sadigh, D., and Ermon, S. Long horizon temperature scaling. In International Conference on Machine Learning, pp.\ 31422--31434. PMLR, 2023

    2023

  35. [43]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  36. [44]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  37. [45]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

    Warner, B., Chaffin, A., Clavi \'e , B., Weller, O., Hallstr \"o m, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and infere...

    2024 arXiv

  38. [46]

    Fofo: A benchmark to evaluate llms' format-following capability

    Xia, C., Xing, C., Du, J., Yang, X., Feng, Y., Xu, R., Yin, W., and Xiong, C. Fofo: A benchmark to evaluate llms' format-following capability. arXiv preprint arXiv:2402.18667, 2024

    2024 arXiv

  39. [47]

    Antlm: Bridging causal and masked language models, 2024

    Yu, X., Guo, B., Luo, S., Wang, J., Ji, T., and Wu, Y. Antlm: Bridging causal and masked language models, 2024. URL https://arxiv.org/abs/2412.03275

    2024 arXiv

  40. [48]

    Character-level convolutional networks for text classification, 2016

    Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification, 2016. URL https://arxiv.org/abs/1509.01626

    2016 arXiv

  41. [49]

    Zhao, S., Israel, D., Broeck, G. V. d., and Grover, A. Prepacking: A simple method for fast prefilling and increased throughput in large language models. arXiv preprint arXiv:2404.09529, 2024

    2024 arXiv

  42. [50]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.