Pith. sign in

REVIEW 2 major objections 6 minor 90 references

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that in masked language models, the main cost of masking is not the [MASK] tokens themselves but the corrupted, ambiguous meaning they leave behind, and that expanding each mask into multiple dependency-linked states…

desk verdict A solid method paper whose headline analysis—'corrupted semantics matters more than unreal tokens'—is confounded by the Repeated MLM design, so the analytical claim should be treated cautiously even though the ExLM architecture itself is worth refereeing. read the letter →

arxiv 2501.13397 v5 pith:D3PFMI63 submitted 2025-01-23 cs.CL cs.LG

Kangjie Zheng , Junwei Yang , Siyue Liang , Bin Feng , Zequn Liu , Wei Ju , Zhiping Xiao , Ming Zhang This is my paper · ORCID
classification cs.CLcs.LG
keywords maskedlanguagemodelingcorruptedsemantics[MASK]tokenssemanticmultimodalitystatesexpansiontransitionmatrixSMILESrepresentationlearningmolecularpropertyprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked language models recover tokens that were replaced by [MASK], but this paper argues that the operation damages the input twice: it inserts tokens that do not exist in real text, and it destroys part of the context's meaning. To separate the two, the paper runs a Repeated MLM experiment in which each token is copied k times before masking, so the share of [MASK] tokens is p while the share of tokens whose meaning is fully gone is $p^k$. The results are read as showing that the destroyed meaning, which it calls corrupted semantics, drives performance loss far more than the presence of unreal tokens. On that diagnosis it builds ExLM, which replaces each [MASK] with several expanded hidden states, tells the states apart with 2D positions, and models their dependencies with a transition matrix. The paper reports downstream gains over BERT-style baselines on GLUE and SQuAD and over SMILES-BERT on molecular property prediction.

What carries the argument

The load-bearing mechanism is context enhancement through states expansion and dependency capture. A single [MASK] embedding is cloned into k hidden states, 2D RoPE gives each clone a distinct position (i,1) through (i,k), and an attention-like upper-triangular matrix E acts as a directed acyclic graph adjacency matrix over the expanded states; a CTC-style dynamic program then marginalizes over all alignments of states to the masked target tokens during pre-training. This combination gives the model several candidate meanings per missing token while forcing the candidates into a tractable dependency structure, which is what reduces the ambiguity the paper identifies as the main cost of masking.

What would settle it

Run the Repeated MLM comparison with cells matched on the expected number of visible copies per token, k(1-p), as well as on the corrupted-semantics fraction $p^k$; if downstream accuracy tracks the visible-copy count rather than the corrupted-semantics fraction, the paper's central conclusion is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the corrupted semantics problem has a substantially greater impact on MLM performance than the unreal tokens problem. Corrupted semantics means that masking removes so much context that a masked position can be completed in several incompatible ways; the paper connects this to the multimodality phenomenon and shows that prediction entropy rises as semantics corruption increases. The Repeated MLM experiment is the evidence: with the fraction of [MASK] tokens held at p, varying k changes the fraction of fully destroyed tokens as $p^k$, and performance moves much more when that fraction moves than when only p moves. ExLM is the proposed remedy: each [MASK] is expanded into k hidden states, 2D rotary position embeddings distinguish the clones, and an upper-triangular transition matrix models dependencies among the states while a dynamic-programming objective aligns states to target tokens. On the paper's own terms this machinery lowers prediction entropy and improves text and SMILES benchmarks.

Load-bearing premise

The load-bearing assumption is that the Repeated MLM experiment isolates corrupted semantics: it treats p and k as moving only the share of [MASK] tokens and the share of fully destroyed tokens, when higher k also leaves more surviving visible copies of each token, so the observed tolerance to [MASK] tokens could come partly from added redundancy.

Editorial extensions

If this is right

  • If corrupted semantics is the dominant failure mode, masked language models should tolerate far higher mask ratios than 15% as long as the surviving context still pins down the meaning, which the Repeated MLM results display.
  • Expanding each [MASK] into multiple dependency-linked states is a workable remedy: ExLM improves average GLUE over the vanilla MLM and beats a same-cost vanilla MLM++ baseline.
  • The remedy transfers across modalities: on the MoleculeNet benchmark, ExLM beats SMILES-BERT trained with the same data, architecture, and hyperparameters.
  • Both added components matter: ablations show that removing the transition matrix or the 2D RoPE lowers accuracy, with the transition matrix having the larger effect.
  • The optimal number of expanded states is coupled to mask ratio: k around 4 works best at 15% masking, while higher mask ratios benefit from larger k up to a point before redundancy hurts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same diagnosis predicts that masking strategies which preserve semantic clues, for example replacing tokens with plausible distractors instead of erasing them, should recover part of the same benefit, a cheaper alternative to ExLM's states expansion that the paper does not test.
  • Editorial inference: the state-alignment objective is a latent-alignment formulation, so the machinery could in principle be reused for other sequence tasks with multiple valid outputs per input position, beyond masked pre-training.
  • Editorial inference: the Repeated MLM protocol does not equate the expected number of visible copies per token across cells, so the quantitative magnitude of the corrupted-semantics effect remains open; a replication that matches visible copies as well as $p^k$ would be the decisive check.
  • Editorial inference: the text and SMILES results suggest the expansion is domain-agnostic, but the paper does not run it on proteins, code, or other sequence modalities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies the impact of [MASK] tokens in masked language models by distinguishing two potential harms: the presence of unreal tokens and the corruption of context semantics. To disentangle these, the authors introduce a "Repeated MLM" experiment in which each input token is repeated k times before masking with probability p; they argue that the share of unreal tokens is p while the share of corrupted semantics is p^k. From this experiment they conclude that corrupted semantics is the dominant factor degrading MLM performance. Based on this analysis, they propose ExLM, which expands each [MASK] token into k hidden states, distinguishes these states with 2D RoPE, and models dependencies among them with a transition matrix trained via a dynamic-programming state-alignment objective. ExLM is evaluated on GLUE, SQuAD 2.0, and MoleculeNet, outperforming same-data/settings MLM baselines, with additional ablations showing the importance of the transition matrix and 2D RoPE. The paper releases code.

Significance. If the central analytical claim were sound, the paper would provide a useful reframing of MLM degradation, suggesting that the main cost of masking is the loss of coherent context semantics rather than the mere presence of [MASK] symbols. The proposed ExLM architecture is a nontrivial and plausible extension, and the empirical evaluation is more careful than is typical: it includes same-data/settings baselines (RoBERTa*, SMILES-BERT*), an equal-compute control (Vanilla MLM++), and component ablations, with code released. However, the central analytical conclusion is not established by the Repeated MLM design because of the redundancy confound detailed below. The method-side evidence is considerably stronger and may survive even if the analytical claim is weakened, which is why the paper warrants major revision rather than rejection.

major comments (2)
  1. [Section 3.1, Figure 3] The Repeated MLM experiment does not isolate the corrupted-semantics share from the amount of visible evidence available to the model. In the setup, each of the k copies of a token is independently masked with probability p, so the expected number of visible copies of each original token is k(1-p). This quantity changes across the grid in Figure 3: at (k=1, p=15%) it is 0.85, while at (k=8, p=78.9%) it is about 1.69. Cells with high k and high p therefore supply strictly more recoverable evidence per token, making the reconstruction task easier for reasons unrelated to the corrupted-semantics share p^k. The control argument in Section 3.1 and Appendix C holds the [MASK] share p fixed but does not hold the evidence level fixed; the same confound affects the constant-corruption diagonal comparisons, since p^k can be constant while k(1-p) varies. As a result, the observation that performance varies more with p^k than with p could be explained by the redundancy gradient rather than by the relative importance of corrupted semantics versus unreal tokens. The central analytical claim of Section 3.2 is therefore not established as written.
  2. [Section 3.2, Appendix C] The proof in Appendix C establishes only the expectation and variance of the proportion of fully masked tokens (p^k). It does not establish that all other quantities that could affect MLM difficulty are held constant across the compared cells. In particular, the expected number of visible copies per token, k(1-p), and the total sequence length (which is scaled by k, as noted in Appendix B.1) both vary across the grid. The paper needs an explicit control for these variables. One possible fix is to adjust p as a function of k so that k(1-p) is constant (e.g., p = 1 - c/k for a fixed c), or to pad the input with separately masked filler tokens. Without such a control, the conclusions of Section 3.2 and the entropy analysis in Figure 5 remain confounded.
minor comments (6)
  1. [Throughout] The model name is written inconsistently as "EXLM" and "ExLM"; please unify the notation.
  2. [Figure 12] The label "Vanila MLM" should read "Vanilla MLM".
  3. [Appendix I] The text says "warmup stesp" instead of "warmup steps".
  4. [Section 5.2] The sentence "We evaluate the EXLM model using the the GLUE" contains a duplicated article.
  5. [Table 2 caption] The word "pervious" should be "previous".
  6. [Section 4.3] The complexity statement switches from O(M × L^2) to O(M) via parallelization; please clarify whether the claim refers to wall-clock time under idealized parallel hardware or to a different computational metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main performance claims are anchored to external same-budget baselines, and the analytical concerns are control/interpretation issues rather than constructive reductions.

full rationale

The paper's central performance claims are self-contained. ExLM is evaluated against RoBERTa* and SMILES-BERT* trained with the same data, architecture, and hyperparameters, and against Vanilla MLM++ under equal training cost; these are external same-budget comparisons, not quantities fitted by the paper's own definitions. The state-alignment objective is a marginal likelihood over actual target tokens computed by dynamic programming; it is not fitted to benchmark scores, so no fitted input is renamed as a prediction. The transition-matrix and 2D-RoPE components are ablated rather than assumed. The only self-citations (e.g., Huang et al. 2024 for the reused CUDA dynamic-programming implementation) concern a reusable algorithm and are not load-bearing for the conclusion. Two validity concerns are worth flagging but are not circularity under the stated hard rules: (1) the Repeated MLM control varies the expected number of surviving copies per token, k(1-p), across the grid, so the Section 3.2 conclusion that corrupted semantics dominates is confounded by redundancy; (2) the ExLM entropy comparison averages per-state entropies, so the measured reduction can reflect splitting the predictive distribution rather than reducing total uncertainty. Neither concern exhibits a claim derived from its own definition by construction or a fitted parameter renamed as a prediction, so this paper receives a circularity score of 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claims rest on (i) the decoupling experiment in Section 3.1, whose validity requires that repetition changes nothing except the corruption fraction, an assumption the paper does not test; (ii) ExLM's expanded-state architecture, which borrows its DP alignment from DA-Transformer and its positional mechanism from RoPE; and (iii) benchmark comparisons that, apart from the equal-compute Vanilla MLM++ control, are standard. No external entity or fitted constant is needed beyond the hyperparameters listed, but the analysis that motivates the method depends on the untested decoupling premise.

free parameters (4)
  • k (number of expanded states) = 4 for text, 2 for SMILES, 2/4/8 explored in ablations
    Selected by validation performance; Table 4 shows k=4 optimal at 15% mask and larger k needed at higher mask ratios; results depend on k.
  • mask ratio p = 0.15 (text and SMILES pre-training)
    Standard MLM setting from BERT; Table 4 shows performance varies with p, and ExLM's advantage over MLM grows at high p, so p is an input choice the conclusions depend on.
  • corrupted-semantics exclusion threshold = cells with p^k < 0.0005 excluded
    In the Repeated MLM grid, cells with less than 0.05% corrupted semantics are dropped as 'unstable'; this post hoc cutoff shapes the reported comparison grid.
  • entropy filtering rule = only fully masked tokens (all k copies masked) averaged over k copies
    Appendix E.1 defines the entropy metric with this filter; the multimodality trend in Figure 5 is computed under this rule, which excludes partially visible tokens.
assumptions (4)
  • domain assumption Repeating each token k times preserves the semantic content of the original sequence after masking a fraction p of copies.
    Section 3.1 and Figure 2: the experiment's validity requires that unmasked copies fully restore the lost meaning and that the model can exploit this redundancy in the same way it uses original context.
  • ad hoc to paper The two factors of interest are fully described by the share of [MASK] tokens (p) and the share of corrupted tokens (p^k), with no other quantity (e.g., amount of visible evidence per token, sequence length) varying in a way that affects the comparison.
    Section 3.1 and Appendix C: E[s] = p^k is derived, but the protocol also changes the expected number of visible copies per token, k(1-p). This is the load-bearing decoupling premise.
  • domain assumption Marginalizing over all state-to-target alignments via the DA-Transformer DP recursion correctly scores the likelihood of target tokens in the expanded-state model.
    Section 4.3, equations for L_SA and f_{i,u}; the recursion requires the DAG to contain a valid path for each target sequence, which is assumed rather than proven for arbitrary mask patterns.
  • domain assumption Fine-tuning the model on inputs repeated k times (same as pre-training) yields a fair estimate of downstream transfer.
    Section 3.2: 'during downstream fine-tuning, the input is repeated with the same repetition times k as in pre-training.' The unusual repeated fine-tuning distribution is never validated against ordinary fine-tuning.
invented entities (2)
  • Expanded hidden states ([MASK] clones)
    purpose: Multiple latent states per masked position to represent multiple plausible meanings and reduce intra-token multimodality.
    Validated only in-paper via the case study (Figures 7 and 10) and entropy drop (Appendix P.1); no external falsifiable handle (e.g., a probe on an independent task or human annotation) is provided.
  • Corrupted semantics quantity (p^k)
    purpose: Quantifies the fraction of tokens whose meaning is fully destroyed by masking, used to decouple the two [MASK] effects.
    Defined by the authors' own repetition/masking construction in Appendix C; it is a mathematical identity for their setup, not an independently measurable quantity, though entropy correlates with it in-paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models." pith.science (2026). https://pith.science/paper/D3PFMI63

@misc{pith2026250113397,
  author       = {Pith},
  title        = {Pith review of: ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D3PFMI63}},
  note         = {Machine review of arXiv:2501.13397}
}
read the original abstract

Masked Language Models (MLMs) have achieved remarkable success in many self-supervised representation learning tasks. MLMs are trained by randomly masking portions of the input sequences with [MASK] tokens and learning to reconstruct the original content based on the remaining context. This paper explores the impact of [MASK] tokens on MLMs. Analytical studies show that masking tokens can introduce the corrupted semantics problem, wherein the corrupted context may convey multiple, ambiguous meanings. This problem is also a key factor affecting the performance of MLMs on downstream tasks. Based on these findings, we propose a novel enhanced-context MLM, ExLM. Our approach expands [MASK] tokens in the input context and models the dependencies between these expanded states. This enhancement increases context capacity and enables the model to capture richer semantic information, effectively mitigating the corrupted semantics problem during pre-training. Experimental results demonstrate that ExLM achieves significant performance improvements in both text modeling and SMILES modeling tasks. Further analysis confirms that ExLM enriches semantic representations through context enhancement, and effectively reduces the semantic multimodality commonly observed in MLMs.

Figures

Figures reproduced from arXiv: 2501.13397 by the authors.

Figure 1
Figure 1. Illustrations of the vanilla MLM (a) and EXLM (b). MLM can be affected by the multimodality problem. In EXLM, the model creates multiple hidden states for each [MASK] token (e.g., [M1,1], [M1,2], [M2,1], [M2,2]). By leveraging a larger semantic space and explicitly modeling the dependencies between these states, the model can capture richer semantic information in the enhanced context while mitigating the effects of… view at source ↗
Figure 2
Figure 2. Illustrations of the Repeated MLM experiment. Each token in the model input is repeated before being masked. The ar￾tificial redundancy introduced into the input ensures that replacing a token with [MASK] ([M]) does not necessarily lead to semantic corruption in the context, as the repeated tokens provide additional information to preserve the original semantics. A token is regarded as having corrupted semantics onl… view at source ↗
Figure 3
Figure 3. Results of the Repeated MLM experiment. These are the evaluation results (i.e., accuracy) of MLMs with different repetition times k and mask ratios p on the MNLI task (Williams et al., 2018), with results of similar performance highlighted in similar colors. Specifically, let’s define a sequence of tokens as X = [x1, x2, . . . , xn]. In the MLM pre-training process, some of the tokens in X are randomly replaced with… view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Entropy analysis in the Repeated MLM experiment. We visualize the entropy (in bits) of different models during mask prediction. Larger semantics corruption significantly leads to an increase in the entropy of the model’s prediction distribution. cantly different recons…
Figure 6
Figure 6. Figure 6: Overview of our proposed EXLM. EXLM creates multiple expanded states for each [MASK] token, providing a larger semantic space and a stronger ability to capture the missing semantics in the context. Additionally, it explicitly models the semantic dependencies between th…
Figure 7
Figure 7. Figure 7: Case study. We visualize the model’s predictions when the input is “This is [MASK], and I’m very [MASK] to see this.” (k = 4). The yellow nodes correspond to the expanded states of the first [MASK] token, while the brown nodes correspond to those of the second [MASK] t…
Figure 8
Figure 8. Figure 8: The results of the Repeated MLM experiments on MNLI-m, QNLI, QQP, and RTE tasks. We use accuracy in these four tasks as the metrics. And for all tasks, higher values are better. Similar performance are marked with similar colors. We extend the Repeated MLM experiment t…
Figure 9
Figure 9. Figure 9: The training loss, mask prediction accuracy and perplexity (PPL) curves of MLMs in the Repeated MLM experiments. We plot the training curves of the MLMs with different repetition times k and mask ratios p in the Repeated MLM experiments, as shown in [PITH_FULL_IMAGE:f…
Figure 10
Figure 10. Figure 10: We visualize the model’s predictions when the input is “This is [MASK], and I’m very [MASK] to see this.” (k = 4). The yellow nodes represent the expanded states corresponding to the first [MASK] token, while the brown nodes represent the expanded states corresponding…
Figure 11
Figure 11. Figure 11: The visualization of the DAG from EXLM. It shows that the edge weight between different nodes is directly related to the semantic dependency between those nodes. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Entropy analysis of EXLM with different mask ratios p and numbers of expanded states k. EXLM demonstrates significantly lower uncertainty than MLM, and a larger k further reduces the EXLM’s uncertainty, enhancing its better modeling capability. P.2. Training Efficienc…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

90 extracted references · 43 canonical work pages

  1. [1]

    Unilmv2: Pseudo-masked language models for unified language model pre-training

    Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Gao, J., Piao, S., Zhou, M., et al. Unilmv2: Pseudo-masked language models for unified language model pre-training. In International conference on machine learning, pp.\ 642--652. PMLR, 2020

    2020

  2. [2]

    Beit: Bert pre-training of image transformers

    Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021

    arXiv 2021

  3. [3]

    The fifth pascal recognizing textual entailment challenge

    Bentivogli, L., Clark, P., Dagan, I., and Giampiccolo, D. The fifth pascal recognizing textual entailment challenge. In TAC, 2009

    2009

  4. [4]

    Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation

    Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation (SemEval), 2017

    2017

  5. [5]

    Chemberta: large-scale self-supervised pretraining for molecular property prediction

    Chithrananda, S., Grand, G., and Ramsundar, B. Chemberta: large-scale self-supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885, 2020

    arXiv 2010

  6. [6]

    Electra: Pre-training text encoders as discriminators rather than generators

    Clark, K. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020

    arXiv 2003

  7. [7]

    The pascal recognising textual entailment challenge

    Dagan, I., Glickman, O., and Magnini, B. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005

    2005

  8. [8]

    "Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction

    Dai, Y., Li, L., Zhou, C., Feng, Z., Zhao, E., Qiu, X., Li, P., and Tang, D. " is whole word masking always better for chinese bert?": Probing on chinese grammatical error correction. arXiv preprint arXiv:2203.00286, 2022

    work page Pith review arXiv 2022

Show all 90 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  2. [10]

    Dolan, W. B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP), 2005

    2005

  3. [11]

    Unified language model pre-training for natural language understanding and generation

    Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32, 2019

    2019

  4. [12]

    Glm: General language model pretraining with autoregressive blank infilling

    Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360, 2021

    2021 arXiv

  5. [13]

    Prottrans: Toward understanding the language of life through self-supervised learning

    Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intell...

    2021

  6. [14]

    A bioactivity foundation model using pairwise meta-learning

    Feng, B., Liu, Z., Huang, N., Xiao, Z., Zhang, H., Mirzoyan, S., Xu, H., Hao, J., Xu, Y., Zhang, M., et al. A bioactivity foundation model using pairwise meta-learning. Nature Machine Intelligence, 6 0 (8): 0 962--974, 2024

    2024

  7. [15]

    Codebert: A pre-trained model for programming and natural languages

    Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020

    2002 arXiv

  8. [16]

    Contextual representation learning beyond masked language modeling

    Fu, Z., Zhou, W., Xu, J., Zhou, H., and Li, L. Contextual representation learning beyond masked language modeling. arXiv preprint arXiv:2204.04163, 2022

    2022 arXiv

  9. [17]

    The third pascal recognizing textual entailment challenge

    Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. The third pascal recognizing textual entailment challenge. In ACL-PASCAL workshop on textual entailment and paraphrasing, 2007

    2007

  10. [18]

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks

    Graves, A., Fern \'a ndez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp.\ 369--376, 2006

    2006

  11. [19]

    O., and Socher, R

    Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017

    2017 arXiv

  12. [20]

    Graphcodebert: Pre-training code representations with data flow

    Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366, 2020

    2009 arXiv

  13. [21]

    B., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I

    Haim, R. B., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I. The second pascal recognising textual entailment challenge. In PASCAL Challenges Workshop on Recognising Textual Entailment, 2006

    2006

  14. [22]

    J., Oktay, D., Lin, Z., Verkuil, R., Tran, V

    Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with a language model. Science, pp.\ eads0018, 2025

    2025

  15. [23]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022 a

    2022

  16. [24]

    Deberta: Decoding-enhanced bert with disentangled attention

    He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020

    2006 arXiv

  17. [25]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

    He, P., Gao, J., and Chen, W. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021

    2021 arXiv

  18. [26]

    Diffusionbert: Improving generative masked language models with diffusion models

    He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. Diffusionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029, 2022 b

    2022 arXiv

  19. [27]

    Long short-term memory

    Hochreiter, S. Long short-term memory. Neural Computation MIT-Press, 1997

    1997

  20. [28]

    A decoding algorithm for length-control summarization based on directed acyclic transformers

    Huang, C., Zhou, H., Jen, C., Zheng, K., Zaiane, O., and Mou, L. A decoding algorithm for length-control summarization based on directed acyclic transformers. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNL...

    2024 doi

  21. [29]

    Directed acyclic transformer for non-autoregressive machine translation

    Huang, F., Zhou, H., Liu, Y., Li, H., and Huang, M. Directed acyclic transformer for non-autoregressive machine translation. In International Conference on Machine Learning, pp.\ 9410--9428. PMLR, 2022

    2022

  22. [30]

    Directed acyclic transformer pre-training for high-quality non-autoregressive text generation

    Huang, F., Ke, P., and Huang, M. Directed acyclic transformer pre-training for high-quality non-autoregressive text generation. Transactions of the Association for Computational Linguistics, 2023

    2023

  23. [31]

    Scaling sentence embeddings with large language models

    Jiang, T., Huang, S., Luan, Z., Wang, D., and Zhuang, F. Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645, 2023

    2023 arXiv

  24. [32]

    S., Zettlemoyer, L., and Levy, O

    Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8: 0 64--77, 2020

    2020

  25. [33]

    Rethinking positional encoding in language pre-training

    Ke, G., He, D., and Liu, T.-Y. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595, 2020

    2006 arXiv

  26. [34]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  27. [35]

    Albert: A lite bert for self-supervised learning of language representations

    Lan, Z. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019

    1909 arXiv

  28. [36]

    and Li, J

    Li, X. and Li, J. Bellm: Backward dependency enhanced large language model for sentence embeddings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 7...

    2024

  29. [37]

    Mask more and mask later: Efficient pre-training of masked language models by disentangling the [mask] token

    Liao, B., Thulke, D., Hewavitharana, S., Ney, H., and Monz, C. Mask more and mask later: Efficient pre-training of masked language models by disentangling the [mask] token. arXiv preprint arXiv:2211.04898, 2022

    2022 arXiv

  30. [38]

    Language models of protein sequences at the scale of evolution enable accurate structure prediction

    Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022: 0 500902, 2022

    2022

  31. [39]

    Evolutionary-scale prediction of atomic-level protein structure with a language model

    Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379 0 (6637): 0 1123--1130, 2023

    2023

  32. [40]

    F., and Liang, Y

    Liu, S., Demirel, M. F., and Liang, Y. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. Advances in neural information processing systems, 32, 2019

    2019

  33. [41]

    Pre-training molecular graph representation with 3d geometry

    Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728, 2021

    2021 arXiv

  34. [42]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019

    1907 arXiv

  35. [43]

    Pretraining text encoders with adversarial mixture of training signal generators

    Meng, Y., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P., Han, J., and Song, X. Pretraining text encoders with adversarial mixture of training signal generators. arXiv preprint arXiv:2204.03243, 2022

    2022 arXiv

  36. [44]

    Representation deficiency in masked language modeling

    Meng, Y., Krishnan, J., Wang, S., Wang, Q., Mao, Y., Fang, H., Ghazvininejad, M., Han, J., and Zettlemoyer, L. Representation deficiency in masked language modeling. arXiv preprint arXiv:2302.02060, 2023

    2023 arXiv

  37. [45]

    Warped language models for noise robust language understanding

    Namazifar, M., Tur, G., and Hakkani-T "u r, D. Warped language models for noise robust language understanding. In 2021 IEEE spoken language technology workshop (SLT), pp.\ 981--988. IEEE, 2021

    2021

  38. [46]

    Large language model for molecular chemistry

    Pan, J. Large language model for molecular chemistry. Nature Computational Science, 3 0 (1): 0 5--5, 2023

    2023

  39. [47]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019

    1912 arXiv

  40. [48]

    E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L

    Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computati...

    2018 doi

  41. [49]

    Know what you don’t know: Unanswerable questions for SQuAD

    Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for SQuAD . In ACL, 2018

    2018

  42. [50]

    Representation learning with large language models for recommendation

    Ren, X., Wei, W., Xia, L., Su, L., Cheng, S., Wang, J., Yin, D., and Huang, C. Representation learning with large language models for recommendation. In Proceedings of the ACM on Web Conference 2024, pp.\ 3464--3475, 2024

    2024

  43. [51]

    Self-supervised graph transformer on large-scale molecular data

    Rong, Y., Bian, Y., Xu, T., Xie, W., Wei, Y., Huang, W., and Huang, J. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33: 0 12559--12571, 2020

    2020

  44. [52]

    Large-scale chemical language representations capture molecular structure and properties

    Ross, J., Belgodere, B., Chenthamarakshan, V., Padhi, I., Mroueh, Y., and Das, P. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4 0 (12): 0 1256--1264, 2022

    2022

  45. [53]

    Non-autoregressive machine translation with latent alignments

    Saharia, C., Chan, W., Saxena, S., and Norouzi, M. Non-autoregressive machine translation with latent alignments. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 1098--110...

    2020 doi

  46. [54]

    found in translation

    Schwaller, P., Gaudin, T., Lanyi, D., Bekas, C., and Laino, T. “found in translation”: predicting outcomes of complex organic chemistry reactions using neural sequence-to-sequence models. Chemical science, 9 0 (28): 0 6091--6098, 2018

    2018

  47. [55]

    First Quora dataset release: Question pairs, 2017

    Shankar, I., Nikhil, D., and Kornél, C. First Quora dataset release: Question pairs, 2017. URL https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs

    2017

  48. [56]

    Viterbi decoding of directed acyclic transformer for non-autoregressive machine translation

    Shao, C., Ma, Z., and Feng, Y. Viterbi decoding of directed acyclic transformer for non-autoregressive machine translation. In Findings of EMNLP 2022, 2022

    2022

  49. [57]

    Fast and accurate deep bidirectional language representations for unsupervised learning

    Shin, J., Lee, Y., Yoon, S., and Jung, K. Fast and accurate deep bidirectional language representations for unsupervised learning. arXiv preprint arXiv:2004.08097, 2020

    2004 arXiv

  50. [58]

    D., Ng, A

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013

    2013

  51. [59]

    M., Kotha, S., Fried, D., Neubig, G., and Raghunathan, A

    Springer, J. M., Kotha, S., Fried, D., Neubig, G., and Raghunathan, A. Repetition improves language model embeddings. arXiv preprint arXiv:2402.15449, 2024

    2024 arXiv

  52. [60]

    a rk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., G \

    St \"a rk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., G \"u nnemann, S., and Li \`o , P. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pp.\ 20479--20502. PMLR, 2022

    2022

  53. [61]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  54. [62]

    Saprot: Protein language modeling with structure-aware vocabulary

    Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp.\ 2023--10, 2023

    2023

  55. [63]

    V., Xiao, L., Chopra, A., Chaffin, M

    Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. Transfer learning enables predictions in network biology. Nature, 618 0 (7965): 0 616--624, 2023

    2023

  56. [64]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Tong, Z., Song, Y., Wang, J., and Wang, L. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 0 10078--10093, 2022

    2022

  57. [65]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  58. [66]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP, 2018

    2018

  59. [67]

    Improving text embeddings with large language models

    Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023

    2023 arXiv

  60. [68]

    Bevt: Bert pretraining of video transformers

    Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., and Yuan, L. Bevt: Bert pretraining of video transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 14733--14743, 2022

    2022

  61. [69]

    Smiles-bert: large scale unsupervised pre-training for molecular property prediction

    Wang, S., Guo, Y., Wang, Y., Sun, H., and Huang, J. Smiles-bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pp.\ 429--436, 2019

    2019

  62. [70]

    Diffusion language models are versatile protein learners

    Wang, X., Zheng, Z., Ye, F., Xue, D., Huang, S., and Gu, Q. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402.18567, 2024 a

    2024 arXiv

  63. [71]

    Dplm-2: A multimodal diffusion protein language model

    Wang, X., Zheng, Z., Ye, F., Xue, D., Huang, S., and Gu, Q. Dplm-2: A multimodal diffusion protein language model. arXiv preprint arXiv:2410.13782, 2024 b

    2024 arXiv

  64. [72]

    Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. In TACL, 2019

    2019

  65. [73]

    Smiles, a chemical language and information system

    Weininger, D. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28 0 (1): 0 31--36, 1988

    1988

  66. [74]

    Should you mask 15\ arXiv preprint arXiv:2202.08005, 2022

    Wettig, A., Gao, T., Zhong, Z., and Chen, D. Should you mask 15\ arXiv preprint arXiv:2202.08005, 2022

    2022 arXiv

  67. [75]

    A broad-coverage challenge corpus for sentence understanding through inference

    Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2018

    2018

  68. [76]

    N., Gomes, J., Geniesse, C., Pappu, A

    Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9: 0 513--530, 2018

    2018

  69. [77]

    Xia, J., Zhao, C., Hu, B., Gao, Z., Tan, C., Liu, Y., Li, S., and Li, S. Z. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023

    2023

  70. [78]

    Simmim: A simple framework for masked image modeling

    Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu, H. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9653--9663, 2022

    2022

  71. [79]

    Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism

    Xiong, Z., Wang, D., Liu, X., Zhong, F., Wan, X., Li, X., Li, Z., Luo, X., Chen, K., Jiang, H., et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63 0 (16): 0 8749--8760, 2019

    2019

  72. [80]

    scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data

    Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., and Yao, J. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence, 4 0 (10): 0 852--866, 2022

    2022

  73. [81]

    Mol-ae: Auto-encoder based molecular representation learning with 3d cloze test objective

    Yang, J., Zheng, K., Long, S., Nie, Z., Zhang, M., Dai, X., Ma, W.-Y., and Zhou, H. Mol-ae: Auto-encoder based molecular representation learning with 3d cloze test objective. bioRxiv, pp.\ 2024--04, 2024

    2024

  74. [82]

    Analyzing learned molecular representations for property prediction

    Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59 0 (8): 0 3370--3388, 2019

    2019

  75. [83]

    Spelling error correction with soft-masked bert

    Zhang, S., Huang, H., Liu, J., and Li, H. Spelling error correction with soft-masked bert. arXiv preprint arXiv:2005.07421, 2020

    2005 arXiv

  76. [84]

    Towards a unified training for levenshtein transformer

    Zheng, K., Wang, L., Wang, Z., Chen, B., Zhang, M., and Tu, Z. Towards a unified training for levenshtein transformer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  77. [85]

    Smi-editor: Edit-based smiles language model with fragment-level supervision

    Zheng, K., Liang, S., Yang, J., Feng, B., Liu, Z., Ju, W., Xiao, Z., and Zhang, M. Smi-editor: Edit-based smiles language model with fragment-level supervision. arXiv preprint arXiv:2412.05569, 2024 a

    2024 arXiv

  78. [86]

    Esm all-atom: Multi-scale protein language model for unified molecular modeling

    Zheng, K., Long, S., Lu, T., Yang, J., Dai, X., Zhang, M., Nie, Z., Ma, W.-Y., and Zhou, H. Esm all-atom: Multi-scale protein language model for unified molecular modeling. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  79. [87]

    Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert

    Zhong, Q., Ding, L., Liu, J., Du, B., and Tao, D. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198, 2023 a

    2023 arXiv

  80. [88]

    Revisiting token dropping strategy in efficient bert pretraining

    Zhong, Q., Ding, L., Liu, J., Liu, X., Zhang, M., Du, B., and Tao, D. Revisiting token dropping strategy in efficient bert pretraining. arXiv preprint arXiv:2305.15273, 2023 b

    2023 arXiv

  81. [89]

    Uni-mol: A universal 3d molecular representation learning framework

    Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., Zhang, L., and Ke, G. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6K2RM6wVqKu

    2023

  82. [90]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.