Pith. sign in

REVIEW 2 major objections 4 minor 36 references

Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For a small Sepedi corpus, the pre-training objective determines which metric improves: masking does better on BLEU, standard next-word training on perplexity.

desk verdict New Sepedi datasets and first generative baselines are worth having, but the central occlusion-vs-non-occlusion comparison is confounded by unmatched model architectures. read the letter →

arxiv 2501.15281 v1 pith:SJQEP3XW submitted 2025-01-25 cs.CL cs.AIcs.LG

Simon P. Ramalepe , Thipe I. Modipa , Marelie H. Davel This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords TransformerTextgenerationPre-trainingOcclusion-basedtrainingSepediLow-resourcedlanguagesPerplexityBLEU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether masking input tokens during pre-training helps a text-generation model when the available corpus is small. It builds two GPT-2-style models for Sepedi, one trained with standard next-word prediction and one that also randomly occludes tokens, then fine-tunes both on radio-news data. On validation loss and perplexity the non-occlusion model comes out ahead, while the occlusion model scores higher BLEU against reference text. The authors take this as evidence that the choice of pre-training objective determines which quality metric improves in a low-resource setting.

What carries the argument

The central mechanism is the occlusion-based pre-training objective layered onto the GPT-2 decoder. In this variant, a hyperparameter-tuned fraction (0.3) of the input tokens is randomly replaced with a mask, and the model must recover those tokens from the surrounding context while still learning to predict the next token left-to-right. The paper compares this against the pure next-word objective, using validation loss, perplexity, and BLEU as the yardsticks, and attributes the divergent behaviour of the models to the bidirectional context that masking introduces.

What would settle it

Retrain SepGPT and SepGPT-OCC with identical architecture (same number of layers, attention heads, and all other hyperparameters) and compare validation loss, perplexity, and BLEU on the same SepNews-2 test set. If the gaps vanish or reverse, the original conclusion about objectives is not supported; if they persist, the small-scale finding is confirmed.

Watch

Extended reading notes

Core claim

On the SepMono corpus (432,970 sentences, roughly 11.4 million tokens), the authors pre-train SepGPT with the standard autoregressive GPT-2 objective and SepGPT-OCC with the same objective plus random token occlusion at probability 0.3. The non-occlusion model reaches a validation perplexity of 16.04 versus 31.89 for the occlusion model, with validation losses of 2.78 and 3.46. After fine-tuning on SepNews-1 and testing on a held-out time period (SepNews-2), the perplexity gap narrows to 14.87 versus 16.48, but the occlusion model produces the higher BLEU score: 48.84% versus 44.98% for the fine-tuned models, and 29.48% versus 24.19% before fine-tuning. The authors interpret this as showing that occlusion-based pre-training makes the generated text more similar to reference text, even though it yields a less confident probability model by perplexity.

Load-bearing premise

The two models differ not only in pre-training objective but also in size: the non-occlusion model has 8 transformer layers and 8 attention heads, the occlusion model 6 layers and 4 heads, so the performance gaps could be caused by capacity rather than by the objective.

Editorial extensions

If this is right

  • For a small Sepedi corpus, occlusion-based pre-training yields higher BLEU scores than standard next-word pre-training, both before and after fine-tuning.
  • Non-occlusion pre-training gives lower validation loss and perplexity, which the authors describe as a more reliable probability model.
  • Fine-tuning reduces the validation-perplexity gap between the two approaches from a factor of about two to a small margin.
  • The newly curated SepMono and SepNews datasets provide baselines and resources for future Sepedi and low-resource language modeling.
  • The reported BLEU scores (44.98% and 48.84% after fine-tuning) set a new benchmark for Sepedi text generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the architecture mismatch in Table 4 is corrected, the BLEU advantage of occlusion might shrink or disappear, so the headline claim should be re-tested with matched model sizes.
  • The opposite ordering of perplexity and BLEU suggests that evaluating generation quality by perplexity alone can be misleading for low-resource masked pre-training.
  • The same two-objective comparison could be run on other low-resourced languages to see whether the pattern generalizes beyond Sepedi.
  • The authors' qualitative note that generated sentences are 'mostly grammatically correct' should be verified with human evaluation before being relied upon.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper curates two new Sepedi text datasets (SepMono, a mixed monolingual corpus, and SepNews, a radio-news corpus split temporally into SepNews-1 and SepNews-2), pre-trains two GPT-2-style transformer models on SepMono using a standard next-token objective and an occlusion/masking-augmented objective, and fine-tunes both on SepNews-1. Evaluation is reported via validation/test loss, perplexity, and BLEU scores against reference texts. The abstract claims that non-occlusion models achieve better validation loss and perplexity, while occlusion-based models achieve slightly higher BLEU scores on generated text. The paper also compares these models with several other low-resource transformer baselines and presents example generated text.

Significance. If the comparison were valid, the paper would offer a useful empirical data point for pre-training objectives on very small corpora in a low-resource language: standard next-token prediction giving lower perplexity while masking-based pre-training giving higher BLEU overlap. The datasets themselves, the temporal split of the news corpus into fine-tuning and test partitions, and the reproducible fine-tuning protocol (gradual unfreezing, early stopping) are contributions that could support future Sepedi NLP work. However, the central comparative claim is currently not supported because the two models differ in architecture as well as objective; the paper itself does not acknowledge or control for this confound.

major comments (2)
  1. [§3.2 'Model training' and Table 4] The central comparison is confounded. The non-occlusion SepGPT model uses 8 transformer layers and 8 attention heads, while the occlusion-based SepGPT-OCC model uses 6 layers and 4 heads (Table 4). Because the hyperparameters were optimized independently for each model, the reported differences in validation loss (2.78 vs 3.46) and validation perplexity (16.04 vs 31.89) cannot be attributed to the pre-training objective rather than to model capacity. The same confound invalidates the BLEU comparison. To support the abstract's claim, the authors need to either compare models that are matched in depth, width, and parameter count, or include an explicit ablation showing that the capacity difference does not drive the observed pattern.
  2. [§3.3 'Evaluation metrics' and §4 'Results'] The BLEU evaluation is described without a generation protocol. The paper does not state the decoding method (greedy, sampling, beam search), temperature, prompt set, number of generated texts, or how generated texts are aligned with reference texts. Consequently, the reported BLEU differences ('5% higher', '3.86% higher') are not reproducible or interpretable. The authors should specify the full generation setup and report variance or confidence intervals, ideally over multiple seeds or prompts.
minor comments (4)
  1. [Table 2] The table contains formatting and typographical errors: 'T rainingV alidationT esting' should be a single header, and the unique-token total '125,04' appears to be missing a digit; please correct these.
  2. [§4 'Results'] The sentence 'We further generated text from the trained models and computed a BLEU score' would benefit from specifying which of the four models were used for generation and whether the BLEU score is computed on pre-trained, fine-tuned, or both model variants; as written, the following percentages are ambiguous.
  3. [§4 'Results'] The phrase 'the the BLEU score' contains a duplicated article; please proofread.
  4. [§2.2 'Closely related studies'] The comparison with Sudanese GPT-2, RoGPT-2, and BanglaGPT in Table 3 is acknowledged by the authors as not directly comparable because datasets and languages differ, but the conclusion that 'our models obtained higher and comparable results' relies on this table; I suggest softening that conclusion or moving the external comparison to a clearly labeled reference row.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical comparisons on held-out validation and temporally disjoint test data, with no fitted parameter fed back into a derivation and no load-bearing self-citation chain.

full rationale

The paper's central claims are empirical: the non-occlusion model achieves lower validation loss and perplexity, while the occlusion-based model achieves a slightly higher BLEU score on generated text. These outcomes are measured on held-out validation splits, a temporally disjoint test set (SepNews-2), and external comparison baselines (Sudanese GPT-2/BERT/RoBERTa, BanglaGPT, RoGPT-2). No equation in the paper defines one result in terms of another result; perplexity and BLEU are standard external metrics applied to model outputs, not quantities derived from the training objective by construction. The only self-citation is reference [26], used to support a background linguistic claim about Sepedi's disjunctive writing system; this is not load-bearing for the pre-training comparison. The comparison is admittedly confounded because the two models differ in architecture (8 layers and 8 heads vs 6 layers and 4 heads, Table 4), so the attribution of the metric gap to the pre-training objective is a validity concern, not a circularity concern. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled via citation. The derivation chain is thus self-contained with respect to circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central comparison rests on the assumptions that the GPT-2 architecture and BPE tokenizer suit Sepedi, that BLEU is a meaningful quality proxy, that the temporal split tests generalization, and that the per-model hyperparameter sweeps allow a fair comparison. These are stated or implicit domain assumptions, not derived facts. No new theoretical entities are introduced.

free parameters (6)
  • occlusion probability = 0.3
    Chosen by validation loss among {0.1, 0.3, 0.5}; this parameter is central to the occlusion model and is not justified independently of the validation data.
  • number of transformer layers = 6 (occlusion), 8 (non-occlusion)
    Tuned by random sweep; the two models differ, so the comparison of pre-training objectives is entangled with model capacity.
  • number of attention heads = 4 (occlusion), 8 (non-occlusion)
    Tuned by random sweep; differs between models and affects model capacity.
  • learning rate = 2e-4
    Tuned via W&B random sweep for both models.
  • dropout = 0.3
    Tuned via random sweep for both models.
  • vocabulary size = 50,225
    Chosen for the BPE tokenizer; affects the embedding dimension and model capacity.
assumptions (5)
  • standard math Language modeling probability factorization follows the chain rule (Eq. 1).
    Used to define perplexity and the training objective.
  • domain assumption GPT-2 and BPE tokenization are suitable for Sepedi.
    Invoked in Sections 2.1 and 3.2; no separate validation that this architecture/tokenizer is optimal for Sepedi.
  • domain assumption BLEU is an acceptable proxy for generated text quality.
    Section 3.3 acknowledges BLEU is mostly for translation and that human evaluation is the standard; still uses BLEU as the main quality signal.
  • domain assumption The temporal split of SepNews-2 is a strong proxy for unseen contexts.
    Section 3.1: the test set from a different time period is assumed to measure generalization.
  • ad hoc to paper Masking tokens during autoregressive pre-training (occlusion) helps the model learn structural representations.
    Section 3.2: this is the motivating hypothesis of the study, not established by prior work in the generative setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset." pith.science (2026). https://pith.science/paper/SJQEP3XW

@misc{pith2026250115281,
  author       = {Pith},
  title        = {Pith review of: Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SJQEP3XW}},
  note         = {Machine review of arXiv:2501.15281}
}
read the original abstract

Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.

Figures

Figures reproduced from arXiv: 2501.15281 by the authors.

Figure 1
Figure 1. Comparison of training curves: (a) Occlusion-based and (b) Non [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Occlusion hyperparameter search In [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 32 canonical work pages

  1. [1]

    UPB Scientific Bulletin84(4) (2022)

    Buzea, M.C., TrÃĎÆŠuÃĹâĎćan-Matu, ÃĹ., Rebedea, T.: Automatic Romanian text generation using GPT-2. UPB Scientific Bulletin84(4) (2022)

    work page 2022

  2. [2]

    Celikyilmaz, A., Clark, E., Gao, J.: Evaluation of Text Generation: A Survey (May 2021)

    work page 2021

  3. [3]

    International Journal of Computer Applications120(10), 33–38 (Jun 2015)

    Chandel, H., Vatta, S.: Occlusion Detection and Handling: A Review. International Journal of Computer Applications120(10), 33–38 (Jun 2015)

    work page 2015

  4. [4]

    In: Interspeech 2019

    Chang, C.T., Chuang, S.P., Lee, H.Y.: Code-Switching Sentence Generation by Generative Adversarial Networks and its Application to Data Augmentation. In: Interspeech 2019. pp. 554–558. International Speech Communication Association (Sep 2019). https://doi.org/10.21437/Interspeech.2019-3214 Table 4: Optimal hyperparameters for each model Hyperparameter Occ...

    work page doi:10.21437/interspeech.2019-3214 2019

  5. [5]

    In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

    Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., GuzmÃČÂąn, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsuper- vised Cross-lingual Representation Learning at Scale. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 8440–...

    work page 2020

  6. [6]

    In: Proceedings of the 2019 Conference of the North

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North. pp. 4171–4186. Association for Computational Lin- guistics, Minneapolis, Minnesota (2019)

    work page 2019

  7. [7]

    In: Interspeech 2019

    Gao, Y., Feng, J., Liu, Y., Hou, L., Pan, X., Ma, Y.: Code-switching sentence generation by Bert and Generative Adversarial Networks. In: Interspeech 2019. pp.3525–3529.InternationalSpeechCommunicationAssociation(Sep2019).https: //doi.org/10.21437/Interspeech.2019-2501

    work page doi:10.21437/interspeech.2019-2501 2019

  8. [8]

    In: Calzolari, N., Choukri, K., Declerck, T., DoÃĎÅÿan, M.U., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., Piperidis, S

    Goldhahn, D., Eckart, T., Quasthoff, U.: Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages. In: Calzolari, N., Choukri, K., Declerck, T., DoÃĎÅÿan, M.U., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Eighth International Confer- ence on Language Resources and Ev...

    work page 2012

Show all 36 references
  1. [9]

    In: Cohn, T.,He,Y.,Liu,Y.(eds.)FindingsoftheAssociationforComputationalLinguistics: EMNLP 2020

    Gupta, D., Ekbal, A., Bhattacharyya, P.: A Semi-supervised Approach to Generate the Code-Mixed Text using Pre-trained Encoder and Transfer Learning. In: Cohn, T.,He,Y.,Liu,Y.(eds.)FindingsoftheAssociationforComputationalLinguistics: EMNLP 2020. pp. 2267–2280. Association for C...

    2020 doi

  2. [10]

    Harbecke, D.: Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling (Jan 2021)

    2021

  3. [11]

    In: Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers)

    Howard, J., Ruder, S.: Universal Language Model Fine-tuning for Text Classifica- tion. In: Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers). pp. 328–339. Melbourne, Australia (2018)

    2018

  4. [12]

    In: Bosselut, A., Chandu, K., Dhole, K., Gangal, V., Gehrmann, S., Jer- nite, Y., Novikova, J., Perez-Beltrachini, L

    Howcroft, D.M., Gkatzia, D.: Most NLG is Low-Resource: here’s what we can do about it. In: Bosselut, A., Chandu, K., Dhole, K., Gangal, V., Gehrmann, S., Jer- nite, Y., Novikova, J., Perez-Beltrachini, L. (eds.) Proceedings of the 2nd Workshop on Natural Language Generation, E...

    2022 doi

  5. [13]

    In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y

    Lee, N., Bang, Y., Madotto, A., Fung, P.: Towards Few-shot Fact-Checking via Perplexity. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y. (eds.) Proceed- ings of the 2021 Conference of the No...

    2021

  6. [14]

    Li, J., Tang, T., Zhao, W.X., Wen, J.R.: Pretrained Language Model for Text Generation: A Survey. vol. 5, pp. 4492–4499 (Aug 2021), iSSN: 1045-0823

    2021

  7. [15]

    Marivate, V., Njini, D., Madodonga, A., Lastrucci, R., Dzingirai, I., Rajab, J.: The Vuk’uzenzele South African multilingual corpus (Feb 2023)

    2023

  8. [16]

    In: Mabuya, R., Ra- mukhadi, P., Setaka, M., Wagner, V., van Zaanen, M

    Marivate, V., Sefara, T., Chabalala, V., Makhaya, K., Mokgonyane, T., Mokoena, R., Modupe, A.: Investigating an Approach for Low Resource Language Dataset Creation, Curation and Classification: Setswana and Sepedi. In: Mabuya, R., Ra- mukhadi, P., Setaka, M., Wagner, V., van Z...

    2020

  9. [17]

    In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V

    Martin, G., Mswahili, M.E., Jeong, Y.S., Woo, J.: SwahBERT: Language Model of Swahili. In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V. (eds.) Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language...

    2022 doi

  10. [18]

    In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

    Martin, L., Muller, B., Ortiz SuÃČÂąrez, P.J., Dupont, Y., Romary, L., de la Clergerie, ÃČ., Seddah, D., Sagot, B.: CamemBERT: a Tasty French Language Model. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Associatio...

    2020 doi

  11. [19]

    McKellar, C.: Autshumato Monolingual Sepedi Corpus (Sep 2022), accepted: 2022- 12-15T06:35:19Z Artwork Medium: Text; UTF8 Interview Medium: Text; UTF8 Publisher: CTexTÃĆÂő (Centre for Text Technology, North-West University)

    2022

  12. [20]

    Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 6843–6850 (Jul 2019)

    Mielke, S.J., Eisner, J.: Spell Once, Summon Anywhere: A Two-Level Open- Vocabulary Language Model. Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 6843–6850 (Jul 2019)

    2019

  13. [21]

    In: 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI)

    Niculescu, M.A., Ruseti, S., Dascalu, M.: RoGPT2: Romanian GPT2 for Text Generation. In: 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI). pp. 1154–1161. IEEE, Washington, DC, USA (Nov 2021). https://doi.org/10.1109/ICTAI52525.2021.00183

    2021

  14. [22]

    In: Gurevych, I., Miyao, Y

    Pratapa, A., Bhat, G., Choudhury, M., Sitaram, S., Dandapat, S., Bali, K.: Lan- guage Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data. In: Gurevych, I., Miyao, Y. (eds.) Proceedings of the 56th Annual Meeting of the Association for Computational Li...

    2018 doi

  15. [23]

    Puttkammer, M., Schlemmer, M., Pienaar, W., Bekker, R.: NCHLT Sepedi text corpora (May 2014)

    2014

  16. [24]

    openAI blog1(8), 9 (Feb 2019)

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners. openAI blog1(8), 9 (Feb 2019)

    2019

  17. [25]

    Journal of Machine Learning Research21(140), 1–67 (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text- to-Text Transformer. Journal of Machine Learning Research21(140), 1–67 (2020)

    2020

  18. [26]

    In: Proceedings of South Africa Telecommu- nication Networks and Applications Conference (SATNAC)

    Ramalepe, S.P., Modipa, T.I., Davel, M.H.: The development of a Sepedi text generation model using transformers. In: Proceedings of South Africa Telecommu- nication Networks and Applications Conference (SATNAC). pp. 51–56. Fancourt, Western Cape, South Africa, (2022)

    2022

  19. [27]

    https://doi.org/10

    Salim, M., Murad, H., Das, D., Ahmed, F.: BanglaGPT: A Generative Pretrained Transformer-Based Model for Bangla Language (Sep 2023). https://doi.org/10. 1109/ICICT4SD59951.2023.10303383

    2023

  20. [28]

    Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence pp

    Samanta, B., Reddy, S., Jagirdar, H., Ganguly, N., Chakrabarti, S.: A deep genera- tive model for code switched Text. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence pp. 5175–5181 (Aug 2019)

    2019

  21. [29]

    In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2

    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural net- works. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. pp. 3104–3112. NIPS’14, MIT Press, Cambridge, MA, USA (Dec 2014)

    2014

  22. [30]

    Tarunesh, I., Kumar, S., Jyothi, P.: From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text (Jul 2021), arXiv:2107.06483 [cs]

    2021 arXiv

  23. [31]

    In: Advances in Neural Information Processing Systems

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, ÃĚ., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)

    2017

  24. [32]

    APSIPA Transactions on Signal and Informa- tion Processing 13(2) (2024)

    Wei, C., Wang, Y.C., Wang, B., Kuo, C.C.J.: An Overview of Language Models: Recent Developments and Outlook. APSIPA Transactions on Signal and Informa- tion Processing 13(2) (2024). https://doi.org/10.1561/116.00000010

    2024 doi

  25. [33]

    In: Calzolari, N., BÃČÂľchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S

    Wenzek, G., Lachaux, M.A., Conneau, A., Chaudhary, V., GuzmÃČÂąn, F., Joulin, A., Grave, E.: CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In: Calzolari, N., BÃČÂľchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Mae...

    2020

  26. [34]

    Winata, G.I.: Multilingual Transfer Learning for Code-Switched Language and Speech Neural Modeling (Apr 2021), arXiv:2104.06268 [cs, eess]

    2021 arXiv

  27. [35]

    Journal of Big Data9(1), 39 (Apr 2022)

    Wongso, W., Lucky, H., Suhartono, D.: Pre-trained transformer-based language models for Sundanese. Journal of Big Data9(1), 39 (Apr 2022). https://doi.org/ 10.1186/s40537-022-00590-7

    2022 doi

  28. [36]

    In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track Proceedings

    Zintgraf, L.M., Cohen, T.S., Adel, T., Welling, M.: Visualizing Deep Neural Net- work Decisions: Prediction Difference Analysis. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track Proceedings. OpenRevi...

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.