BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context

Alexis Matzopoulos; Charl Hendriks; Francois Meyer; Hishaam Mahomed

REVIEW 3 major objections 5 minor 1 cited by

BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BabyLMs trained on 13 million isiXhosa words beat a standard model on POS tagging and NER.

desk verdict Useful new benchmark numbers for isiXhosa with BabyLM architectures, but the headline claim is weakened by a real hyperparameter confound between the BabyLMs and the RoBERTa baseline. read the letter →

arxiv 2501.03855 v1 pith:XGKXILTH submitted 2025-01-07 cs.CL

Alexis Matzopoulos , Charl Hendriks , Hishaam Mahomed , Francois Meyer This is my paper · ORCID

classification cs.CL

keywords BabyLMdata-efficientlanguagemodellinglow-resourcelanguagesisiXhosaELC-BERTmaskedlatentsemanticmodelingpart-of-speechtaggingnamedentityrecognition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that sample-efficient 'BabyLM' architectures, developed for a fixed English corpus, transfer to a genuinely low-resource language. Pretraining ELC-BERT and MLSM on a 13-million-word isiXhosa corpus yields better part-of-speech tagging and named-entity recognition than a vanilla RoBERTa model trained on the same data, and in some cases beats the large multilingual XLM-R model. The authors care because low-resource languages typically have far less text than even the BabyLM constraint, so architectures that learn well from small data could provide real gains. They also argue the remaining gap to the best multilingual models is caused by the absence of curated, developmentally plausible pretraining data for isiXhosa.

What carries the argument

The argument rides on two architectural mechanisms. ELC-BERT replaces standard residual connections with learned, layer-specific weights, so each layer combines previous layers' outputs in a weighted rather than equal way, which the authors link to stronger word-level representations and faster, more sample-efficient pretraining. MLSM replaces token-identity prediction with masked latent semantic modeling: a teacher model produces sparse semantic categories for masked tokens, and a student model is trained to predict those categories, learning coarser but more robust semantic structure that helps named-entity recognition. Both models are pretrained on the 13m-word WURA isiXhosa corpus, and evaluated on MasakhaPOS, MasakhaNER, and MasakhaNEWS.

What would settle it

Retrain the RoBERTa baseline with the BabyLM hyperparameters (learning rate 5e-4, batch size 128, sequence length 128) on the same WURA corpus and compare POS and NER scores; if the gap narrows to near zero, the architecture-specific claim is unsupported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that two BabyLM architectures pretrained from scratch on the WURA isiXhosa corpus outperform a RoBERTa baseline on the word-level tasks of POS tagging and NER: ELC-BERT reaches 87.7 POS accuracy and 88.6 NER F1, against RoBERTa's 87.0 and 85.4, while MLSM reaches 87.4 and 87.0. ELC-BERT also beats the XLM-R skyline on both of these tasks, and the authors attribute the gains to each architecture's specific mechanism for learning with limited data. The paper is careful to note that these models do not beat the strongest multilingual skylines on all tasks, and that the limiting factor appears to be pretraining-data quality rather than the architectures themselves.

Load-bearing premise

The paper assumes the performance difference between the BabyLMs and the RoBERTa baseline is caused by architecture, but the models also differ in learning rate, batch size, and sequence length, so a confound is untested.

Editorial extensions

If this is right

Data-efficient architectures can narrow the gap for low-resource languages without large pretraining corpora.
ELC-BERT's learned layer weighting appears to be the more transferable of the two mechanisms, matching its English BabyLM success.
Named-entity recognition benefits more than topic classification from these architectures, suggesting word-level tasks are the sweet spot.
Pretraining-data quality, not just architecture, determines how far these gains go; curated corpora remain a bottleneck.
Cross-lingual transfer models like XLM-R can be beaten on some tasks by smaller models trained from scratch on one language.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Our inference: since only isiXhosa is tested, transfer to other low-resource languages is plausible but unproven.
Our inference: because the baseline and BabyLMs differ in learning rate, batch size, and sequence length, the true size of the architectural advantage is uncertain; a matched-hyperparameter comparison would sharpen the claim.
Our inference: the layer-weighting analysis suggests ELC-BERT's emphasis on the embedding layer is tied to syntactic information, hinting that the architecture could combine well with richer morphological pretraining for Bantu languages.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper for what it is: a clean, well-written empirical study that pretrains ELC-BERT and MLSM from scratch on 13m words of isiXhosa and evaluates them on three Masakha tasks. The numbers are new, the data and benchmarks are public, and the authors report standard deviations over five finetuning runs. That is a solid contribution to low-resource NLP, especially for under-resourced African languages.

The main soft spot is exactly where the stress-test points. The comparison in Table 1 is not architecture-controlled. Appendix A shows RoBERTa with LR 5e-5, sequence length 512, batch size 8, while ELC-BERT runs at LR 5e-4, sequence length 128, batch size 128, and MLSM at LR 1e-4, sequence length 128, batch size 64. That is a 10x learning-rate difference and a 4x difference in tokens per batch. The claimed +3.2 NER F1 for ELC-BERT over RoBERTa could easily come from those settings rather than from weighted residual connections or semantic prediction. The paper never tests this, and the abstract overstates the result by attributing the gains to data-efficient architectures. The POS gain (0.7 points) is within one standard deviation, and the NTC result reverses direction. So the evidence for an architecture effect is thinner than the framing suggests.

To the paper's credit, the authors are transparent about the hyperparameters and include honest limitations. They note the single language, the limited task set, and the data-quality gap. The qualitative analyses of layer weights and semantic categories are suggestive, not conclusive. None of this is fatal; the result is plausible and worth checking. But as reported, the paper is more accurately a comparison of training configurations than of architectures.

I would send this to peer review, but conditionally. The reviewer should ask for a matched-configuration rerun: same learning rate, batch size, and sequence length across models, or at least a systematic ablation. If that is not feasible, the conclusion should be reframed as "BabyLM recipes transfer to isiXhosa" rather than "BabyLM architectures beat vanilla pretraining." With that change, the paper would be a useful empirical data point for people working on low-resource LMs and for the BabyLM community.

Referee Report

3 major / 5 minor

Summary. The paper pretrains two BabyLM architectures (ELC-BERT and MLSM) on a 13-million-word isiXhosa corpus (WURA) and evaluates them on MasakhaPOS, MasakhaNER, and MasakhaNEWS, comparing against a RoBERTa baseline pretrained from scratch on the same corpus and against three large multilingual PLMs. The authors report that both BabyLMs outperform the RoBERTa baseline on POS tagging and NER (with ELC-BERT gaining +3.2 F1 on NER), that ELC-BERT in some tasks beats XLM-R, and that neither BabyLM outperforms RoBERTa on news topic classification. They attribute the gains to the architectures' sample-efficient design and present visual analyses of ELC-BERT layer contributions and MLSM semantic categories.

Significance. If the central claim—that BabyLM architectures provide data-efficiency gains for a genuinely low-resource language—is supported, the paper would be a useful contribution to low-resource NLP, showing that architectural innovations from the English BabyLM challenge transfer to a very different language and data regime. The paper has notable strengths: it uses public pretraining corpora and public benchmarks, reports means and standard deviations over five finetuning runs, and includes a limitations section that is candid about the single-language scope. However, the empirical comparison that supposedly isolates the architecture effect is confounded with major hyperparameter differences, so the headline conclusion is currently not established.

major comments (3)

[Appendix A, Table 2] The central comparison in Table 1 is not architecture-controlled. Appendix A, Table 2 shows that RoBERTa is pretrained with learning rate 5e-5, sequence length 512, and batch size 8, while ELC-BERT uses 5e-4, 128, and 128 and MLSM uses 1e-4, 128, and 64. These settings produce a 4x difference in tokens per batch between RoBERTa (4096) and ELC-BERT (16384), a 2x difference for MLSM (8192), and a 10x learning-rate difference. The abstract and conclusion attribute the performance differences to the BabyLM architectures, but a comparison that varies architecture, learning rate, batch size, and sequence length simultaneously does not support that attribution. A matched-configuration rerun (RoBERTa trained with the same LR/batch/sequence-length settings as the BabyLMs, or BabyLMs trained with RoBERTa's settings) or a series of ablations is needed before the data-efficiency claim can be accepted.
[Section 4, Table 1] The claimed gains are not uniform across tasks. For POS tagging, ELC-BERT is 87.7±0.5 versus RoBERTa 87.0±0.1 and MLSM 87.4±0.1; the ELC-BERT gain of 0.7 points is within one standard deviation of its own estimate, and the MLSM gain is 0.4 points. For NTC, both BabyLMs are below RoBERTa by 2.2–2.6 F1. Only the NER gains (ELC-BERT +3.2, MLSM +1.6) are clearly outside the reported noise. The sentence in Section 4 that 'both BabyLMs outperform the baseline on POS and NER' therefore overstates the evidence, and the paper should either report significance tests/effect sizes or temper the claim accordingly.
[Section 5, Figures 2 and 3] The mechanism analyses are suggestive but do not control for the confounds above, and they are based on a small number of hand-picked examples. Figure 3 shows semantic overlap for four named entities (two persons, two locations), and Appendix B extends this to four POS-tagged words; no quantitative measure of overlap is given for the full evaluation set, and there is no comparison with a RoBERTa-based semantic distribution. The claim that 'this pattern indicates that MLSM effectively encodes the semantic properties of these named entities, to which we attribute its NER performance gains' is therefore a post-hoc interpretation rather than an established causal link. The wording should be softened, or the analysis expanded to a systematic evaluation.

minor comments (5)

[Section 4] There is a typo in 'computate requirements' in the paragraph beginning 'ELC-BERT demonstrates superior efficiency'; it should read 'compute requirements'.
[Table 1 caption] The caption says 'We boldface best per-category performance and underline best overall,' but the table as rendered does not show underlining, making 'best overall' ambiguous. Please clarify how 'best overall' is defined and ensure the formatting matches the description.
[Figure 1] Figure 1 plots downstream performance against pretraining epochs but does not show the standard deviations reported in Table 1; adding error bars or bands would help readers assess whether the early-training differences are meaningful.
[Appendix A] The paper says the training scripts from the BabyLM submissions were used, but no URL or repository link is given. Providing the exact scripts and configurations would improve reproducibility.
[Section 2.2.2] The 'barbecue' example is English-only; adding an isiXhosa example would make the description of MLSM more directly connected to the paper's language of study.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's empirical claims are self-contained against external corpora and benchmarks.

full rationale

The paper's central claims are empirical comparisons of pretraining architectures on a low-resource language. Pretraining uses the public WURA isiXhosa corpus, evaluation uses public MasakhaPOS, MasakhaNER, and MasakhaNEWS benchmarks, and the BabyLM architectures (ELC-BERT and MLSM) are taken from independently published prior work cited by the authors. No quantity is defined in terms of another, no fitted parameter is renamed as a prediction, and no load-bearing conclusion rests on a self-citation chain. The only self-citation is Meyer et al. (2024), Nguni-XLMR, used as an external skyline for comparison rather than as justification for the paper's own methods or results. The semantic-category-overlap analysis in Section 5 is observational and does not reduce to any fitted value. The hyperparameter mismatch between the BabyLM models and the RoBERTa baseline (Appendix A, Table 2) is a validity concern about the architecture-effect interpretation, but it is not a form of circular reasoning. Accordingly, no circular steps are identified and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new entities are introduced. The central claim rests on public corpora and benchmarks, plus hand-modified hyperparameters for ELC-BERT. The free parameters listed are training configuration choices, not fitted scientific constants, but they affect the comparison and are not controlled across models.

free parameters (4)

ELC-BERT learning rate = 5e-4
Changed from the original BabyLM submission's 1e-2 because training loss was unstable; chosen after experimentation (Appendix A).
ELC-BERT batch size = 128
Reduced from 256 due to computational constraints (Appendix A).
ELC-BERT pretraining epochs = 200
Reduced from the submission's 2000 epochs; authors state downstream performance plateaus by 200 epochs (Appendix A).
MLSM teacher/student hyperparameters = LR 1e-4, SL 128, H 12, BS 64
Kept from the MLSM submission; not tuned here, but still a modelling choice inherited from prior work.

assumptions (3)

domain assumption WURA corpus is a sufficiently clean and representative isiXhosa pretraining corpus.
Section 3 relies on WURA (Oladipo et al. 2023) without independent quality checks; the paper's discussion of data quality depends on this assumption.
domain assumption BabyLM architectures transfer to non-English languages without adaptation.
The paper assumes ELC-BERT and MLSM, designed for English, will work on isiXhosa with only hyperparameter changes.
domain assumption MasakhaPOS, MasakhaNER, and MasakhaNEWS provide reliable evaluation signals.
Used as ground truth; annotation noise would affect all models equally, but absolute numbers depend on dataset quality.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context." pith.science (2026). https://pith.science/paper/XGKXILTH

@misc{pith2026250103855,
  author       = {Pith},
  title        = {Pith review of: BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGKXILTH}},
  note         = {Machine review of arXiv:2501.03855}
}

read the original abstract

The BabyLM challenge called on participants to develop sample-efficient language models. Submissions were pretrained on a fixed English corpus, limited to the amount of words children are exposed to in development (<100m). The challenge produced new architectures for data-efficient language modelling, which outperformed models trained on trillions of words. This is promising for low-resource languages, where available corpora are limited to much less than 100m words. In this paper, we explore the potential of BabyLMs for low-resource languages, using the isiXhosa language as a case study. We pretrain two BabyLM architectures, ELC-BERT and MLSM, on an isiXhosa corpus. They outperform a vanilla pretrained model on POS tagging and NER, achieving notable gains (+3.2 F1) for the latter. In some instances, the BabyLMs even outperform XLM-R. Our findings show that data-efficient models are viable for low-resource languages, but highlight the continued importance, and lack of, high-quality pretraining data. Finally, we visually analyse how BabyLM architectures encode isiXhosa.

Figures

Figures reproduced from arXiv: 2501.03855 by the authors.

**Figure 2.** Layer contribution heatmaps of isiXhosa ELC-BERT at different stages of pretraining. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Top 10 semantic categories predicted by isiXhosa MLSM for named entities (sampled from MasakhaNER). [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Layer contribution heatmaps of isiXhosa ELC-BERT at different stages of pretraining. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Top 10 semantic categories predicted by isiXhosa MLSM for target words (sampled from MasakhaPOS). [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Sample-Efficient Language Model for Hinglish Conversational AI
cs.CL 2025-04 reject novelty 4.0 of 10

Fine-tuning Qwen2.5-3B and 7B with LoRA on synthetic Hinglish dialogues raises human-rated fluency and coherence, though the benchmark is small and synthetic.

Reference graph

Works this paper leans on

24 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F

David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, ...

2022
[4]

David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunn...

2023
[5]

Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow

Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. https://aclanthology.org/2022.coling-1.382 Adapting pre-trained language models to A frican languages via multilingual adaptive fine-tuning . In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336--4349, Gyeongju, Republic of Korea. ...

2022
[6]

G \'a bor Berend. 2023 a . https://doi.org/10.18653/v1/2023.conll-babylm.26 Better together: Jointly using masked latent semantic modeling and masked language modeling for sample efficient pre-training . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 298--307, Singapore. Association for Comp...

work page doi:10.18653/v1/2023.conll-babylm.26 2023
[7]

G \'a bor Berend. 2023 b . https://doi.org/10.18653/v1/2023.findings-acl.876 Masked latent semantic modeling: an efficient pre-training alternative to masked language modeling . In Findings of the Association for Computational Linguistics: ACL 2023, pages 13949--13962, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.findings-acl.876 2023
[8]

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

doi:10.18653/v1/2020.acl-main.747 2020

Show all 24 references

[9]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

2019 doi
[10]

Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. ...

2023
[11]

Bonaventure F. P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, and Chris Emezue. 2022. https://doi.org/10.18653/v1/2022.sustainlp-1.11 A fro LM : A self-active learning-based multilingual pretrained l...

2022 doi
[12]

Eberhard, Gary F

David M. Eberhard, Gary F. Simons, , and Charles D. Fenning. 2019. Ethnologue: Languages of the World, 22 edition. SIL International

2019
[13]

Lucas Georges Gabriel Charpentier and David Samuel. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.20 Not all layers are equally as important: Every layer counts BERT . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learnin...

2023 doi
[14]

Richards, Steven F

Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Judith K. Montgomery, Charles R. Greenwood, D. Kimbrough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. https://doi.org/10.1044/2016\_AJSLP-15-0169 Mapping the early language environment using all-day recordings and...

2017 doi
[15]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

2019 arXiv
[16]

Francois Meyer, Haiyue Song, Abhisek Chakrabarty, Jan Buys, Raj Dabre, and Hideki Tanaka. 2024. https://aclanthology.org/2024.lrec-main.1071 NGLUE ni: Benchmarking and adapting pretrained language models for nguni languages . In Proceedings of the 2024 Joint International Conf...

2024
[17]

Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. https://doi.org/10.18653/v1/2021.mrl-1.11 Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages . In Proceedings of the 1st Workshop on Multilingual Representation ...

2021 doi
[18]

Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Owodunni, Odunayo Ogundepo, David Adelani, and Jimmy Lin. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.11 Better quality pre-training data and t5 models for A frican languages . In Proceedings of the 2023 Co...

2023 doi
[19]

David Samuel, Andrey Kutuzov, Lilja vrelid, and Erik Velldal. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.146 Trained on 100 million words and still in shape: BERT meets B ritish N ational C orpus . In Findings of the Association for Computational Linguistics: EACL 20...

2023 doi
[20]

Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. https://doi.org/10.18653/v1/P19-1452 BERT rediscovers the classical NLP pipeline . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593--4601, Florence, Italy. Association for ...

2019 doi
[21]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

2023 arXiv
[22]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

2023 arXiv
[23]

Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.1 Findings of the B aby LM challenge: Sample-efficien...

2023 doi
[24]

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

2021 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F

David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, ...

2022

[4] [4]

David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunn...

2023

[5] [5]

Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow

Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. https://aclanthology.org/2022.coling-1.382 Adapting pre-trained language models to A frican languages via multilingual adaptive fine-tuning . In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336--4349, Gyeongju, Republic of Korea. ...

2022

[6] [6]

G \'a bor Berend. 2023 a . https://doi.org/10.18653/v1/2023.conll-babylm.26 Better together: Jointly using masked latent semantic modeling and masked language modeling for sample efficient pre-training . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 298--307, Singapore. Association for Comp...

work page doi:10.18653/v1/2023.conll-babylm.26 2023

[7] [7]

G \'a bor Berend. 2023 b . https://doi.org/10.18653/v1/2023.findings-acl.876 Masked latent semantic modeling: an efficient pre-training alternative to masked language modeling . In Findings of the Association for Computational Linguistics: ACL 2023, pages 13949--13962, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.findings-acl.876 2023

[8] [8]

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

doi:10.18653/v1/2020.acl-main.747 2020