Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Non-Contextual BERT or FastText? A Comparative Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Non-contextual BERT embeddings from the first embedding layer outperform FastText on most Marathi classification tasks, at table-lookup cost.

desk verdict Useful Marathi embedding benchmark, but the abstract overstates the case: on the paper's own Table 2, FastText wins as often as BERT and most gaps are within noise. read the letter →

arxiv 2411.17661 v3 pith:ASAWZVEV submitted 2024-11-26 cs.CL cs.LG

Abhay Shanbhag , Suramya Jadhav , Amogh Thakurdesai , Ridhima Sinare , Raviraj Joshi This is my paper · ORCID
classification cs.CLcs.LG MSC 68T5068T07
keywords non-contextualBERTembeddingsFastTextMarathilow-resourceNLPwordclassificationSVDcompressioncontextualvs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether BERT can be used without running the transformer: it takes the token embeddings from BERT's first layer, averages them per sentence, and pits them against FastText vectors on Marathi sentiment, hate speech, and news classification. Its claim is that these non-contextual BERT embeddings generally beat FastText, while full contextual BERT beats both. The reason to care is speed and memory: non-contextual embeddings are a table lookup, so if the claim holds, low-resource pipelines can get BERT-pretrained quality without a GPU forward pass. The paper also finds that compressing BERT's 768 dimensions down to FastText's 300 with SVD erases the advantage, making FastText the better choice among compressed non-contextual options.

What carries the argument

The central object is the non-contextual embedding: the output of BERT's first embedding layer, a lookup table that maps each token ID to a vector before any transformer layer adds context. The paper averages those token vectors to form a sentence vector and feeds it to a multinomial logistic regression. To make the comparison fair, it compresses the 768-dimensional BERT vectors to 300 dimensions using singular value decomposition, matching FastText's dimensionality, and reports 5-fold cross-validated accuracy. This extraction-and-compression setup is what lets the paper isolate embedding quality from model depth.

What would settle it

Compute verbatim and near-duplicate overlap between the MahaSent, MahaHate, and MahaNews examples and the Marathi pretraining corpus described in Joshi (2022a). If a substantial fraction of benchmark sentences appear in pretraining, rerun the comparison on examples confirmed absent from pretraining; the non-contextual BERT-over-FastText ordering would invert or shrink if memorization, not representation quality, was driving it.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a practical ordering of embeddings for Marathi: with a multinomial logistic regression classifier, contextual MahaBERT and MuRIL embeddings outperform all FastText models on every dataset; among non-contextual embeddings, MahaBERT and MuRIL extracted from the first embedding layer outperform MahaFT and IndicFT on most tasks, with sentiment classification and long-document news as exceptions; and SVD compression from 768 to 300 dimensions systematically degrades non-contextual BERT so that FastText then wins. The paper attributes the non-contextual advantage to MahaBERT's larger embedding table, larger pretraining data, and contextual training objective, all obtained at table-lookup cost.

Load-bearing premise

The comparison assumes the Marathi benchmark texts were not already present in the pretraining corpus of MahaBERT and MuRIL; the paper does not test for this overlap, so if it exists the BERT scores are inflated and the reported ordering could be an artifact.

Editorial extensions

If this is right

  • Marathi NLP pipelines can swap FastText for non-contextual MahaBERT embeddings at the same inference cost and expect better accuracy on sentiment, hate speech, and short-text news classification.
  • SVD-compressing BERT vectors down to 300 dimensions is not a neutral way to level the playing field: it lowers non-contextual BERT below FastText, so uncompressed embeddings should be used when dimension is not a constraint.
  • Contextual BERT remains the best representation on every task tested, preserving its role when a full forward pass is affordable.
  • The FastText advantage on long-document news shows the ranking is task-dependent, not a universal law.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit: if non-contextual BERT works from a lookup table, the embedding matrix can be precomputed once and served with the same infrastructure as FastText, so a latency and memory benchmark would quantify the practical gain.
  • The paper does not check whether the benchmark texts overlap the pretraining corpus of the BERT models; a membership test would determine whether the non-contextual BERT advantage survives without data leakage.
  • The same experimental design transfers to other languages that have a BERT and a FastText trained on the same corpus, which would show whether the finding is about Marathi or about embedding quality.
  • The high fold variance on the sentiment dataset suggests a paired significance test across folds could separate real exceptions from noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper compares non-contextual embeddings extracted from the first embedding layer of MahaBERT and MuRIL with FastText embeddings (IndicFT and MahaFT) on six Marathi text-classification tasks: MahaSent sentiment, MahaHate 2-class and 4-class hate speech, and MahaNews SHC, LDC, and LPC news classification. All embeddings are classified with multinomial logistic regression. The paper also evaluates contextual BERT embeddings and SVD-compressed 300-dimensional BERT variants. The headline claim is that non-contextual BERT embeddings outperform FastText embeddings, with contextual BERT performing best and compression hurting non-contextual BERT.

Significance. The practical question is worthwhile: non-contextual BERT embeddings require only a table lookup and could replace FastText in low-resource settings if they are reliably better. The study covers six tasks, two BERT models, two FastText models, and a dimensionality-leveling analysis, and it provides 5-fold cross-validation variance estimates for the main non-contextual comparison. However, the central empirical claim is not yet established: the paper's own Table 2 shows FastText ahead on half the tasks, the reported differences are comparable to cross-validation noise, and no significance tests or confidence intervals are given. The paper also does not test for pretraining/test-set overlap, which is relevant because the models and benchmarks share L3Cube provenance. No code or embedding artifacts are released, which limits reproducibility. These issues are fixable with additional analysis, so a major revision is appropriate.

major comments (4)
  1. [Abstract and §5.2, Table 2] The central claim is not supported by the paper's own variance estimates. Table 2 shows MahaFT outperforming MahaBERT on MahaSent (77.28 vs 76.56), SHC (87.25 vs 86.66), and LDC (92.79 vs 92.47), so FastText wins on three of the six tasks. The BERT wins on MahaHate-4class, MahaHate-2class, and LPC are 2.44, 0.48, and 1.56 points, respectively; with per-fold standard deviations of roughly 0.3–1.1 and no paired significance test, these gaps are within noise. Section 5.2 itself acknowledges the LDC reversal as stable, yet the abstract states without qualification that non-contextual BERT embeddings outperform FastText. Please either add a proper significance analysis (paired tests or confidence intervals across folds) and report effect sizes, or rephrase the headline claim to reflect the mixed evidence.
  2. [§4.1 and Tables 1–2] Error bars are reported only for non-contextual embeddings and MahaFT; IndicFT and all contextual models appear only as single-split accuracies in Table 1. Consequently, the comparative claims in §5.1 (contextual BERT consistently outperforms FastText) and the general 'FastText embeddings' claim in the abstract are not supported by any variance estimate. Please provide cross-validation or bootstrap intervals for every condition, including IndicFT and the contextual variants, so that the reader can judge whether the observed ordering is reliable.
  3. [§3 and §6] Potential pretraining/test overlap is a confound that is not tested. Section 6 states that both MahaBERT and MahaFT were trained on the same L3Cube-MahaCorpus of 752 million tokens, while MahaSent, MahaHate, and MahaNews are also L3Cube releases. If benchmark sentences appear in the pretraining corpus, both embedding types could be inflated, and the relative comparison could change when tested on truly unseen text. Please measure n-gram overlap between the pretraining corpus and the benchmark test sets, and rerun the headline comparisons on a non-overlapping subset.
  4. [§4] The SVD compression procedure is underspecified. The paper does not state whether the SVD is fit on the training embeddings only or on the full dataset, nor how the top 300 components are applied to test embeddings. If the SVD is fit on the full dataset, the compressed results leak test information and the conclusion that compression harms non-contextual BERT performance is unverified. Please clarify the fitting procedure and, if necessary, redo the compression within each cross-validation fold.
minor comments (5)
  1. [Table 2] The header 'A vg' should read 'Avg'; also, the table caption says 'Non-contextual embedding' but the table includes MahaFT, which is not a BERT model—clarify the caption.
  2. [Throughout] The model name is written as 'Muril' in Tables 1–2 and the text but as 'MuRIL' in the references; please use the official capitalization consistently.
  3. [Introduction and §4] The definition of non-contextual embeddings is inconsistent: the introduction says 'extracted just before the first embedding layer,' while Section 4 says 'output of the first embedding layer was used.' Please clarify which tensor is used.
  4. [§4.2 and Figures 3–4] The abbreviation 'T-SNE' should be 't-SNE', and the text should refer to Figure 3 for LDC and Figure 4 for all datasets consistently.
  5. [§4.1] The paper reports accuracy only; for hate speech detection in particular, class-balanced macro-F1 would be a useful additional metric even if the datasets are stated to be balanced.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the BERT-vs-FastText comparison is an empirical benchmark with external baselines; no fitted parameter is renamed as a prediction and the central claim does not reduce to a self-citation chain.

full rationale

The paper is an empirical comparison, not a derivation. Non-contextual BERT embeddings are obtained by table lookup and FastText embeddings by averaging word vectors; both are then passed to the same multinomial logistic regression classifier on fixed data splits. No quantity in the paper is defined in terms of another result, and no fitted parameter is later reported as a prediction. The headline claim that non-contextual BERT embeddings outperform FastText is supported only by the measured accuracies in Tables 1 and 2, which are in principle falsifiable against the held-out test sets and include external baselines such as MuRIL and IndicFT. The fact that MahaBERT, MahaFT, and the benchmark datasets originate from L3Cube is a potential overlap or benchmark-provenance confound, but it is not a definitional equivalence: the evaluation does not use the test labels to construct the embeddings, and the comparison remains empirically meaningful even if the statistical support for the abstract's claim is weak. Citations to Joshi (2022a) establish model and corpus provenance only; they are not the evidence for the performance ranking. There is no circular step to quote or exhibit.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central comparison rests on several unverified assumptions about data provenance, vocabulary coverage, and metric choice. No new entities are introduced.

free parameters (2)
  • SVD target dimensionality = 300
    Chosen by hand to match FastText's 300 dimensions for a fair comparison; not fitted to data but affects all compressed-embedding results.
  • MLR hyperparameters = unreported
    Logistic regression hyperparameters (solver, regularization strength, iterations) are not stated, so results depend on unreported choices.
assumptions (4)
  • domain assumption Evaluation datasets are disjoint from pretraining corpora of MahaBERT and MuRIL.
    If overlap exists, BERT's accuracy advantage over FastText could be inflated by memorization; not tested in the paper.
  • domain assumption FastText custom vocabulary covers all test words, or OOV words are handled.
    The embedding matrix is built only from train and validation vocabulary; test-time handling of unseen words is unspecified.
  • domain assumption Accuracy is an appropriate metric given balanced classes.
    The paper asserts datasets are balanced but does not provide class distributions.
  • domain assumption The first embedding layer output is a valid static word representation for sentence averaging.
    Standard in static BERT embedding extraction, but the exact layer output is not fully specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Non-Contextual BERT or FastText? A Comparative Analysis." pith.science (2026). https://pith.science/paper/ASAWZVEV

@misc{pith2026241117661,
  author       = {Pith},
  title        = {Pith review of: Non-Contextual BERT or FastText? A Comparative Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASAWZVEV}},
  note         = {Machine review of arXiv:2411.17661}
}
read the original abstract

Natural Language Processing (NLP) for low-resource languages, which lack large annotated datasets, faces significant challenges due to limited high-quality data and linguistic resources. The selection of embeddings plays a critical role in achieving strong performance in NLP tasks. While contextual BERT embeddings require a full forward pass, non-contextual BERT embeddings rely only on table lookup. Existing research has primarily focused on contextual BERT embeddings, leaving non-contextual embeddings largely unexplored. In this study, we analyze the effectiveness of non-contextual embeddings from BERT models (MuRIL and MahaBERT) and FastText models (IndicFT and MahaFT) for tasks such as news classification, sentiment analysis, and hate speech detection in one such low-resource language Marathi. We compare these embeddings with their contextual and compressed variants. Our findings indicate that non-contextual BERT embeddings extracted from the model's first embedding layer outperform FastText embeddings, presenting a promising alternative for low-resource NLP.

Figures

Figures reproduced from arXiv: 2411.17661 by the authors.

Figure 1
Figure 1. Embedding extraction workflow for contextual and non-contextual representations [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. SVD compression of BERT embeddings in English, with BERT embeddings outperform￾ing FastText. This trend continues in hate speech detection, where Rajput et al. (2021) find that neu￾ral network classifiers using contextual BERT em￾beddings perform better than those with FastText embeddings alone, further supporting BERT’s ef￾fectiveness. Additionally, Chanda (2021) assess contextual BERT embeddings against traditiona… view at source ↗
Figure 3
Figure 3. T-SNE Plot For BERT and FastText Embeddings (c stands for compressed) . [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: T-SNE Visualisation A Appendix A.1 T-SNE Visualisations Refer to [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 15 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Enhancing Depressive Post Detection in Bangla: A Comparative Study of TF-IDF, BERT and FastText Embeddings

    Saad Ahmed, Mahdi H Sazan, Miraz A B M Muntasir, Rahman, Saad Ahmed Sazan, Mahdi H. Miraz, and M Muntasir Rahman. 2024. https://api.semanticscholar.org/CorpusID:271108715 Enhancing depressive post detection in bangla: A comparative study of tf-idf, bert and fasttext embeddings . ArXiv, abs/2407.09187

    work page Pith review arXiv 2024

  4. [4]

    Deepak Suresh Asudani, Naresh Kumar Nagwani, and Pradeep Singh. 2023. https://api.semanticscholar.org/CorpusID:257098478 Impact of word embedding models on text analytics in deep learning environment: a review . Artificial Intelligence Review, pages 1 -- 81

    work page 2023

  5. [5]

    Ashis Kumar Chanda. 2021. https://api.semanticscholar.org/CorpusID:237278096 Efficacy of bert embeddings on predicting disaster from twitter data . ArXiv, abs/2108.10698

    work page Pith review arXiv 2021

  6. [6]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    arXiv 2018

  7. [7]

    Organization of Knowledge and Advanced Technologies

    Ashwin Geet D'Sa, Irina Illina, and D. Fohr. 2020. https://api.semanticscholar.org/CorpusID:213529509 Bert and fasttext embeddings for automatic detection of toxic speech . 2020 International Multi-Conference on: “Organization of Knowledge and Advanced Technologies” (OCTA), pages 1--5

    work page 2020

  8. [8]

    Raviraj Joshi. 2022 a . https://aclanthology.org/2022.wildre-1.17 L 3 C ube- M aha C orpus and M aha BERT : M arathi monolingual corpus, M arathi BERT language models, and resources . In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference, pages 97--101, Marseille, France. European Language Resources Association

    work page 2022

Show all 26 references
  1. [9]

    Raviraj Joshi. 2022 b . L3cube-mahanlp: Marathi natural language processing datasets, models, and library. arXiv preprint arXiv:2205.14728

    2022 arXiv

  2. [10]

    Elif Kabullar and İlker Türker. 2022. Performance comparison of word embedding methods in text classification for various number of features

    2022

  3. [11]

    Khapra, and Pratyush Kumar

    Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.445 I ndic NLPS uite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual langua...

    2020 doi

  4. [12]

    M Moneb Khaled, Muhammad Al-Barham, Osama Ahmad Alomari, and Ashraf Elnagar. 2023. Arabic news articles classification using different word embeddings. In International Conference on Emerging Trends and Applications in Artificial Intelligence, pages 125--136. Springer

    2023

  5. [13]

    Asma Sadia Khan, Fariba Tasnia Khan, Tanjim Mahmud, Salman Karim Khan, Nahed Sharmen, Mohammad Shahadat Hossain, and Karl Andersson. 2024. Integrating bert embeddings with svm for prostate cancer prediction. In 2024 6th International Conference on Electrical Engineering and In...

    2024

  6. [14]

    Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. https://arxiv.org/abs/2103.107...

    2021 arXiv

  7. [15]

    Pranav Malik, Aditi Aggrawal, and Dinesh Kumar Vishwakarma. 2021. https://api.semanticscholar.org/CorpusID:233990897 Toxic speech detection using traditional machine learning models and bert and fasttext embedding with deep neural networks . 2021 5th International Conference o...

    2021

  8. [16]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. https://arxiv.org/abs/1301.3781 Efficient estimation of word representations in vector space . Preprint, arXiv:1301.3781

    2013 arXiv

  9. [17]

    Saloni Mittal, Vidula Magdum, Sharayu Hiwarkhedkar, Omkar Dhekane, and Raviraj Joshi. 2023. L3cube-mahanews: News-based short text and long document classification datasets in marathi. In International Conference on Speech and Language Technologies for Low-resource Languages, ...

    2023

  10. [18]

    Abu Bakr Mohammad, Kareem Eissa, and Samhaa El-Beltagy. 2017. https://doi.org/10.1016/j.procs.2017.10.117 Aravec: A set of arabic word embedding models for use in arabic nlp . Procedia Computer Science, 117:256--265

    2017 doi

  11. [19]

    Hrushikesh Patil, Abhishek Velankar, and Raviraj Joshi. 2022. L3cube-mahahate: A tweet-based marathi hate speech detection dataset and bert models. In Proceedings of the Third Workshop on Threat, Aggression and Cyberbullying (TRAC 2022), pages 1--9

    2022

  12. [20]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. http://www.aclweb.org/anthology/D14-1162 Glove: Global vectors for word representation . In Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543

    2014

  13. [21]

    Aabha Pingle, Aditya Vyawahare, Isha Joshi, Rahul Tangsali, and Raviraj Joshi. 2023. L3cube-mahasent-md: A multi-domain marathi sentiment analysis dataset and transformer models. arXiv preprint arXiv:2306.13888

    2023 arXiv

  14. [22]

    G. K. Rajput, Narinder Singh Punn, Sanjay Kumar Sonbhadra, and Sonali Agarwal. 2021. https://api.semanticscholar.org/CorpusID:235670237 Hate speech detection using static bert embeddings . ArXiv, abs/2106.15537

    2021 arXiv

  15. [23]

    Sello Ralethe. 2020. https://aclanthology.org/2020.lrec-1.301 Adaptation of deep bidirectional transformers for A frikaans language . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2475--2478, Marseille, France. European Language Resources As...

    2020

  16. [24]

    Muhammad Umer, Zainab Imtiaz, Muhammad Ahmad, Michele Nappi, Carlo Maria Medaglia, Gyu Sang Choi, and Arif Mehmood. 2022. https://api.semanticscholar.org/CorpusID:251829920 Impact of convolutional neural network and fasttext embedding on text classification . Multimedia Tools ...

    2022

  17. [25]

    Laurens van der Maaten and Geoffrey Hinton. 2008. http://jmlr.org/papers/v9/vandermaaten08a.html Visualizing data using t-sne . Journal of Machine Learning Research, 9(86):2579--2605

    2008

  18. [26]

    Pan Xie, Hengnian Gu, and Dongdai Zhou. 2024. https://api.semanticscholar.org/CorpusID:271295864 Modeling sentiment analysis for educational texts by combining bert and fasttext . 2024 6th International Conference on Computer Science and Technologies in Education (CSTE), pages...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.