Pith. sign in

REVIEW 4 major objections 5 minor 14 references

Fine-Tuning Small Embeddings for Elevated Performance

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a small pre-trained BERT on new unlabeled Nepali text substantially improves its embeddings over the precursor and sometimes approaches the larger oracle model.

desk verdict A plausible but under-specified demonstration that fine-tuning a small Nepali BERT on fresh data helps; the numbers are believable, but the missing evaluation protocol makes the comparison unfalsifiable as written. read the letter →

arxiv 2411.18099 v1 pith:7NDZLU6J submitted 2024-11-27 cs.CL

Biraj Silwal This is my paper · ORCID
classification cs.CL
keywords NepaliBERTfine-tuninglow-resourceNLPcontextualembeddingstransferlearningNpVec1intrinsicandextrinsicevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small, six-head BERT model for Nepali can be meaningfully upgraded by fine-tuning it on a new unlabeled corpus that includes edited news text and unedited social-media text, rather than by pre-training a larger model from scratch. It matters because Nepali has limited data and compute, so a cheap fine-tuning route to better embeddings would make low-resource NLP more practical. On intrinsic clustering evaluations, the fine-tuned model's average purity rises from 0.65 for its NpVec1 precursor to 0.78, and on an extrinsic news-classification task its F1 rises from 0.74 to 0.81. The full 12-layer NepaliBERT oracle still leads on average with 0.82 purity and 0.87 F1, so the paper's contribution is the large gain over the precursor at small-model cost.

What carries the argument

The mechanism is transfer learning applied to a small BERT: take NpVec1, a six-head, 300-hidden-dimension Nepali BERT, and fine-tune it on a new unlabeled corpus that introduces unregularized social-media text missing from the original pretraining. Fine-tuning adjusts the model's weights on the new data after preprocessing that removes Hindi instances from the Devanagari-script streams, collapses variant vowel spellings, and lexically breaks suffixes off agglutinated words. The evaluation machinery is a three-way comparison: clustering purity on sentiment, relatedness, and named-entity sets for intrinsic quality, and a 60-epoch news-classification task with macro precision, recall, and F1 for extrinsic quality, with NpVec1 as baseline and NepaliBERT as oracle.

What would settle it

Hold one evaluation pipeline fixed and rerun NpVec1, the fine-tuned model, and NepaliBERT on the same held-out Nepali sentences with the same clustering algorithm and number of clusters and the same news-classification train/test split; if the fine-tuned model no longer beats NpVec1 on average purity and F1, the central claim fails.

Watch

Extended reading notes

Core claim

The core discovery, stated as the paper's conclusion, is that fine-tuning a pre-trained BERT model on a completely new unlabeled dataset can lead to substantially better performance than the pre-trained model itself, sometimes competing with a model of greater architecture. The author starts from NpVec1, a Nepali BERT with six attention heads and 300 hidden dimensions pre-trained on 279 million word tokens, and continues training it on an aggregated 140.48-million-token corpus of 43.58 million regularized news tokens and 96.90 million unregularized social-media tokens, after filtering Hindi text, standardizing vowel variants, and applying lexical analysis to agglutinative suffix tokens. The fine-tuned model exceeds NpVec1 on average in both intrinsic clustering purity and extrinsic news-classification F1, while trailing NepaliBERT, the full 12-layer, 768-hidden-dimension oracle. From this the paper concludes that pre-training and fine-tuning a low-resource language model on a wide data domain can substitute for full pre-training when data are scarce.

Load-bearing premise

The central comparison assumes all three models were evaluated under identical intrinsic and extrinsic conditions, including the same evaluation datasets, clustering method and settings, downstream classifier, train/test split, and preprocessing, because the paper does not specify these choices and differing setups could produce the same reported numbers.

Editorial extensions

If this is right

  • If the claim holds, teams working on low-resource languages can improve an existing small BERT checkpoint with unlabeled web-scraped text instead of waiting for enough data and compute to pre-train a full model.
  • A seven-point F1 gain on news classification suggests downstream applications such as sentiment analysis, named-entity recognition, and machine translation can inherit better representations without switching to a larger architecture.
  • The fine-tuned model still trails NepaliBERT by 0.04 average purity and 0.06 F1, so the larger architecture remains the stronger ceiling; the practical value is the gain per unit of compute.
  • The consistency of the fine-tuned model's scores across the three clustering sets suggests it generalizes across domains rather than specializing in one, a property the paper attributes to the wider data domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the unregularized social-media portion, so a direct test of its own attribution would be to fine-tune NpVec1 on the regularized news corpus alone and compare; if the gain disappears, the social-media domain is the active ingredient.
  • Because the paper does not state the clustering algorithm, its parameters, the downstream classifier, or the train/test split, the reported improvements are not yet reproducible; fixing these details would let other low-resource languages adopt the recipe.
  • The same fine-tuning recipe should transfer to other agglutinative low-resource languages with suffix-combining morphology, provided the Hindi-filtering and vowel-standardization steps are replaced with language-appropriate analogues.
  • The scope of the conclusion is one base model and one new corpus, so the wider claim that domain-diverse fine-tuning helps generally is a hypothesis to replicate across other languages and base models rather than an established law.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes fine-tuning a smaller, incomplete BERT model for Nepali (NpVec1, 6 attention heads, 300 hidden dimensions) on a newly collected corpus of 140.48M tokens that mixes regularized news data and unregularized social media data. It reports intrinsic evaluation via clustering purity on Sentiment, Relatedness, and Named Entity sets, and extrinsic evaluation as a news classification task with precision, recall, and F1. The key reported results are that fine-tuning improves the NpVec1 baseline substantially (average intrinsic purity 0.65 to 0.78; F1 0.74 to 0.81) while remaining below the NepaliBERT oracle on average (0.82 purity, 0.87 F1). The paper concludes that fine-tuning a pre-trained BERT model on a new unlabeled dataset can substantially improve performance over the precursor model and sometimes approach a larger architecture.

Significance. If the reported results are reproducible, the paper would make a useful empirical contribution for low-resource languages, showing that continued fine-tuning of a smaller BERT model on domain-relevant unlabeled data can yield meaningful gains over the original pretrained checkpoint. A positive aspect is that the evaluation is not circular: the three compared models are produced independently and assessed on external tasks, so the reported gains are not artifacts of fitting to the test sets. However, the manuscript as submitted is missing almost the entire experimental protocol needed to substantiate the central claim. The comparisons in Tables 2 and 3 rest on unspecified intrinsic evaluation datasets, clustering algorithms and parameters, downstream classifier details, train/test splits, and fine-tuning hyperparameters. Without these, the paper cannot support a causal attribution of the score differences to fine-tuning, and the results cannot be independently verified. The topic is timely and the direction is plausible, but the current level of detail is too low for the paper to be accepted in its present form.

major comments (4)
  1. [§3.3, §4.2, §4.6] The fine-tuning objective and hyperparameters are never specified. The paper states that the NpVec1 model was fine-tuned on the aggregated corpus, but it does not state the training objective (masked language modeling, next-sentence prediction, or another variant), the number of epochs, learning rate, batch size, optimizer, warmup schedule, masking probability, or any regularization. Without these details, the causal link between fine-tuning and the improved scores in Tables 2 and 3 cannot be verified or reproduced.
  2. [§4.2, Table 2] The intrinsic evaluation protocol is severely underspecified. The paper reports clustering purity for 'Sentiment', 'Relatedness', and 'Named Entity' sets, but it does not identify these datasets, their sizes, or their sources. It also does not state the clustering algorithm, the number of clusters k, the distance metric, the token-to-embedding pooling method, or the preprocessing applied to the evaluation texts. Purity is highly sensitive to the choice of k and the clustering method, so differences across models could be manufactured by using different settings. The same protocol must be explicitly fixed for all three models for Tables 2 to support the paper's conclusions.
  3. [§4.6, Table 3] The extrinsic evaluation on news classification is equally underspecified. The manuscript does not describe the news corpus, the number of classes, the train/test split, the classifier architecture, the optimizer, the learning rate, the regularization, or whether any validation-based early stopping was used. The F1 scores come from a single 60-epoch run with no reported variance or statistical test, so the 7-point gain over NpVec1 could be due to one favorable seed or a split that overlaps with the fine-tuning corpus. At minimum, the authors need to provide the full classification setup, multiple seeds, and confidence intervals.
  4. [§3.2.1] The Hindi filtering step is concerning and may directly affect the fine-tuning corpus. The authors report that the langdetect library labeled all data instances as Hindi, which suggests the detection step failed; they then filtered using a regular expression of 100 most common Hindi words. The paper does not report how many instances were removed, the false-positive rate for Nepali text, or the resulting corpus quality. Since the central claim is that new unlabeled data improves the model, the composition and quality of that data must be documented for the result to be interpretable.
minor comments (5)
  1. [Title and §3.3] The title is rendered as 'FINE -T UNING' and there are inconsistent capitalizations of the model name ('NPVec1' vs. 'NpVec1') throughout the paper; these should be unified.
  2. [§2] The NepBERTa citation in the first paragraph is missing a space and the reference entry for Levy & Goldberg (2014) lacks the full publication venue and title.
  3. [§3.2.1] There are typos such as 'use off regular expressions' and the phrase 'a pattern of 100 most occurring Hindi words' should be 'the 100 most frequent Hindi words'.
  4. [§4.1, Table 1] The table header contains 'Regualted Corpus' and the text says 'a few ruls of the scraping script'; these should be corrected to 'Regularized Corpus' and 'a few runs'.
  5. [§4.3–4.5, Figures 2–10] The cluster figures are presented without axes labels, cluster-count information, or any explanation of how the clusters were produced, so they do not provide verifiable evidence and should either be described or removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the fine-tuning claim is an empirical comparison against external baselines and no fitted value is relabeled as a prediction.

full rationale

The paper's central claim is that fine-tuning the NpVec1 BERT model on a new unlabeled corpus improves embedding quality relative to that model's precursor. The evidence is an empirical comparison of three independently produced models: NpVec1 as baseline, the fine-tuned model, and NepaliBERT as oracle, evaluated on intrinsic clustering purity (Table 2) and extrinsic news-classification macro F1 (Table 3). There is no derivation in which an output is defined in terms of the fitted model, no parameter fitted to the evaluation data and then reported as a predicted result, and no load-bearing step justified by a self-citation. The evaluation protocol is underspecified — datasets, clustering hyperparameters, downstream classifier configuration, and random seeds are not given — but underspecification is a reproducibility and attribution risk, not a circularity: it does not make the reported gains equivalent to the model's inputs by construction. The fine-tuned model is compared against external checkpoints on held-out tasks, so the comparison is self-contained relative to the paper's claim, even if not fully reproducible from the manuscript alone.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several unstated choices: fine-tuning hyperparameters, evaluation configurations, preprocessing parameters, and the assumption that the chosen metrics reflect embedding quality. None of these are reported, and the corpus is not available, so the ledger is mostly composed of unverified empirical assumptions.

free parameters (4)
  • Fine-tuning hyperparameters = not reported
    Learning rate, batch size, number of epochs, and optimizer settings determine the finetuned model but are not given.
  • Intrinsic clustering parameters = not reported
    Number of clusters, distance metric, and token aggregation method for purity scores are unspecified.
  • Extrinsic classification configuration = not reported
    The news classification task lacks split sizes, class distribution, classifier type, and regularization; only '60 epochs' is stated.
  • Corpus preprocessing thresholds = not reported
    The Hindi-word regex pattern, vowel-standardization rules, and lexical analysis tokenizer are described qualitatively without parameters.
assumptions (4)
  • domain assumption Clustering purity and downstream F1 are valid proxies for embedding quality.
    The entire comparison rests on these metrics reflecting real semantic and syntactic quality.
  • domain assumption langdetect and the regex-based Hindi filter correctly removed non-Nepali text.
    Section 3.2.1 reports that langdetect labeled all instances as Hindi, so the filter relied on a manually-created regex pattern whose quality is unverified.
  • domain assumption The aggregated corpus is representative of Nepali language use and does not unfairly overlap the evaluation sets.
    Data collection from news and social media is described, but there is no leakage check between finetuning data and evaluation data.
  • domain assumption The base NpVec1 model is a proper BERT variant with 6 heads and 300 hidden units as cited.
    The paper takes the model description from Koirala and Niraula (2021) without independent verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine-Tuning Small Embeddings for Elevated Performance." pith.science (2026). https://pith.science/paper/7NDZLU6J

@misc{pith2026241118099,
  author       = {Pith},
  title        = {Pith review of: Fine-Tuning Small Embeddings for Elevated Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7NDZLU6J}},
  note         = {Machine review of arXiv:2411.18099}
}
read the original abstract

Contextual Embeddings have yielded state-of-the-art results in various natural language processing tasks. However, these embeddings are constrained by models requiring large amounts of data and huge computing power. This is an issue for low-resource languages like Nepali as the amount of data available over the internet is not always sufficient for the models. This work has taken an incomplete BERT model with six attention heads pretrained on Nepali language and finetuned it on previously unseen data. The obtained results from intrinsic and extrinsic evaluations have been compared to the results drawn from the original model baseline and a complete BERT model pretrained on Nepali language as the oracle. The results demonstrate that even though the oracle is better on average, finetuning the small embeddings drastically improves results compared to the original baseline.

Figures

Figures reproduced from arXiv: 2411.18099 by the authors.

Figure 1
Figure 1. Preprocessing techniques 3.3 TRANSFER LEARNING FOR WORD EMBEDDING GENERATION ON BERT BASED MODELS Transfer learning is a machine learning technique where a model trained on one task is fine-tuned on a second related task. This allows the model to benefit from the knowledge it has gained while solving the first task and apply it to the second task. The idea is to leverage the features learned by the model on the firs… view at source ↗
Figure 2
Figure 2. Finetuned model [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 5
Figure 5. Finetuned model [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figures from the paper (1 more)
Figure 8
Figure 8. Figure 8: Finetuned model [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 4 canonical work pages

  1. [1]

    Enriching word vectors with subword information

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5: 0 135--146, 2017

    work page 2017

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  3. [3]

    Speech & language processing

    Dan Jurafsky. Speech & language processing. Pearson Education India, 2000

    2000

  4. [4]

    Npvec1: Word embeddings for nepali-construction and evaluation

    Pravesh Koirala and Nobal B Niraula. Npvec1: Word embeddings for nepali-construction and evaluation. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pp.\ 174--184, 2021

    work page 2021

  5. [5]

    Dependency-based word embeddings

    Omer Levy and Yoav Goldberg. Dependency-based word embeddings. 2014

    work page 2014

  6. [6]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013

    2013

  7. [7]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp.\ 1532--1543, 2014

    2014

  8. [8]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, 2018

    2018

Show all 14 references
  1. [9]

    Nepalibert: Pre-training of masked language model in nepali corpus

    Shushanta Pudasaini, Subarna Shakya, Aakash Tamang, Sajjan Adhikari, Sunil Thapa, and Sagar Lamichhane. Nepalibert: Pre-training of masked language model in nepali corpus. In 2023 7th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), pp....

    2023

  2. [10]

    Nepberta: Nepali language model trained in a large corpus

    Sulav Timilsina, Milan Gautam, and Binod Bhattarai. Nepberta: Nepali language model trained in a large corpus. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natur...

    2022

  3. [11]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  4. [12]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  5. [13]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  6. [14]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.