{"id":"e15ea1bf-948b-4e3c-962d-4bffd4e88991","arxiv_id":"2411.18099","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning a small six-head Nepali BERT on a mixed news and social media corpus improved average intrinsic purity from 0.65 to 0.78 and downstream F1 from 0.74 to 0.81, while still trailing the larger NepaliBERT model.","lead":"This paper fine-tunes a small Nepali BERT model on newly collected news and social media text and reports improved embeddings compared to its base model. It is a concrete low-resource experiment, but the missing training and evaluation details limit how much of the result can be verified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are not supported by a specified evaluation protocol; without fixed clustering/classifier setups and datasets, the observed purity/F1 differences cannot be attributed to fine-tuning.","rationale":"The reader's weakest assumption singles out missing evaluation details—datasets, clustering parameters, downstream classifier, train/test split, and preprocessing—and that is exactly where the paper is most vulnerable. The reported tables are the only evidence for the headline result, and their meaning depends on choices that are entirely absent. I agree that this warrants a conditional verdict rather than rejection, because the paper does not contain internal contradictions that prove the result false; it simply lacks the information needed to check it. My stress-test adds the observation that the fine-tuning objective is also unspecified, which doubles the difficulty of interpreting the numbers, but the core load-bearing concern remains the under-specified evaluation. Therefore the reader's conditional verdict stands unchanged.","tokens_in":5467,"tokens_out":4390,"duration_ms":42103,"concrete_test":"Release the complete evaluation code and data splits, and re-run all three models under identical conditions: same intrinsic datasets with a fixed clustering algorithm (state k, distance metric, and token-embedding pooling) and the same extrinsic dataset with a fixed train/test split, classifier, and hyperparameters; repeat over 10 seeds and report mean±std. If the fine-tuned model's average purity/F1 no longer exceeds NpVec1 by the reported margins, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that fine-tuning NpVec1 on a new unlabeled corpus substantially improves embeddings—rests entirely on Tables 2 and 3. Neither table identifies the intrinsic evaluation data, the clustering algorithm or its parameters (e.g., k, distance metric, token-to-embedding pooling), nor the extrinsic news-classification dataset, classifier architecture, train/test split, or random seeds. Purity is extremely sensitive to the choice of k and clustering method; a per-model difference in these choices can manufacture arbitrary leads. The extrinsic F1 is reported from a single 60-epoch run with no validation-based early stopping or variance estimate, so the 7-point gain over NpVec1 could reflect one favorable seed or a split that overlaps with the fine-tuning corpus. Because the paper also does not specify the fine-tuning objective (masked LM, next-sentence prediction, or something else), the causal link between fine-tuning and the numbers is doubly unverifiable. Without artifacts or exact protocols, the comparison is unfalsifiable from the manuscript alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes fine-tuning a smaller, incomplete BERT model for Nepali (NpVec1, 6 attention heads, 300 hidden dimensions) on a newly collected corpus of 140.48M tokens that mixes regularized news data and unregularized social media data. It reports intrinsic evaluation via clustering purity on Sentiment, Relatedness, and Named Entity sets, and extrinsic evaluation as a news classification task with precision, recall, and F1. The key reported results are that fine-tuning improves the NpVec1 baseline substantially (average intrinsic purity 0.65 to 0.78; F1 0.74 to 0.81) while remaining below the NepaliBERT oracle on average (0.82 purity, 0.87 F1). The paper concludes that fine-tuning a pre-trained BERT model on a new unlabeled dataset can substantially improve performance over the precursor model and sometimes approach a larger architecture.","tokens_in":5751,"tokens_out":2938,"duration_ms":28216,"significance":"If the reported results are reproducible, the paper would make a useful empirical contribution for low-resource languages, showing that continued fine-tuning of a smaller BERT model on domain-relevant unlabeled data can yield meaningful gains over the original pretrained checkpoint. A positive aspect is that the evaluation is not circular: the three compared models are produced independently and assessed on external tasks, so the reported gains are not artifacts of fitting to the test sets. However, the manuscript as submitted is missing almost the entire experimental protocol needed to substantiate the central claim. The comparisons in Tables 2 and 3 rest on unspecified intrinsic evaluation datasets, clustering algorithms and parameters, downstream classifier details, train/test splits, and fine-tuning hyperparameters. Without these, the paper cannot support a causal attribution of the score differences to fine-tuning, and the results cannot be independently verified. The topic is timely and the direction is plausible, but the current level of detail is too low for the paper to be accepted in its present form.","major_comments":[{"comment":"The fine-tuning objective and hyperparameters are never specified. The paper states that the NpVec1 model was fine-tuned on the aggregated corpus, but it does not state the training objective (masked language modeling, next-sentence prediction, or another variant), the number of epochs, learning rate, batch size, optimizer, warmup schedule, masking probability, or any regularization. Without these details, the causal link between fine-tuning and the improved scores in Tables 2 and 3 cannot be verified or reproduced.","section":"§3.3, §4.2, §4.6"},{"comment":"The intrinsic evaluation protocol is severely underspecified. The paper reports clustering purity for 'Sentiment', 'Relatedness', and 'Named Entity' sets, but it does not identify these datasets, their sizes, or their sources. It also does not state the clustering algorithm, the number of clusters k, the distance metric, the token-to-embedding pooling method, or the preprocessing applied to the evaluation texts. Purity is highly sensitive to the choice of k and the clustering method, so differences across models could be manufactured by using different settings. The same protocol must be explicitly fixed for all three models for Tables 2 to support the paper's conclusions.","section":"§4.2, Table 2"},{"comment":"The extrinsic evaluation on news classification is equally underspecified. The manuscript does not describe the news corpus, the number of classes, the train/test split, the classifier architecture, the optimizer, the learning rate, the regularization, or whether any validation-based early stopping was used. The F1 scores come from a single 60-epoch run with no reported variance or statistical test, so the 7-point gain over NpVec1 could be due to one favorable seed or a split that overlaps with the fine-tuning corpus. At minimum, the authors need to provide the full classification setup, multiple seeds, and confidence intervals.","section":"§4.6, Table 3"},{"comment":"The Hindi filtering step is concerning and may directly affect the fine-tuning corpus. The authors report that the langdetect library labeled all data instances as Hindi, which suggests the detection step failed; they then filtered using a regular expression of 100 most common Hindi words. The paper does not report how many instances were removed, the false-positive rate for Nepali text, or the resulting corpus quality. Since the central claim is that new unlabeled data improves the model, the composition and quality of that data must be documented for the result to be interpretable.","section":"§3.2.1"}],"minor_comments":[{"comment":"The title is rendered as 'FINE -T UNING' and there are inconsistent capitalizations of the model name ('NPVec1' vs. 'NpVec1') throughout the paper; these should be unified.","section":"Title and §3.3"},{"comment":"The NepBERTa citation in the first paragraph is missing a space and the reference entry for Levy & Goldberg (2014) lacks the full publication venue and title.","section":"§2"},{"comment":"There are typos such as 'use off regular expressions' and the phrase 'a pattern of 100 most occurring Hindi words' should be 'the 100 most frequent Hindi words'.","section":"§3.2.1"},{"comment":"The table header contains 'Regualted Corpus' and the text says 'a few ruls of the scraping script'; these should be corrected to 'Regularized Corpus' and 'a few runs'.","section":"§4.1, Table 1"},{"comment":"The cluster figures are presented without axes labels, cluster-count information, or any explanation of how the clusters were produced, so they do not provide verifiable evidence and should either be described or removed.","section":"§4.3–4.5, Figures 2–10"}],"recommendation":"major_revision","confidential_remarks":"The missing experimental detail is the dominant issue: the paper needs a complete protocol for fine-tuning, intrinsic evaluation, and extrinsic evaluation, along with variance estimates, before the central claim can be assessed. I do not see evidence of circularity or deliberate overclaiming, but the current manuscript is not reproducible. The authors should also consider whether the underlying experiments were run with identical preprocessing and evaluation settings across the three models; if not, the reported gains would not be attributable to fine-tuning even after the protocol is documented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest, honest incremental result with a serious reproducibility gap. The authors fine-tune the small NpVec1 BERT (6 heads, 300 hidden) on a newly collected 140M-token Nepali corpus mixing news and social media, and report that the finetuned model beats its precursor on intrinsic clustering purity (0.78 vs 0.65) and extrinsic news classification F1 (0.81 vs 0.74), while staying below NepaliBERT (0.82 purity, 0.87 F1). Those directions are plausible and consistent with the transfer-learning literature.\n\nWhat's new: the specific finetuned model, the new corpus, and the evaluation numbers. That is real, though modest. The paper cites the relevant prior work (NpVec1, NepaliBERT, NepBERTa) and does not overclaim: the abstract explicitly says the oracle is better on average. The writing is clear enough for a short workshop-style paper.\n\nWhere it falls down: the evaluation section is a black box. Tables 2 and 3 give no details on the intrinsic evaluation data, the clustering algorithm, k, distance metric, or how token embeddings were pooled into word/sentence vectors. The extrinsic task is \"news classification\" but the dataset, label set, train/test split, classifier architecture, and random seeds are absent. The fine-tuning procedure is also unspecified: no objective function (masked LM? NSP?), no hyperparameters, no number of steps, no compute budget. There are no error bars, no multiple seeds, and no code or data release. The 60-epoch extrinsic run with no validation-based stopping could easily be one lucky seed.\n\nThe consequence is not that the result is false; it is that the central comparison is unfalsifiable as written. If the intrinsic and extrinsic setups differed between models, the reported gains could be artifacts. This is a load-bearing omission because the paper's only evidence is these tables. The stress-test note is right on target.\n\nWho this is for: readers working on low-resource NLP for Nepali or similar languages might get value from the corpus and the idea; but they would need the artifacts to build on it. As a paper, it is not ready for publication without a major revision that pins down the protocol and releases code/data.\n\nRecommendation: I would not desk-reject it outright, but I would send it to a referee only with the expectation of heavy revision. If the author can supply the missing experimental details and artifacts, the result could be a useful incremental contribution. For now, treat the numbers as plausible but unverified.","headline":"A plausible but under-specified demonstration that fine-tuning a small Nepali BERT on fresh data helps; the numbers are believable, but the missing evaluation protocol makes the comparison unfalsifiable as written.","tokens_in":6152,"tokens_out":2186,"would_cite":false,"duration_ms":19288,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a small pre-trained BERT on new unlabeled Nepali text substantially improves its embeddings over the precursor and sometimes approaches the larger oracle model.","keywords":["Nepali","BERT fine-tuning","low-resource NLP","contextual embeddings","transfer learning","NpVec1","NepaliBERT","intrinsic and extrinsic evaluation"],"falsifier":"Hold one evaluation pipeline fixed and rerun NpVec1, the fine-tuned model, and NepaliBERT on the same held-out Nepali sentences with the same clustering algorithm and number of clusters and the same news-classification train/test split; if the fine-tuned model no longer beats NpVec1 on average purity and F1, the central claim fails.","tokens_in":5249,"feed_emoji":"📈","tokens_out":14085,"duration_ms":103619,"temperature":0.7,"pith_summary":"The paper claims that a small, six-head BERT model for Nepali can be meaningfully upgraded by fine-tuning it on a new unlabeled corpus that includes edited news text and unedited social-media text, rather than by pre-training a larger model from scratch. It matters because Nepali has limited data and compute, so a cheap fine-tuning route to better embeddings would make low-resource NLP more practical. On intrinsic clustering evaluations, the fine-tuned model's average purity rises from 0.65 for its NpVec1 precursor to 0.78, and on an extrinsic news-classification task its F1 rises from 0.74 to 0.81. The full 12-layer NepaliBERT oracle still leads on average with 0.82 purity and 0.87 F1, so the paper's contribution is the large gain over the precursor at small-model cost.","feed_headline":"Small fine-tuned BERT beats its precursor on Nepali tests","feed_subtitle":"Fine-tuning on unlabeled Nepali text lifts a six-head BERT past its precursor at a fraction of the pre-training cost","key_machinery":"The mechanism is transfer learning applied to a small BERT: take NpVec1, a six-head, 300-hidden-dimension Nepali BERT, and fine-tune it on a new unlabeled corpus that introduces unregularized social-media text missing from the original pretraining. Fine-tuning adjusts the model's weights on the new data after preprocessing that removes Hindi instances from the Devanagari-script streams, collapses variant vowel spellings, and lexically breaks suffixes off agglutinated words. The evaluation machinery is a three-way comparison: clustering purity on sentiment, relatedness, and named-entity sets for intrinsic quality, and a 60-epoch news-classification task with macro precision, recall, and F1 for extrinsic quality, with NpVec1 as baseline and NepaliBERT as oracle.","core_discovery":"The core discovery, stated as the paper's conclusion, is that fine-tuning a pre-trained BERT model on a completely new unlabeled dataset can lead to substantially better performance than the pre-trained model itself, sometimes competing with a model of greater architecture. The author starts from NpVec1, a Nepali BERT with six attention heads and 300 hidden dimensions pre-trained on 279 million word tokens, and continues training it on an aggregated 140.48-million-token corpus of 43.58 million regularized news tokens and 96.90 million unregularized social-media tokens, after filtering Hindi text, standardizing vowel variants, and applying lexical analysis to agglutinative suffix tokens. The fine-tuned model exceeds NpVec1 on average in both intrinsic clustering purity and extrinsic news-classification F1, while trailing NepaliBERT, the full 12-layer, 768-hidden-dimension oracle. From this the paper concludes that pre-training and fine-tuning a low-resource language model on a wide data domain can substitute for full pre-training when data are scarce.","pith_inferences":["The paper does not ablate the unregularized social-media portion, so a direct test of its own attribution would be to fine-tune NpVec1 on the regularized news corpus alone and compare; if the gain disappears, the social-media domain is the active ingredient.","Because the paper does not state the clustering algorithm, its parameters, the downstream classifier, or the train/test split, the reported improvements are not yet reproducible; fixing these details would let other low-resource languages adopt the recipe.","The same fine-tuning recipe should transfer to other agglutinative low-resource languages with suffix-combining morphology, provided the Hindi-filtering and vowel-standardization steps are replaced with language-appropriate analogues.","The scope of the conclusion is one base model and one new corpus, so the wider claim that domain-diverse fine-tuning helps generally is a hypothesis to replicate across other languages and base models rather than an established law."],"forward_implications":["If the claim holds, teams working on low-resource languages can improve an existing small BERT checkpoint with unlabeled web-scraped text instead of waiting for enough data and compute to pre-train a full model.","A seven-point F1 gain on news classification suggests downstream applications such as sentiment analysis, named-entity recognition, and machine translation can inherit better representations without switching to a larger architecture.","The fine-tuned model still trails NepaliBERT by 0.04 average purity and 0.06 F1, so the larger architecture remains the stronger ceiling; the practical value is the gain per unit of compute.","The consistency of the fine-tuned model's scores across the three clustering sets suggests it generalizes across domains rather than specializing in one, a property the paper attributes to the wider data domain."],"supporting_citations":[{"why":"Supplies the NpVec1 baseline BERT model and its 279-million-token pre-training corpus, the object that is fine-tuned.","marker":"Koirala & Niraula, 2021"},{"why":"Supplies the NepaliBERT oracle model used as the upper bound in both intrinsic and extrinsic evaluations.","marker":"Pudasaini et al., 2023"},{"why":"Supplies the BERT architecture and the transfer-learning fine-tuning procedure that the paper applies to NpVec1.","marker":"Devlin et al., 2018"}],"fun_headline_variants":["Fine-tuned six-head BERT beats Nepali baseline","Small BERT fine-tuned outperforms its Nepali precursor","Nepali BERT boost: fine-tuning small model lifts scores","Fine-tuning lifts small BERT past its Nepali base","Six-head BERT fine-tuned improves Nepali NLP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes all three models were evaluated under identical intrinsic and extrinsic conditions, including the same evaluation datasets, clustering method and settings, downstream classifier, train/test split, and preprocessing, because the paper does not specify these choices and differing setups could produce the same reported numbers.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned six-head BERT beats Nepali baseline","Small BERT fine-tuned outperforms its Nepali precursor","Nepali BERT boost: fine-tuning small model lifts scores","Fine-tuning lifts small BERT past its Nepali base","Six-head BERT fine-tuned improves Nepali NLP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2175,"prompt_tokens":889,"completion_tokens":1286,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1216}},"tokens_in":505,"tokens_out":1286,"duration_ms":9062,"temperature":1.0,"reasoning_tokens":1216,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:30:17.772982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold one evaluation pipeline fixed and rerun NpVec1, the fine-tuned model, and NepaliBERT on the same held-out Nepali sentences with the same clustering algorithm and number of clusters and the same news-classification train/test split; if the fine-tuned model no longer beats NpVec1 on average purity and F1, the central claim fails.","supporting_citations":[{"cited_title":"Npvec1: Word embeddings for nepali-construction and evaluation","cited_arxiv_id":null,"evidence_quote":"Supplies the NpVec1 baseline BERT model and its 279-million-token pre-training corpus, the object that is fine-tuned."}],"review_version":1}