{"id":"1bfe6ad7-8b4d-49b7-95f9-f076252d851d","arxiv_id":"2411.17661","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Non-contextual (first-layer) BERT embeddings generally outperform FastText on six Marathi classification tasks, though they lose on sentiment and long-document news.","lead":"This paper compares static, table-lookup BERT embeddings against FastText embeddings for Marathi text classification tasks like sentiment, news, and hate speech. It finds the static BERT embeddings generally win by a few accuracy points, but with notable exceptions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported BERT-vs-FastText differences are often within cross-validation noise; Table 2 shows FastText ahead on three of six tasks, so the abstract's superiority claim is not established.","rationale":"The reader's verdict is CONDITIONAL and identifies pretraining-data leakage as the weakest assumption. I agree that leakage is plausible and worth testing: MahaBERT is trained on L3Cube-MahaCorpus, and the three evaluation datasets are L3Cube releases, so exact or near-exact overlap could inflate BERT scores. However, the more directly load-bearing problem is that the reported effect sizes are within the paper's own cross-validation noise. Table 2 is the only place where variance is reported, and it shows FastText winning on half of the task/dataset combinations when averaged over folds. Without a significance test, the strong claim in the abstract is unsupported regardless of leakage. This is not a question of external consensus or novelty; it is an internal-evidence problem: the paper's own numbers contradict the strength of the conclusion. A paired significance test on the existing folds would settle the matter. The reader's leak concern would also be worth checking (e.g., n-gram overlap with the pretraining corpus), but the statistical fragility alone is sufficient to require a revised, more cautious claim. Since the reader already recommends conditional acceptance with requests for additional validation, my concern does not change the overall verdict; it reinforces the conditionality.","tokens_in":8121,"tokens_out":3493,"duration_ms":32213,"concrete_test":"Recompute the comparison using the exact fold splits from Table 2 and run a paired significance test (e.g., McNemar's test on the test-fold predictions, or a paired bootstrap with 10,000 resamples) between non-contextual MahaBERT and MahaFT for each of the six tasks. Report the difference, 95% confidence interval, and p-value for each task. If fewer than four of the six tasks show a statistically significant advantage for MahaBERT at p<0.05, the abstract's claim that non-contextual BERT outperforms FastText should be softened to 'comparable' or 'task-dependent.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that non-contextual BERT embeddings outperform FastText is not supported by the paper's own variance estimates. Table 2 reports 5-fold cross-validation averages for non-contextual embeddings and MahaFT. Comparing MahaBERT versus MahaFT: MahaSent 76.56 vs 77.28, SHC 86.66 vs 87.25, LDC 92.47 vs 92.79 -- FastText wins on these three. MahaBERT wins on MahaHate-4class (64.92 vs 62.48), MahaHate-2class (84.23 vs 83.75), and LPC (81.71 vs 80.15), but the margins are 2.44, 0.48, and 1.56 points respectively. Given per-fold standard deviations of roughly 0.3-1.1, most of these gaps are within noise, and the paper reports no significance test, confidence interval, or paired comparison. The single-split results in Table 1 also show MahaBERT below MahaFT on MahaSent and LDC, so the headline pattern is not consistent across evaluation settings. The reader's concern about pretraining/benchmark overlap (MahaBERT and MahaFT are both trained on L3Cube corpora while the benchmarks are L3Cube releases) is a real confound, but it is secondary: even if no leakage exists, the evidence as presented does not establish that non-contextual BERT is reliably better than FastText.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares non-contextual embeddings extracted from the first embedding layer of MahaBERT and MuRIL with FastText embeddings (IndicFT and MahaFT) on six Marathi text-classification tasks: MahaSent sentiment, MahaHate 2-class and 4-class hate speech, and MahaNews SHC, LDC, and LPC news classification. All embeddings are classified with multinomial logistic regression. The paper also evaluates contextual BERT embeddings and SVD-compressed 300-dimensional BERT variants. The headline claim is that non-contextual BERT embeddings outperform FastText embeddings, with contextual BERT performing best and compression hurting non-contextual BERT.","tokens_in":8443,"tokens_out":6878,"duration_ms":54735,"significance":"The practical question is worthwhile: non-contextual BERT embeddings require only a table lookup and could replace FastText in low-resource settings if they are reliably better. The study covers six tasks, two BERT models, two FastText models, and a dimensionality-leveling analysis, and it provides 5-fold cross-validation variance estimates for the main non-contextual comparison. However, the central empirical claim is not yet established: the paper's own Table 2 shows FastText ahead on half the tasks, the reported differences are comparable to cross-validation noise, and no significance tests or confidence intervals are given. The paper also does not test for pretraining/test-set overlap, which is relevant because the models and benchmarks share L3Cube provenance. No code or embedding artifacts are released, which limits reproducibility. These issues are fixable with additional analysis, so a major revision is appropriate.","major_comments":[{"comment":"The central claim is not supported by the paper's own variance estimates. Table 2 shows MahaFT outperforming MahaBERT on MahaSent (77.28 vs 76.56), SHC (87.25 vs 86.66), and LDC (92.79 vs 92.47), so FastText wins on three of the six tasks. The BERT wins on MahaHate-4class, MahaHate-2class, and LPC are 2.44, 0.48, and 1.56 points, respectively; with per-fold standard deviations of roughly 0.3–1.1 and no paired significance test, these gaps are within noise. Section 5.2 itself acknowledges the LDC reversal as stable, yet the abstract states without qualification that non-contextual BERT embeddings outperform FastText. Please either add a proper significance analysis (paired tests or confidence intervals across folds) and report effect sizes, or rephrase the headline claim to reflect the mixed evidence.","section":"Abstract and §5.2, Table 2"},{"comment":"Error bars are reported only for non-contextual embeddings and MahaFT; IndicFT and all contextual models appear only as single-split accuracies in Table 1. Consequently, the comparative claims in §5.1 (contextual BERT consistently outperforms FastText) and the general 'FastText embeddings' claim in the abstract are not supported by any variance estimate. Please provide cross-validation or bootstrap intervals for every condition, including IndicFT and the contextual variants, so that the reader can judge whether the observed ordering is reliable.","section":"§4.1 and Tables 1–2"},{"comment":"Potential pretraining/test overlap is a confound that is not tested. Section 6 states that both MahaBERT and MahaFT were trained on the same L3Cube-MahaCorpus of 752 million tokens, while MahaSent, MahaHate, and MahaNews are also L3Cube releases. If benchmark sentences appear in the pretraining corpus, both embedding types could be inflated, and the relative comparison could change when tested on truly unseen text. Please measure n-gram overlap between the pretraining corpus and the benchmark test sets, and rerun the headline comparisons on a non-overlapping subset.","section":"§3 and §6"},{"comment":"The SVD compression procedure is underspecified. The paper does not state whether the SVD is fit on the training embeddings only or on the full dataset, nor how the top 300 components are applied to test embeddings. If the SVD is fit on the full dataset, the compressed results leak test information and the conclusion that compression harms non-contextual BERT performance is unverified. Please clarify the fitting procedure and, if necessary, redo the compression within each cross-validation fold.","section":"§4"}],"minor_comments":[{"comment":"The header 'A vg' should read 'Avg'; also, the table caption says 'Non-contextual embedding' but the table includes MahaFT, which is not a BERT model—clarify the caption.","section":"Table 2"},{"comment":"The model name is written as 'Muril' in Tables 1–2 and the text but as 'MuRIL' in the references; please use the official capitalization consistently.","section":"Throughout"},{"comment":"The definition of non-contextual embeddings is inconsistent: the introduction says 'extracted just before the first embedding layer,' while Section 4 says 'output of the first embedding layer was used.' Please clarify which tensor is used.","section":"Introduction and §4"},{"comment":"The abbreviation 'T-SNE' should be 't-SNE', and the text should refer to Figure 3 for LDC and Figure 4 for all datasets consistently.","section":"§4.2 and Figures 3–4"},{"comment":"The paper reports accuracy only; for hate speech detection in particular, class-balanced macro-F1 would be a useful additional metric even if the datasets are stated to be balanced.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The authors are affiliated with L3Cube, and the benchmark datasets and models all originate from the same lab. This makes the overlap test in major comment 3 particularly important and also worth attention from the editor: the paper is partly a self-benchmark. I do not see this as disqualifying, but the revision should make the provenance of data and models transparent and should avoid overclaiming based on within-noise differences. If the authors cannot provide significance tests or overlap analysis, the paper's contribution would be limited to a descriptive benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful empirical benchmark for Marathi embeddings, but read the abstract with suspicion. On the paper's own 5-fold CV (Table 2), MahaBERT beats MahaFT on three tasks (MahaHate-4c, MahaHate-2c, LPC) and loses on three (MahaSent, SHC, LDC). The winning margins are 2.44, 0.48, and 1.56 points; the standard deviations are roughly 0.4–1.1. Most gaps are within noise, and the paper reports no significance tests, confidence intervals, or paired comparisons. The stress-test note is on target. The paper's own treatment of MahaSent—calling the deviation noise—doesn't get applied to its winning margins.\n\nWhat's genuinely good: the SVD compression from 768 to 300 dimensions is a sensible fairness move, and the paper does report variance for the non-contextual embeddings. It also explicitly flags the LDC reversal as stable, which is honest. The three-way comparison (contextual, non-contextual, compressed) across six Marathi tasks gives practitioners a useful reference table.\n\nNow the soft spots, in proportion. First, the headline comparison in Section 5.2 uses uncompressed 768-d BERT against 300-d FastText, undercutting the paper's own argument that dimensionality differences are unfair. Second, the novelty claim is contradicted in the same reference list: Rajput et al. (2021) uses static BERT embeddings for hate speech, yet the paper says \"no prior work has examined the effectiveness of non-contextual BERT embeddings.\" That's an internal inconsistency. Third, the L3Cube provenance overlap—MahaBERT and MahaFT trained on L3Cube corpora while the benchmarks are L3Cube releases—is a real confound, though secondary to the noise problem. Fourth, no code or data are released, so the numbers can't be independently checked.\n\nWho should read this: Marathi or low-resource Indic NLP practitioners who want a rough sense of embedding choices. The tables have value even if the conclusion is weaker than advertised. Who shouldn't: anyone looking for a settled verdict on BERT-vs-FastText.\n\nRecommendation: send to peer review, but require the authors to either run paired significance tests or soften the abstract, and to release code. A claim like \"non-contextual BERT embeddings are competitive with FastText and sometimes better\" would be defensible and still useful.","headline":"Useful Marathi embedding benchmark, but the abstract overstates the case: on the paper's own Table 2, FastText wins as often as BERT and most gaps are within noise.","tokens_in":8978,"tokens_out":2542,"would_cite":false,"duration_ms":22925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Non-contextual BERT embeddings from the first embedding layer outperform FastText on most Marathi classification tasks, at table-lookup cost.","keywords":["non-contextual BERT embeddings","FastText","Marathi","low-resource NLP","word embeddings","text classification","SVD compression","contextual vs non-contextual"],"falsifier":"Compute verbatim and near-duplicate overlap between the MahaSent, MahaHate, and MahaNews examples and the Marathi pretraining corpus described in Joshi (2022a). If a substantial fraction of benchmark sentences appear in pretraining, rerun the comparison on examples confirmed absent from pretraining; the non-contextual BERT-over-FastText ordering would invert or shrink if memorization, not representation quality, was driving it.","tokens_in":7951,"feed_emoji":"📊","tokens_out":7103,"duration_ms":57992,"temperature":0.7,"pith_summary":"This paper asks whether BERT can be used without running the transformer: it takes the token embeddings from BERT's first layer, averages them per sentence, and pits them against FastText vectors on Marathi sentiment, hate speech, and news classification. Its claim is that these non-contextual BERT embeddings generally beat FastText, while full contextual BERT beats both. The reason to care is speed and memory: non-contextual embeddings are a table lookup, so if the claim holds, low-resource pipelines can get BERT-pretrained quality without a GPU forward pass. The paper also finds that compressing BERT's 768 dimensions down to FastText's 300 with SVD erases the advantage, making FastText the better choice among compressed non-contextual options.","feed_headline":"BERT table-lookup beats FastText on Marathi tasks","feed_subtitle":"Using only BERT's first-layer embedding table is cheap and beats FastText on most Marathi tasks, until compression erases the gain.","key_machinery":"The central object is the non-contextual embedding: the output of BERT's first embedding layer, a lookup table that maps each token ID to a vector before any transformer layer adds context. The paper averages those token vectors to form a sentence vector and feeds it to a multinomial logistic regression. To make the comparison fair, it compresses the 768-dimensional BERT vectors to 300 dimensions using singular value decomposition, matching FastText's dimensionality, and reports 5-fold cross-validated accuracy. This extraction-and-compression setup is what lets the paper isolate embedding quality from model depth.","core_discovery":"On the paper's own terms, the discovery is a practical ordering of embeddings for Marathi: with a multinomial logistic regression classifier, contextual MahaBERT and MuRIL embeddings outperform all FastText models on every dataset; among non-contextual embeddings, MahaBERT and MuRIL extracted from the first embedding layer outperform MahaFT and IndicFT on most tasks, with sentiment classification and long-document news as exceptions; and SVD compression from 768 to 300 dimensions systematically degrades non-contextual BERT so that FastText then wins. The paper attributes the non-contextual advantage to MahaBERT's larger embedding table, larger pretraining data, and contextual training objective, all obtained at table-lookup cost.","pith_inferences":["A direct extension the paper leaves implicit: if non-contextual BERT works from a lookup table, the embedding matrix can be precomputed once and served with the same infrastructure as FastText, so a latency and memory benchmark would quantify the practical gain.","The paper does not check whether the benchmark texts overlap the pretraining corpus of the BERT models; a membership test would determine whether the non-contextual BERT advantage survives without data leakage.","The same experimental design transfers to other languages that have a BERT and a FastText trained on the same corpus, which would show whether the finding is about Marathi or about embedding quality.","The high fold variance on the sentiment dataset suggests a paired significance test across folds could separate real exceptions from noise."],"forward_implications":["Marathi NLP pipelines can swap FastText for non-contextual MahaBERT embeddings at the same inference cost and expect better accuracy on sentiment, hate speech, and short-text news classification.","SVD-compressing BERT vectors down to 300 dimensions is not a neutral way to level the playing field: it lowers non-contextual BERT below FastText, so uncompressed embeddings should be used when dimension is not a constraint.","Contextual BERT remains the best representation on every task tested, preserving its role when a full forward pass is affordable.","The FastText advantage on long-document news shows the ranking is task-dependent, not a universal law."],"supporting_citations":[{"why":"Supplies both MahaBERT and MahaFT, the pair trained on the same Marathi corpus that anchors the comparison.","marker":"Joshi, 2022a"},{"why":"Supplies MuRIL, the multilingual BERT model used alongside MahaBERT.","marker":"Khanuja et al., 2021"},{"why":"Supplies IndicFT, the Indic FastText baseline, and details of its training corpus.","marker":"Kakwani et al., 2020"},{"why":"Supplies the MahaSent sentiment dataset used for evaluation.","marker":"Pingle et al., 2023"},{"why":"Supplies the MahaHate hate speech dataset with 2-class and 4-class tasks.","marker":"Patil et al., 2022"},{"why":"Supplies the MahaNews headline, long-document, and long-paragraph classification datasets.","marker":"Mittal et al., 2023"},{"why":"Defines the BERT architecture from which contextual and non-contextual embeddings are extracted.","marker":"Devlin et al., 2018"}],"fun_headline_variants":["Non-contextual BERT beats FastText on most Marathi tasks","Marathi: BERT table lookup beats FastText, until compressed","Cheap BERT embeddings outperform FastText on Marathi","BERT's first layer beats FastText on Marathi","Compression erases BERT's edge over FastText"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the Marathi benchmark texts were not already present in the pretraining corpus of MahaBERT and MuRIL; the paper does not test for this overlap, so if it exists the BERT scores are inflated and the reported ordering could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Non-contextual BERT beats FastText on most Marathi tasks","Marathi: BERT table lookup beats FastText, until compressed","Cheap BERT embeddings outperform FastText on Marathi","BERT's first layer beats FastText on Marathi","Compression erases BERT's edge over FastText"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000975,"raw_usage":{"total_tokens":4103,"prompt_tokens":868,"completion_tokens":3235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":3150}},"tokens_in":484,"tokens_out":3235,"duration_ms":21407,"temperature":1.0,"reasoning_tokens":3150,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:51:47.649520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute verbatim and near-duplicate overlap between the MahaSent, MahaHate, and MahaNews examples and the Marathi pretraining corpus described in Joshi (2022a). If a substantial fraction of benchmark sentences appear in pretraining, rerun the comparison on examples confirmed absent from pretraining; the non-contextual BERT-over-FastText ordering would invert or shrink if memorization, not representation quality, was driving it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MahaHate hate speech dataset with 2-class and 4-class tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MahaNews headline, long-document, and long-paragraph classification datasets."}],"review_version":1}