{"id":"5b8bc7af-1a4b-40da-9a50-748788f228a0","arxiv_id":"2502.02100","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BERTopic with Indic BERT embeddings produces higher topic coherence scores than LDA on Marathi datasets of long, medium, and short documents.","lead":"The authors compared BERTopic and LDA on Marathi text and report that BERTopic with Indic BERT embeddings scores higher on topic coherence across long, medium, and short documents. The result is a practical model recommendation for Marathi NLP teams, though key configuration details are missing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coherence comparison is not interpretable: coherence metric, LDA topic count, and hyperparameters are unreported, so the BERTopic-vs-LDA gap could be a configuration artifact.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the coherence numbers in Tables 1 and 2 are not comparable across the two pipelines because the evaluation setup is undisclosed. This is the single most important threat to the central claim because the claim is entirely about a comparison of scores. If the coherence metric and LDA's topic count were chosen differently, the direction or magnitude of the gap could change. The paper also contains a secondary internal contradiction (the text says MahaBERT-V2 is 'consistently better' but Table 1 shows other models scoring higher), but that concerns a supporting observation, not the headline claim. The concrete test proposed above would settle whether the BERTopic advantage is a genuine property of the methods or a configuration artifact. Since the paper is a short empirical benchmark with no code or data release, the conditional verdict is appropriate; this concern does not change the reader's verdict but reinforces it. The paper is not fatally flawed, but it cannot be evaluated as presented. The test is feasible and would either confirm the headline or expose its fragility.","tokens_in":4847,"tokens_out":4017,"duration_ms":35676,"concrete_test":"Obtain the code or the missing configuration from the authors, or reconstruct it from the paper, and rerun LDA on the same preprocessed corpora with a grid of topic counts (e.g., 5, 10, 20, 50, 100) and multiple random seeds. Compute coherence for both BERTopic and LDA using the same metric (e.g., c_v) and the same number of top words. If no LDA configuration approaches or exceeds BERTopic's reported coherence range, the claim survives; if some configurations do, the headline comparison is not robust and needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Tables 1 and 2 (Section IV), which report topic coherence for BERTopic and LDA across three datasets. The comparison is not auditable because the evaluation configuration is unspecified. The paper never names the coherence metric (c_v, c_npmi, or u_mass); if BERTopic scores are computed with one metric and LDA with another, the numerical gap is meaningless. More importantly, the number of topics for LDA is not reported, whereas BERTopic's topic count is determined automatically by HDBSCAN. Coherence is highly sensitive to the number of topics; if LDA was run at a suboptimal topic count, the apparent advantage of BERTopic could be an artifact of mismatched configuration rather than a property of the methods. The paper also does not state how many top words per topic were used to compute coherence, and it reports single numbers without variance or multiple seeds, so the stability of the gap is unknown. Because the three datasets (LPC, LDC, SHC) are undescribed in size, source, and preprocessing, the result may be dataset-specific. Without these details, the headline claim that BERTopic outperforms LDA for Marathi has not been established as a general finding; it remains an observation about an unspecified setup.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares BERTopic combined with several Marathi/Indic BERT and Sentence-BERT models (MahaSBERT-STS, MahaSBERT, MahaBERT-V2, IndicSBERT, IndicSBERT-STS, MuRIL) against Gensim LDA for topic modeling on three Marathi datasets referred to as LPC, LDC, and SHC. The headline claim is that BERTopic yields substantially higher topic coherence (0.63–0.82) than LDA (0.34–0.55) on all three datasets. The paper also asserts that MahaBERT-V2 performs consistently better than the other BERT models. The evaluation uses topic coherence as the main metric, with a brief discussion of topic diversity, but no topic diversity results are reported.","tokens_in":5047,"tokens_out":2806,"duration_ms":25257,"significance":"If the comparison were fully specified and reproducible, a demonstration that BERTopic with Indic-language embeddings consistently outperforms LDA on a low-resource language like Marathi would be a useful empirical contribution to the growing literature on neural topic modeling for Indic languages. The direction of the reported results is plausible and consistent with findings in other languages, and the use of publicly available L3Cube models is a strength. However, in its current form the evidence is not auditable: the coherence metric, model hyperparameters, topic counts, and dataset characteristics are missing, so the central quantitative claim cannot be independently verified or interpreted as a general property of the methods rather than of a particular unspecified configuration.","major_comments":[{"comment":"The coherence metric is never named. Coherence scores are not comparable across metrics: c_v, c_npmi, and u_mass have different scales and different sensitivities to topic count and corpus statistics. If BERTopic scores were computed with one metric and LDA with another, the numerical gap would be meaningless. Please state the exact metric, the implementation (e.g., Gensim CoherenceModel), the top-word count used, and the reference corpus for computing co-occurrence statistics.","section":"Section IV, Tables 1 and 2"},{"comment":"The LDA topic count k is not reported, while BERTopic's topic count is determined automatically by HDBSCAN. Coherence is highly sensitive to the number of topics, so if LDA was run at a topic count that was not tuned per dataset, the apparent BERTopic advantage could be a configuration artifact. Report k for each dataset, the LDA hyperparameters (passes, chunksize, alpha, eta, random seed), and whether k was selected by a coherence search or fixed a priori.","section":"Section IV, Tables 1 and 2"},{"comment":"The sentence \"In BERTopic, monolingual bert model MahaBERT-V2 performs consistently better than other bert models in the 3 datasets\" is contradicted by the table the authors themselves provide. On the LPC dataset, IndicSBERT and IndicSBERT-STS score 0.72 while MahaBERT-V2 scores 0.71; on the SHC dataset, IndicSBERT-STS scores 0.81, the highest of all models. Please correct either the claim or the analysis, and discuss the actual ordering of model performance across datasets.","section":"Section IV, Table 1 and text"},{"comment":"All reported scores are single numbers with no variance, no number of runs, and no random seeds. The differences among BERTopic variants are small (e.g., 0.70 vs. 0.71 on LPC), and BERTopic's UMAP and HDBSCAN steps are stochastic; a single run does not establish the stability of the ranking. Report results across multiple seeds (or set and document deterministic parameters) and, if possible, provide confidence intervals or standard deviations.","section":"Section IV, Tables 1 and 2"},{"comment":"The three datasets LPC, LDC, and SHC are never described: their sizes, sources, document length distributions, and preprocessing details (beyond removal of stop words, URLs, and non-Devanagari words) are absent. This matters both for assessing the authors' short/medium/long document distinctions and for judging whether the results generalize. Please add a data description table or clear citations and preprocessing details.","section":"Section II and Section IV"}],"minor_comments":[{"comment":"The abstract and methodology mention topic diversity as an evaluation metric, but no topic diversity results appear in Section IV. Either report those numbers or remove the claim.","section":"Abstract and Section III"},{"comment":"\"Maarten Gr.\" should be \"Maarten Grootendorst\".","section":"Section II"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., [5] has a stray \"1.\"; [8] lacks venue and page information; [9] contains a fragment of a URL). Please normalize all entries.","section":"References"},{"comment":"The sentence \"[4] MahaBERT-V2 is outperforming MuRIL in our evaluations, demonstrating superior performance across the tasks\" appears to reference a different study and is not supported by the results in Table 1, where MuRIL scores are comparable to or higher than MahaBERT-V2 on the LDC dataset. Please clarify the source of this claim.","section":"Section IV"},{"comment":"The sentence \"When topics are shown to users, each topic t is typically shown as a list of the M = 5,..., 20 most likely words\" is vague but the paper never states the actual M used for coherence computation. If M is not reported in the main text, this should be part of the evaluation configuration.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on the authors' own L3Cube models and corpora. This is not inherently a problem, but it does mean that an independent replication would need those resources to be clearly identified and accessible. For the journal's standards of reproducibility, the missing evaluation details are the main obstacle. The scope of the paper is quite narrow, but with proper experimental reporting it could be acceptable as an empirical contribution. If the authors respond with the missing configuration details, the most important checks are whether the coherence metric is consistent across both BERTopic and LDA, whether LDA's topic count was tuned fairly, and whether the claim about MahaBERT-V2 is corrected to match their own table."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small empirical comparison that could be useful, but as written the headline number is not auditable. The specific benchmark is new—no one has run BERTopic with six Indic/Marathi embedding models on these datasets—and the direction (BERTopic beats LDA) is plausible. The paper deserves a serious referee only if the authors are pushed to report their setup.\n\nWhat it does well: it is a clean, honest experiment in spirit. Using three Marathi datasets of different document lengths is a good idea, and comparing eight model variants is a concrete contribution to a low-resource language where teams default to LDA. The coherence tables show a consistent gap in favor of BERTopic on all datasets (0.63–0.82 vs 0.34–0.55), and the conclusion is measured.\n\nThe soft spots are serious. The paper never names the coherence metric (c_v, c_npmi, or u_mass). Coherence values are not comparable across methods if the metric or the number of top words changes. The number of topics for LDA is unreported; BERTopic sets it via HDBSCAN. If LDA was run at a suboptimal k, the entire gap could be an artifact. There are no error bars or seeds, and the datasets (LPC, LDC, SHC) are not described in size, source, or preprocessing. The stress-test note is right: the comparison is an observation about an unspecified setup, not a general finding.\n\nThere is also a small internal contradiction. The text says 'MahaBERT-V2 performs consistently better than other bert models in the 3 datasets,' but Table 1 shows on SHC, MahaSBERT-STS (0.79) and IndicSBERT-STS (0.81) outperform MahaBERT-V2 (0.73). That is a careless sentence, and it undermines confidence in the other claims.\n\nThe self-citation to L3Cube models is not a problem here: those are the actual models tested, and MuRIL provides an external baseline. The citation list is thin but appropriate for a short benchmark.\n\nBottom line: if you work on Marathi NLP, the question 'BERTopic vs LDA' is relevant, and this paper gives a directional answer. But do not put this in a publication without the configuration details and variance. For peer review, I would send it to a venue with a strong request for a reproducibility appendix: metric, topic counts, hyperparameters, seeds, dataset descriptions, and code. If those are provided, it is a modest, useful contribution.","headline":"Useful benchmark direction, but the numbers are unauditable as written; send to review only with a demand for full configuration details.","tokens_in":5675,"tokens_out":3144,"would_cite":false,"duration_ms":26933,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BERTopic with Indic BERT embeddings beats LDA for Marathi topics","keywords":["topic modeling","Marathi","BERTopic","Latent Dirichlet Allocation","topic coherence","low-resource languages","BERT embeddings","Indic languages"],"falsifier":"Rerun both pipelines on the same three Marathi datasets with a single specified coherence metric (e.g., c_v) and with LDA's topic count and BERTopic's hyperparameters grid-searched; if LDA's best coherence meets or exceeds BERTopic's, or the gap falls below the numbers reported here, the central claim fails.","tokens_in":4545,"feed_emoji":"📚","tokens_out":5872,"duration_ms":47436,"temperature":0.7,"pith_summary":"This paper compares two families of topic models on Marathi text: classical probabilistic LDA and the embedding-based BERTopic, using six pre-trained BERT models trained on Marathi or Indic languages. It reports that BERTopic with these embeddings produces topics with coherence scores of 0.63–0.82 across three Marathi datasets, while LDA scores 0.34–0.55 on the same datasets. The central claim is that transfer learning from pre-trained BERT models makes neural topic modeling the better practical choice for a low-resource, morphologically rich language like Marathi. If correct, this offers a concrete recipe for topic modeling in Marathi and other Indic languages.","feed_headline":"BERTopic beats LDA for Marathi topic models","feed_subtitle":"Indic-language BERT embeddings lift Marathi topic coherence to 0.63-0.82, above LDA's 0.34-0.55.","key_machinery":"The load-bearing mechanism is BERTopic's pipeline: sentence-transformers turn Marathi sentences into dense embeddings; UMAP reduces dimensionality; HDBSCAN clusters semantically similar documents; and a class-based TF-IDF (c-TF-IDF) scores words within each cluster to form topics. The comparison metric is topic coherence, which measures how related the top words of a topic are. The BERT models supply the semantic representations, and their language-specific versus multilingual training is the variable that distinguishes model performance.","core_discovery":"The paper claims that BERTopic, a neural topic model that clusters sentence embeddings and derives topics with class-based TF-IDF, consistently outperforms LDA on topic coherence for Marathi. On three in-house datasets of long (LDC), medium (LPC), and short (SHC) documents, the best BERTopic configurations reach coherence between 0.71 and 0.82 per dataset, with monolingual MahaBERT-V2 performing best across all three, while LDA's coherence ranges from 0.34 to 0.55. The paper attributes the gap to pre-trained language models capturing semantic relations that LDA's bag-of-words statistics miss.","pith_inferences":["The reported superiority would be on firmer ground if the paper specified which coherence metric (c_v, c_npmi, or u_mass) was used, since absolute scores differ across metrics; the 0.63–0.82 versus 0.34–0.55 gap may not reproduce under a different metric.","With three datasets differing only in document length, the authors leave untested how the BERTopic advantage scales with dataset size, vocabulary, or domain; a natural next experiment is varying topic count and measuring topic diversity, which the paper says it used but never reports.","Because IndicSBERT and MURIL are cross-lingual, the same BERTopic configuration could be applied with minimal modification to Hindi, Tamil, or other Indic languages, making the recipe a testable template beyond Marathi."],"forward_implications":["For Marathi topic modeling, the paper advises choosing BERTopic over LDA when coherence is the goal, with monolingual Marathi BERT models as the first choice.","On short documents (SHC), sentence-BERT models outperform BERT models, while on long documents (LDC) BERT models have the edge, suggesting the optimal embedding model depends on document length.","The success of pre-trained embeddings on Marathi indicates that low-resource languages can inherit effective topic models from transfer learning rather than requiring large topic-specific corpora.","MahaBERT-V2 outperforming the multilingual MURIL on all three datasets suggests that a language-matched monolingual model is worth training even when a strong multilingual model is available."],"supporting_citations":[{"why":"Supplies the BERTopic method (UMAP, HDBSCAN, c-TF-IDF) that the paper applies to Marathi.","marker":"[7]"},{"why":"Provides the Marathi monolingual BERT model (MahaBERT-V2) and corpus that achieve the best BERTopic coherence.","marker":"[4]"},{"why":"Provides the Marathi sentence-BERT models (MahaSBERT) used for short-document BERTopic runs.","marker":"[12]"},{"why":"Provides the cross-lingual Indic sentence-BERT models (IndicSBERT) used as multilingual embeddings.","marker":"[13]"},{"why":"Gives the earlier LDA-based topic model for Marathi (MaTop) that motivates the LDA baseline and its evaluation.","marker":"[10]"},{"why":"Provides the LDA formulation and survey that the paper treats as the standard baseline.","marker":"[6]"}],"fun_headline_variants":["Indic BERT plus BERTopic beats LDA on Marathi topics","Marathi topic models: BERTopic tops LDA with Indic BERT","BERTopic with Indic BERT boosts Marathi coherence to 0.82","Better Marathi topics with BERTopic and Indic BERT","Why BERTopic dethrones LDA for Marathi text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the reported coherence scores for BERTopic and LDA are directly comparable; the paper never states the coherence metric, topic count, hyperparameters, or dataset sizes, so the gap could shift or vanish under matched settings.","fun_headline_variants_meta":{"raw":{"variants":["Indic BERT plus BERTopic beats LDA on Marathi topics","Marathi topic models: BERTopic tops LDA with Indic BERT","BERTopic with Indic BERT boosts Marathi coherence to 0.82","Better Marathi topics with BERTopic and Indic BERT","Why BERTopic dethrones LDA for Marathi text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1147,"prompt_tokens":850,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":201}},"tokens_in":466,"tokens_out":297,"duration_ms":3134,"temperature":1.0,"reasoning_tokens":201,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:20:02.030221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun both pipelines on the same three Marathi datasets with a single specified coherence metric (e.g., c_v) and with LDA's topic count and BERTopic's hyperparameters grid-searched; if LDA's best coherence meets or exceeds BERTopic's, or the gap falls below the numbers reported here, the central claim fails.","supporting_citations":[{"cited_title":"L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi","cited_arxiv_id":"2211.11187","evidence_quote":"Provides the Marathi sentence-BERT models (MahaSBERT) used for short-document BERTopic runs."},{"cited_title":"L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT","cited_arxiv_id":"2304.11434","evidence_quote":"Provides the cross-lingual Indic sentence-BERT models (IndicSBERT) used as multilingual embeddings."},{"cited_title":"Aswani Kumar, Review and Implementation of Topic Modeling in Hindi, https://www.tandfonline.com/doi/epdf/10.1080/08839514.2019.1661576?needAccess=true","cited_arxiv_id":null,"evidence_quote":"Provides the LDA formulation and survey that the paper treats as the standard baseline."}],"review_version":1}