{"id":"73c5f86b-9cb3-43eb-a112-fc07970a4f6a","arxiv_id":"2501.05260","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A weighted ensemble of TF-IDF and BERT features reaches 82.04% accuracy on Marathi plagiarism detection, outperforming BERT-only (80.64%).","lead":"This paper tests a hybrid plagiarism detection system for Marathi text, combining BERT sentence embeddings with TF-IDF vectors. The authors report a small accuracy gain from the hybrid over BERT alone, and release a machine-translated Marathi dataset for future work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 1.4-point accuracy gain over BERT-only is undermined because the ensemble weights and model combination were selected on the same test set that produced the headline accuracy.","rationale":"The most load-bearing condition for the central claim is not just dataset fidelity but the validity of the reported comparison. Even if the translated Marathi dataset perfectly preserved labels, the headline 82.04% versus 80.64% comparison would still be uninterpretable because the ensemble was tuned on the test set. The text in Sections 3.4 and 4.1 admits iterative refinement of model combinations, weights, and dimensions; Table 4 reports results on the single 20% split. Because the combined system has more free parameters (which classifiers, their weights, and the ensemble weights) than the BERT-only system, selecting the best configuration on the test set can manufacture a margin that does not generalize. The reader's label-preservation concern is real but concerns the dataset's external validity; the test-set tuning concern is more immediate because it threatens the internal validity of the central claim. A nested re-evaluation using an untouched holdout would settle it. Since the authors provide code and data, this is a feasible conditional requirement rather than a fatal flaw. I therefore leave the verdict at CONDITIONAL; no change is needed to the reader's verdict, but the stated reason should emphasize the test-set tuning protocol as the primary blocker.","tokens_in":817,"tokens_out":885,"duration_ms":60422,"concrete_test":"Hold out the 20% test split and do not use it for any model choice. On the remaining 80%, use nested 5-fold cross-validation to tune all hyperparameters, choose classifier combinations and weights, and select WBERT/WTF-IDF. Freeze the winning configuration, retrain on the full 80%, and evaluate once on the held-out 20%. Do the same for an equivalent BERT-only system (WBERT=1 with no TF-IDF branch). If the frozen ensemble does not beat the frozen BERT-only baseline on the untouched test set by a margin outside a bootstrap 95% confidence interval, the 82.04% versus 80.64% result is a test-set selection artifact rather than evidence for the hybrid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the weighted ensemble reaches 82.04% versus 80.64% for BERT-only (Section 4.2, Table 4). This claim requires the comparison to be a fair out-of-sample test. The paper describes no validation split: Section 3.4 says model combinations and weights were \"iteratively refined to achieve optimal performance\" and Section 4.1 reports the winning configuration as the one that \"achieved optimal results\" on the data. Since the only held-out portion mentioned is the 20% used for testing, the reported configuration appears to have been selected on the same test set whose accuracy is then quoted. The combined system has extra tunable degrees of freedom (model combination, classifier weights, WBERT/WTF-IDF) relative to the BERT-only baseline, so maximizing on a fixed test split inflates its expected accuracy. A single 80/20 split with no nested validation and no error bars cannot distinguish a genuine 1.4-point improvement from a selection artifact. The label-preservation concern is real but bears on external validity; the test-set tuning concern directly threatens the internal comparison, so it is the more load-bearing of the two.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a Marathi plagiarism/paraphrase detection system that combines TF-IDF vectors and MahaSBERT-STS BERT embeddings in a weighted ensemble of classifiers. The authors construct a Marathi dataset by machine-translating the MIT Plagiarism Detection Dataset (an SNLI-derived corpus), compare two translation models, and evaluate several classifiers plus the proposed ensemble. The headline result is 82.04% accuracy for the ensemble with WBERT = 0.6 versus 80.64% for BERT-only, with an additional comparison on a translated version of MRPC used as validation.","tokens_in":11315,"tokens_out":4973,"duration_ms":48447,"significance":"If the result were established under a proper out-of-sample evaluation, the contribution would be modest but useful for low-resource Marathi NLP: it shows that a hybrid of classical TF-IDF features and modern sentence embeddings can be assembled with publicly available tools, and it releases code and a translated corpus. The paper is reproducible in principle: the code and dataset are public, the models are named, and the hyperparameters are given in Table 7. However, the central numerical claim currently rests on a test-set selection procedure and a single split, so the significance of the reported improvement cannot be assessed from the present evidence.","major_comments":[{"comment":"","section":"Section 3.4 and Section 4.1 (with Section 4.2, Table 4)"},{"comment":"","section":"Section 3.1 and Section 3.2 (with Tables 2-3)"},{"comment":"","section":"Section 4.2, Table 4"}],"minor_comments":[{"comment":"","section":"Table 2 caption"},{"comment":"","section":"References"},{"comment":"","section":"Figure 3"},{"comment":"","section":"Section 4.2"},{"comment":"","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a reasonable application-oriented submission, and the public release of code and a translated Marathi corpus is a useful contribution. However, the evaluation protocol needs substantial revision before the headline result can be trusted. The test-set tuning concern is not merely a presentation issue; it affects the validity of the main comparison. I do not see the problems as unfixable, so I am recommending major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is a useful resource contribution wrapped around a central accuracy claim that does not hold up. The new Marathi translation of the MIT plagiarism detection dataset, with code released and a first application of MahaSBERT-STS to the task, is genuinely of interest to the low-resource NLP crowd. The authors also do some things right: they compare two translation models, evaluate on a translated MRPC set, and include an honest limitations section.\n\nThe problem is that the headline 82.04% vs 80.64% gain is very likely a selection artifact. Section 3.4 says the model combinations and weights were \"iteratively refined to achieve optimal performance,\" and Section 4.1 reports the winning configuration as the one that \"achieved optimal results\" — both on the 20% test split. With no validation set, no nested CV, and no error bars, the extra degrees of freedom in the ensemble (which classifiers, per-classifier weights, WBERT/WTF-IDF) mean the reported peak at WBERT=0.6 is fitted, not predicted. A single 80/20 split cannot support a 1.4-point claim under those conditions. This is load-bearing, not a minor quibble: the paper's main contribution, the hybrid ensemble recipe, is not actually demonstrated.\n\nThe second soft spot is the translated data. BERTScore and TransQuest tell you translation quality, not whether the paraphrase relation survived translation. Systematic translator errors would add label noise to both training and testing, and the same translator is used for the MRPC validation set, so that evaluation shares the noise. The authors acknowledge the lack of standardized Marathi datasets, but they don't validate label preservation at all.\n\nWhat is genuinely new: the dataset itself and the exploration of MahaSBERT-STS for this task. The ensemble of TF-IDF and BERT is well known in English (the authors cite Arabi and Akbari 2022 and Agarwal et al. 2018), so the novelty is narrow but real for Marathi. If the dataset is cleaned and re-evaluated with proper held-out tuning, it could be a useful benchmark. As is, I'd treat the 82% number with suspicion.\n\nWho this is for: people working on Marathi NLP who need a starting dataset, and anyone teaching evaluation pitfalls. It deserves a serious referee — the flaws are addressable and the resource is worth vetting — but the paper should not be accepted without a reworked evaluation.\n\nRecommendation: send to peer review, but expect major revision: proper validation split, error bars or significance testing, and some check on label preservation.","headline":"Useful Marathi dataset resource, but the headline accuracy gain is very likely a test-set tuning artifact and the label preservation of the translation is unvalidated.","tokens_in":11676,"tokens_out":2054,"would_cite":false,"duration_ms":19708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"A weighted ensemble of TF-IDF vectors and BERT embeddings achieves 82.04% accuracy on Marathi plagiarism detection, outperforming BERT-only (80.64%) and TF-IDF-only (58.68%) models.","keywords":["Marathi plagiarism detection","low-resource language","BERT embeddings","TF-IDF","weighted ensemble","paraphrase detection","machine translation","semantic similarity"],"falsifier":"Have native Marathi speakers label a random sample of 1,000 pairs from the translated dataset, then compare the ensemble's predictions to those labels; if accuracy is below 82% on that human-validated subset, the reported result is inflated by translation label noise.","tokens_in":10714,"feed_emoji":"📝","tokens_out":6913,"duration_ms":53798,"temperature":0.7,"pith_summary":"This paper claims that for Marathi, a low-resource language, a weighted ensemble combining TF-IDF vectors and BERT sentence embeddings detects plagiarism more accurately than either representation alone. The proposed system reaches 82.04% accuracy on a machine-translated Marathi dataset, versus 80.64% for BERT-only embeddings and 58.68% for TF-IDF only. This matters because most existing Marathi plagiarism detection relies on similarity heuristics or English-centric models, which miss paraphrase structure in under-resourced languages. The paper also contributes a translated Marathi paraphrase corpus built from the MIT Plagiarism Detection Dataset.","feed_headline":"Hybrid TF-IDF + BERT ensemble hits 82% on Marathi plagiarism","feed_subtitle":"Combining statistical and semantic features beats BERT-only models for a low-resource language.","key_machinery":"The weighted voting ensemble is the load-bearing mechanism. Each classifier outputs a plagiarism probability; the BERT-set probability is the weighted average of XGBoost (weight 0.7) and SVC (weight 0.3) predictions, and the TF-IDF-set probability is the weighted average of Logistic Regression (weight 0.1) and LightGBM (weight 0.9) predictions. The final probability is P = 0.6*PBERT + 0.4*PTF-IDF, and a pair is flagged as plagiarized when P > 0.5. The pair-wise representation is created by element-wise subtraction of the reference embedding from the input embedding, which forces the classifier to learn relationship features rather than absolute text content.","core_discovery":"The paper's central discovery is that the combination of statistical and semantic text representations improves plagiarism detection in Marathi. Using element-wise subtraction of MahaSBERT-STS embeddings (768 dimensions) and TF-IDF vectors (400 dimensions) for each reference-input pair, the authors train four classifiers: XGBoost and SVC on BERT features, Logistic Regression and LightGBM on TF-IDF features. A weighted vote with WBERT=0.6 and WTF-IDF=0.4 yields 82.04% accuracy, 85.32% recall, and an F1 of 82.69%, outperforming every single model and representation. The result is presented as evidence that hybrid representations compensate for the weak semantic coverage of BERT models in low-resource languages.","pith_inferences":["We infer that the 1.4-point accuracy gap between the ensemble and BERT-only sits within the range that machine-translation label noise could produce; a human-validated test set is needed to confirm the hybrid's advantage.","We infer that element-wise subtraction may underuse the embeddings; concatenating difference and product features, or using attention-based pair modeling, could increase the gap further.","We infer that the approach extends naturally to other low-resource Indian languages (e.g., Hindi, Gujarati) that have SBERT variants, provided a labeled paraphrase corpus is translated with similar care."],"forward_implications":["If accurate, the 82.04% result shows that TF-IDF's statistical signal complements BERT's semantic signal for Marathi, so hybrid representations should be preferred in low-resource plagiarism detection.","The translated MIT Plagiarism Detection dataset becomes a reusable benchmark for Marathi paraphrase and plagiarism research, enabling direct comparison of future methods.","The same weighted ensemble configuration (0.6 BERT, 0.4 TF-IDF) is a candidate starting point for other low-resource languages with a sentence-BERT model.","The system's high recall (85.32%) suggests it is better at catching true plagiarized pairs than BERT alone, which matters for screening large text collections."],"supporting_citations":[{"why":"Provides the BERT architecture and pretrained embeddings that the semantic representation is built on.","marker":"Devlin et al., 2018"},{"why":"Defines Sentence-BERT, the siamese architecture adapted into MahaSBERT-STS for sentence embeddings.","marker":"Reimers and Gurevych, 2019"},{"why":"Supplies MahaSBERT-STS, the Marathi sentence-embedding model that generates the 768-dimensional BERT features.","marker":"Joshi et al., 2022"},{"why":"Defines TF-IDF weighting used to build the statistical feature vectors.","marker":"Salton and Buckley, 1987"},{"why":"Provides the SNLI corpus from which the MIT Plagiarism Detection Dataset's pairs and labels are derived.","marker":"Bowman et al., 2015"},{"why":"Introduces mBART, the pretrained multilingual model behind the English-to-Marathi translator selected for the dataset.","marker":"Liu et al., 2020"},{"why":"Supplies BERTScore, used to evaluate translation quality and choose between candidate translation models.","marker":"Zhang* et al., 2020"},{"why":"Supplies TransQuest, the translation-quality estimator used alongside BERTScore.","marker":"Ranasinghe et al., 2020b,a"},{"why":"Founds the ensemble-methods theory that the weighted voting system applies.","marker":"Dietterich, 2000"}],"fun_headline_variants":["82% Marathi plagiarism detection with hybrid TF-IDF + BERT","Weighted TF-IDF + BERT ensemble beats BERT alone for Marathi","Marathi plagiarism: hybrid TF-IDF-BERT ensemble hits 82%","Ensemble of TF-IDF and BERT boosts Marathi plagiarism detection","Hybrid features lift Marathi plagiarism detection to 82%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The machine-translated Marathi dataset preserves the original English plagiarism labels, meaning translation errors do not silently change whether a test pair should be considered plagiarized.","fun_headline_variants_meta":{"raw":{"variants":["82% Marathi plagiarism detection with hybrid TF-IDF + BERT","Weighted TF-IDF + BERT ensemble beats BERT alone for Marathi","Marathi plagiarism: hybrid TF-IDF-BERT ensemble hits 82%","Ensemble of TF-IDF and BERT boosts Marathi plagiarism detection","Hybrid features lift Marathi plagiarism detection to 82%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1184,"prompt_tokens":870,"completion_tokens":314,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":217}},"tokens_in":486,"tokens_out":314,"duration_ms":3518,"temperature":1.0,"reasoning_tokens":217,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:12:29.974828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have native Marathi speakers label a random sample of 1,000 pairs from the translated dataset, then compare the ensemble's predictions to those labels; if accuracy is below 82% on that human-validated subset, the reported result is inflated by translation label noise.","supporting_citations":[],"review_version":1}