{"id":"6eabdf14-2ffc-44ff-b044-39958e90d039","arxiv_id":"2506.10154","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On the EmoNoBa Bangla emotion dataset, a boosted decision tree achieves 0.7860 macro F1, the best among classical models tested but below transformer-based baselines.","lead":"This paper benchmarks standard machine learning models on a public Bangla emotion dataset and reports that a decision tree with AdaBoost reaches a macro F1 of 0.7860. It adds a low-resource language case study and LIME explanations, but the result is not compared with prior work on equal terms.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 reports the headline 0.7860 as a 'Macro average F1-score,' while Tables 2 and 4 report only 'F1-score' with no averaging specified; Section 4.3 says macro, micro, and weighted were all computed. The 'highest overall F1' claim thus compares incomparable quantities.","rationale":"Reader's REJECT is appropriate. The main conclusion is a ranking among F1 scores, but Section 4.3 states that macro, micro, and weighted averages were all computed, and Table 3 is the only table that says 'Macro average.' Since Tables 2, 4, and 5 use unlabeled 'F1-score,' the comparison underlying Section 5.1 is not well-defined. On the imbalanced distribution of Figure 2, macro and micro F1 can differ substantially, so the 0.7860-vs-0.63 gap could be an artifact of mixing averaging schemes. This is an internal reporting inconsistency, not a dispute with external consensus. I partially agree with the reader's weakest assumption: the unresolved 15.3% multi-label comments are a serious independent flaw, but the metric mismatch is more immediately load-bearing because it invalidates the claimed ranking even if label handling were clean. The concrete check is to compute all scores under one averaging scheme on the same test split. No code is provided, so the check requires reimplementation, which is feasible on the public EmoNoBa dataset. No independent evidence (e.g., released code or machine-checked proofs) offsets this gap. The verdict remains REJECT; I would not downgrade to CONDITIONAL without corrected tables.","tokens_in":7680,"tokens_out":8798,"duration_ms":101780,"concrete_test":"Obtain the EmoNoBa dataset and rerun the exact pipeline from Table 5 with a fixed 80/15/5 split, computing macro, micro, and weighted F1 for every model on the same test set. Then compare like-for-like. Specifically, if Linear SVM's macro F1 (rather than the reported 0.63) is lower than 0.63, or if Decision Tree+AdaBoost's micro F1 differs from its macro 0.7860, the current table is mixing averaging schemes and the Section 5.1 ranking collapses. A minimal version: recompute only Table 2's SVM unigram row as macro F1 and check whether it is still 0.63.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5.1 is a ranking: Decision Tree+AdaBoost (0.7860) beats Linear SVM (0.63), KNN (0.57), RF (0.57), and BiLSTM (0.3869). But Table 3 labels its headline value 'Macro average F1-score,' whereas Tables 2, 4, and 5 label their values simply 'F1-score.' Given Section 4.3 states all three averaging schemes were computed, the reader cannot tell whether the comparison figures are macro, micro, or weighted. This is not cosmetic: on the imbalanced six-class distribution shown in Figure 2, macro and micro F1 can differ substantially, so reporting SVM as 0.63 and DT+AdaBoost as 0.7860 may simply mix two different metrics. If the 0.63 SVM entry is micro/weighted, the ranking 'highest overall F1' is unsupported even before considering the 15.3% multi-label instances in Section 3.3, whose treatment is also undocumented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical comparison of classical machine-learning models (Linear SVM, KNN, Random Forest, Decision Tree with and without AdaBoost) and a BiLSTM for fine-grained emotion classification on the EmoNoBa Bangla social-media dataset (22,698 comments, six emotion labels). The authors also apply TF-IDF with n-grams, PCA, and LIME explanations for the best model. The headline claim is that Decision Tree with AdaBoost achieves the highest overall F1-score of 0.7860, outperforming the other models and the lexical-feature baseline of Islam et al. (0.4281) while remaining below BanglaBERT-based results (0.7330 and 0.8273). The manuscript is written as a standard benchmark study, with a clear methodology diagram, multiple evaluation metrics, and a short limitations section.","tokens_in":7903,"tokens_out":5204,"duration_ms":60042,"significance":"If the reported results are robust, the paper would provide a useful classical machine-learning baseline for Bangla emotion detection, demonstrating that an explainable boosting ensemble can outperform lexical-feature baselines and approach transformer performance on EmoNoBa. Strengths include the use of a public dataset, comparison with prior published results, inclusion of LIME explanations, and explicit acknowledgment of data imbalance and scope limitations. However, the contribution is currently undercut by unresolved reporting issues: the F1-score averaging scheme is inconsistent across tables, the treatment of multi-label instances is undocumented, no variance estimates or cross-validation are reported, and hyperparameters are largely absent. These issues directly affect the validity of the central 'highest overall F1' claim, so the paper's significance cannot be fully assessed in its present form.","major_comments":[{"comment":"The headline comparison mixes different F1 averaging schemes. Section 4.3 states that macro, micro, and weighted averages were all computed, and Table 3 explicitly labels the Decision Tree results as 'Macro average F1-score' (0.7860 and 0.7799). Tables 2, 4, and 5 report only 'F1-score' without specifying the averaging. Given the strong class imbalance visible in Figure 2 (Joy and Sadness much more frequent than Fear and Surprise), macro and micro/weighted F1 can differ substantially. The central claim in Section 5.1 that 'The Decision Tree, when combined with AdaBoost, recorded the highest overall F1-score of 0.7860' is therefore unsupported unless every compared value in Tables 2 and 5 is also macro-averaged. The authors must state the averaging scheme for every reported F1 value and, ideally, report all three variants.","section":"§4.3, Table 3, Table 5"},{"comment":"The dataset objective in Section 3.2 states that the aim is to identify all emotions expressed in a text, and Section 3.3 reports that 15.3% of entries express more than one emotion. Nevertheless, the classifiers described in Section 4.2 are standard single-label models, and the manuscript nowhere explains how multi-label instances are resolved before training and evaluation. Possible resolutions—dropping such instances, assigning a single dominant label, or creating multiple single-label training examples—would change the training distribution, test labels, and every F1 value in Tables 2–5. This is a load-bearing omission: without a stated and justified multi-label handling rule, the reported scores are not well-defined and cannot be compared with the EmoNoBa baselines in Table 6.","section":"§3.3, §4.2"},{"comment":"No error bars, cross-validation, or significance tests are reported. The evaluation rests on a single stratified split (80% train, 15% test, 5% validation) with no random seed, and the claimed improvement from AdaBoost is 0.7799 to 0.7860—a difference of 0.0061 that is likely within the noise of a single split on a dataset of this size. The statement in Section 5.1 that boosting 'highlight[s] the effectiveness' of the approach needs support from repeated splits or k-fold evaluation with standard deviations, and ideally a paired significance test. Without this, the superiority of the AdaBoost model over the plain Decision Tree is not established.","section":"§3.4, §5.1"},{"comment":"The comparison with prior EmoNoBa results is not yet interpretable. Table 6 lists F1 scores from Islam et al. (0.4281), Chakma et al. (0.7330), and Kabir et al. (0.8273), but the manuscript does not state whether those studies report macro, micro, or weighted F1, nor does it describe differences in data preprocessing (especially multi-label handling) and evaluation splits. Merely placing numbers side by side in Table 6 does not establish comparability. In addition, the citation markers in the text are inconsistent with the reference list: the text refers to 'Kabir et al. (2024) [18]' and 'Chakma et al. (2023) [17]', but reference [17] is Sadhu et al. and reference [18] is Chakma et al. The authors should verify all cited values and protocols.","section":"Table 6, Section 5.1"}],"minor_comments":[{"comment":"There is a typo in the first paragraph: 'Additionaly' should be 'Additionally'.","section":"§5"},{"comment":"The two confusion matrices are introduced as 'a sample' for the best-performing model, but no analysis is provided; the captions do not state whether the matrices are normalized, and the reader cannot tell which classes or splits they refer to.","section":"Figures 11 and 12"},{"comment":"Reference [14] is malformed: it begins 'A comparative study Shah, Kanish, et al.' and appears to merge two different works. The authors should correct all reference entries to their complete bibliographic details.","section":"§4.2, References"},{"comment":"The experimental setup lists the hardware and Python versions but omits all model hyperparameters: SVM regularization C, number of neighbors K, Random Forest tree count, PCA component count, AdaBoost learning rate and estimator count, and the BiLSTM architecture (hidden size, layers, dropout, embedding dimension, training epochs). These details are essential for reproducibility.","section":"§4.4 and Tables 2–5"},{"comment":"The LIME discussion (Figure 10) describes predictions such as 'Not Fear' and 'Not Sadness' without explaining how LIME scores were aggregated or whether the features identified are consistent across instances. A quantitative evaluation of the explanations (e.g., faithfulness or agreement with human intuition) would strengthen the XAI contribution.","section":"§5, LIME explanation"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The underlying experiment may be salvageable, but the reported numbers currently lack the internal consistency and statistical support needed for a benchmark claim. I would be willing to accept a revised version that provides exact F1 averaging for all models, a clear multi-label resolution rule, repeated-split results, and full hyperparameter and split details. If the authors cannot supply these details, rejection would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a straightforward application of standard classifiers and LIME to the public EmoNoBa dataset. The paper is clearly organized and honestly concedes that BanglaBERT does better. What it does not do is support its central ranking claim: Table 3 labels the DT+AdaBoost score as 'Macro average F1-score,' while Tables 2 and 4 just say 'F1-score,' even though Section 4.3 says macro, micro, and weighted were all computed. On an imbalanced dataset, mixing schemes can change rankings. That is not cosmetic; it is the main result.\n\nWhat is actually new is modest: a classical-ML benchmark plus LIME explanations for six Bangla emotion categories. That is a useful demonstration for a low-resource language, and the writing is accessible. The comparison with existing EmoNoBa results in Table 6 is a good idea, though the citation handling is sloppy—the text mentions Kabir et al. (2024) but the reference list has no such entry, and the numbering seems off.\n\nOther soft spots: 15.3% of the entries are multi-label per Section 3.3, but the classifiers are single-label and there is no description of how those instances were flattened. Hyperparameters and code are absent, so the results are not reproducible as reported. The t-SNE and word cloud are decorative.\n\nThe metric issue alone makes the headline unsupported. If the authors clarify the averaging and the multi-label handling, the paper could be a serviceable benchmark, but as posted it is not reliable. I would not cite it as is, and I would not send it to a top venue. That said, the work is sincere and the low-resource context gives it a niche value; a serious referee could push the authors to fix the reporting. I would probably accept it for peer review at a workshop or a low-resource NLP venue, but with the expectation of heavy revision.","headline":"A cleanly written but flawed benchmark: the headline F1 comparison mixes averaging schemes, and the multi-label handling is unstated.","tokens_in":8442,"tokens_out":2746,"would_cite":false,"duration_ms":29379,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On 22,698 Bangla social media comments, a decision tree boosted with AdaBoost reaches a macro F1-score of 0.7860 for six-emotion classification, the best among the classical models tested and within reach of a much larger pretrained…","keywords":["emotion detection","Bangla text classification","low-resource language","sentiment analysis","TF-IDF","AdaBoost","LIME","EmoNoBa dataset"],"falsifier":"Re-run the Decision Tree plus AdaBoost setup on EmoNoBa with an explicit multi-label handling rule, or on the subset of single-emotion comments only, and compare the macro F1 to 0.7860; a large drop would show the headline score depends on the unresolved multi-label transformation.","tokens_in":7461,"feed_emoji":"😊","tokens_out":7374,"duration_ms":79089,"temperature":0.7,"pith_summary":"This paper tries to establish that a classical, interpretable machine-learning pipeline can do serious emotion detection in Bangla social media text. Using 22,698 comments from the EmoNoBa dataset labeled with six emotions, the authors compare TF-IDF n-gram models (Linear SVM, KNN, Random Forest), a decision tree with and without AdaBoost, and a BiLSTM. Their best model, a decision tree boosted with AdaBoost, reaches a macro F1-score of 0.7860, well above the 0.4281 lexical-feature baseline and close to but below the 0.8273 reported for BanglaBERT-large. They also show that PCA consistently hurts performance and that LIME can expose which words drive the AdaBoost decisions. If these results hold, they suggest low-cost, transparent models remain a viable route for emotion analysis in low-resource languages.","feed_headline":"Decision tree plus AdaBoost tops Bangla emotion tests at F1 0.79","feed_subtitle":"On 22,698 EmoNoBa comments, the interpretable model beats all classical baselines and trails only BanglaBERT.","key_machinery":"The central mechanism is a TF-IDF plus n-gram feature representation feeding a decision-tree classifier that is boosted with AdaBoost: TF-IDF encodes word-importance weights for unigrams, bigrams, and trigrams; the decision tree supplies an interpretable rule structure; and AdaBoost reweights hard examples so successive weak trees focus on misclassified comments. LIME, a local surrogate explainer, then attributes each prediction to the words that most influenced it. The paper's negative result is equally part of the machinery: PCA applied to the TF-IDF matrix compresses the feature space but consistently reduces F1, indicating that for this task the discarded dimensions carried signal.","core_discovery":"The paper's central claim is that on the EmoNoBa corpus of 22,698 noisy Bangla comments, a decision tree classifier combined with AdaBoost on TF-IDF unigram features gives the best overall emotion-classification result among the models tested, with a macro F1-score of 0.7860. This beats the other classical models (Linear SVM 0.63, KNN 0.57, Random Forest 0.57), the unboosted decision tree (0.7799), and the deep BiLSTM (0.3869), and it substantially outperforms the 0.4281 lexical-feature benchmark from the dataset paper. The same experiments show that adding PCA lowers F1 scores across all three n-gram classifiers, and that unigram features outperform bigrams and trigrams. The authors present LIME explanations of the boosted decision tree as evidence that the model's predictions are interpretable, and they position these results as a step toward benchmark emotion detection for Bangla.","pith_inferences":["A direct extension would be to re-run the same pipeline on a multi-label formulation of EmoNoBa, using all emotion labels present in the 15.3% multi-emotion comments; the paper's single-label F1 of 0.7860 would likely shift once those overlaps are counted explicitly.","The comparison suggests a testable hypothesis: a compact distilled transformer or a feature-based model with confidence thresholds could close the 0.0413 gap to BanglaBERT while keeping LIME-style explanations, a direction not explored here.","Since LIME explanations are shown only as examples, a quantitative next step is to have Bangla-speaking annotators judge whether the highlighted words are actually emotion-bearing; that would test the interpretability claim directly."],"forward_implications":["For Bangla emotion detection, classical feature-based models remain a reasonable default: the boosted decision tree beats the lexical baseline and all other non-transformer models in this study.","Unigram TF-IDF appears sufficient; bigram and trigram features add complexity without improving F1, so future low-resource pipelines can start with unigrams only.","PCA should not be assumed to help in this setting; the paper's results show it repeatedly lowers F1, so dimensionality reduction needs its own validation on each text-classification task.","Because the best model is a decision tree, its per-prediction reasoning can be shown with LIME word-level explanations, which is harder with large pretrained transformers.","A simple interpretable model can come within roughly four F1 points of the reported BanglaBERT-large score, suggesting a cost-performance tradeoff for low-resource deployment."],"supporting_citations":[{"why":"Introduces the EmoNoBa dataset of 22,698 Bangla comments and provides the lexical-feature baseline F1-score of 0.4281 that the paper's models must beat.","marker":"[8]"},{"why":"Reports the BanglaBERT-large F1-score of 0.8273 that the paper uses as the upper comparison point for its best model.","marker":"[17]"},{"why":"Reports the BanglaBERT-Large with random token drop F1-score of 0.7330 used as the second transformer comparison.","marker":"[18]"},{"why":"Earlier work on Bangla textual emotion analysis; the paper extends this line from SVM-based classification to a six-emotion benchmark.","marker":"[9]"},{"why":"Provides the LSTM architecture that motivates the BiLSTM model tested in the paper.","marker":"[10]"},{"why":"Supplies the Word2Vec embedding approach used for BiLSTM feature extraction.","marker":"[12]"},{"why":"Provides the rationale for choosing decision trees as the interpretable learner that AdaBoost then boosts.","marker":"[16]"}],"fun_headline_variants":["AdaBoost+decision tree hits 0.79 F1 on Bangla emotions","Bangla comment emotions: boosted trees beat SVM, KNN, BiLSTM","LIME explains AdaBoost's 0.79 F1 in Bangla emotion analysis","On 22K Bangla comments, boosted tree F1 0.79, LIME inside","Interpretable Bangla emotion model: AdaBoost tree F1 0.79"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing assumption is that every comment has exactly one of the six emotion labels, even though the paper reports that 15.3% of comments express more than one emotion and never describes how those multi-label comments were assigned to a single class before training and scoring.","fun_headline_variants_meta":{"raw":{"variants":["AdaBoost+decision tree hits 0.79 F1 on Bangla emotions","Bangla comment emotions: boosted trees beat SVM, KNN, BiLSTM","LIME explains AdaBoost's 0.79 F1 in Bangla emotion analysis","On 22K Bangla comments, boosted tree F1 0.79, LIME inside","Interpretable Bangla emotion model: AdaBoost tree F1 0.79"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001052,"raw_usage":{"total_tokens":4396,"prompt_tokens":898,"completion_tokens":3498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":3387}},"tokens_in":514,"tokens_out":3498,"duration_ms":30257,"temperature":1.0,"reasoning_tokens":3387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:32:47.444232+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Decision Tree plus AdaBoost setup on EmoNoBa with an explicit multi-label handling rule, or on the subset of single-emotion comments only, and compare the macro F1 to 0.7860; a large drop would show the headline score depends on the unresolved multi-label transformation.","supporting_citations":[{"cited_title":"Emonoba: A dataset for analyzing fine-grained emotions on noisy bangla texts","cited_arxiv_id":null,"evidence_quote":"Introduces the EmoNoBa dataset of 22,698 Bangla comments and provides the lexical-feature baseline F1-score of 0.4281 that the paper's models must beat."},{"cited_title":"LowResource at BLP-2023 Task 2: Leveraging BanglaBert for Low Resource Sentiment Analysis of Bangla Language","cited_arxiv_id":"2311.12735","evidence_quote":"Reports the BanglaBERT-Large with random token drop F1-score of 0.7330 used as the second transformer comparison."},{"cited_title":"Comparison of Classical Machine Learning Approaches on Bangla Textual Emotion Analysis","cited_arxiv_id":"1907.07826","evidence_quote":"Earlier work on Bangla textual emotion analysis; the paper extends this line from SVM-based classification to a six-emotion benchmark."},{"cited_title":"Long short-term memory","cited_arxiv_id":null,"evidence_quote":"Provides the LSTM architecture that motivates the BiLSTM model tested in the paper."},{"cited_title":"A study of fasttext word embedding effects in document classification in bangla language","cited_arxiv_id":null,"evidence_quote":"Supplies the Word2Vec embedding approach used for BiLSTM feature extraction."},{"cited_title":"Detecting ambiguities in requirements documents using inspections","cited_arxiv_id":null,"evidence_quote":"Provides the rationale for choosing decision trees as the interpretable learner that AdaBoost then boosts."}],"review_version":1}