{"id":"ee20c705-c052-494e-b860-43b982cbb858","arxiv_id":"2608.11049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On the DravidianLangTech 2025 Tamil political sentiment test set, XGBoost scored 0.2835 macro F1 and BERT scored 0.2806, showing the task is far from solved.","lead":"This paper applies XGBoost and BERT to classify Tamil political social media posts into seven sentiment categories, achieving macro F1 scores of about 0.28 on a hidden test set. The low scores underline the difficulty of the task and provide a starting baseline for future low-resource political sentiment analysis.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6 contradicts Table 4: predicted counts are mislabeled/permuted, so the headline F1 values (0.2835/0.2806) are not internally corroborated and need official verification.","rationale":"This paper is a shared-task system description; its main value is as a baseline. The central claim is the two macro-F1 numbers. I checked the internal arithmetic: Table 4's per-class F1 values, macro average (0.2835), accuracy (180/544 = 0.3309), and weighted averages are all consistent with each other, and Table 5 is likewise internally consistent. So if the underlying per-class true-positive counts are correct, the headline numbers are correctly computed. The weak point is the correspondence between the result tables and the actual model predictions. Table 6's predicted counts cannot be reconciled with Table 4's precision/recall values; the counts are a permutation of the Table-4-derived predicted counts but attached to the wrong classes. This is exactly the kind of error that would also occur if the wrong model's confusion matrix or a misaligned table were copied into the paper. Because the test set was unlabeled and the scores came from the shared-task organizers, the reader cannot tell from the manuscript alone whether the headline F1 values are the true official scores or an artifact of table misalignment. The manuscript also contains a related inconsistency (Section 3.3 mentions Logistic Regression while Section 3.4 and the results describe XGBoost), and the Limitation statement calls the performance 'strong' despite F1 ≈ 0.28, which undercuts the paper's own framing as a difficulty baseline. These do not independently falsify the central claim, but they lower confidence in the reporting. A conditional accept requiring release of predictions and official scores plus a corrected Table 6 is the right disposition. I agree with the reader's weakest-assumption identification.","tokens_in":7618,"tokens_out":6935,"duration_ms":58575,"concrete_test":"Obtain the official DravidianLangTech@NAACL 2025 test labels and the exact submitted prediction files for this system (or rerun the stated TF-IDF+XGBClassifier and bert-base-uncased pipelines with the reported hyperparameters on the released splits), then recompute the per-class and macro F1 values. If the XGBoost macro F1 is not 0.2835 or the BERT macro F1 is not 0.2806, or if the correctly labeled predicted counts from the confusion matrix do not match either Table 4 or Table 6, the central claim fails. At minimum, the authors should provide a corrected Table 6 whose predicted column matches the actual model predictions and state which table was used to report the official scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the macro F1 scores from the official DravidianLangTech 2025 test set. The only internal evidence for those scores is Tables 4-6, and those tables cannot all be correct. From Table 4, precision = TP/predicted and recall = TP/support, so predicted count = recall * support / precision. For Opinionated this yields 0.0870 * 46 / 0.1739 ≈ 23 predicted instances, but Table 6 reports 305 predicted Opinionated instances; 305 is actually the derived predicted count for Positive (0.6433 * 171 / 0.3607 ≈ 305). The same permutation appears for other rows: Table 6's Sarcastic 84 is Negative's derived count, Neutral 50 is Sarcastic's, Positive 43 is Substantiated's, Substantiated 15 is None's, Negative 23 is Opinionated's, and None 24 is Neutral's. The set of Table 6 predicted counts is the same multiset as Table 4's predicted counts, but assigned to the wrong labels. Because the headline macro F1 rests on per-class precision/recall, the inconsistency does not by itself prove the F1 numbers are false—Table 4's macro, per-class, accuracy, and weighted averages are mutually consistent—but it means the paper's reporting of the official test result is unreliable as published. Without code, predictions, or official scores, the central comparative claim cannot be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a multiclass sentiment analysis system for political Tamil tweets, developed for the DravidianLangTech 2025 shared task. The authors evaluate two models: an XGBoost classifier with TF-IDF features and a BERT-base-uncased transformer. On the official test set they report macro F1 scores of 0.2835 (XGBoost) and 0.2806 (BERT), concluding that the task is challenging and positioning their results as a baseline for future work.","tokens_in":7871,"tokens_out":3658,"duration_ms":30037,"significance":"If the reported results are accurate, the paper provides a useful, if modest, empirical baseline for political sentiment analysis in a low-resource language (Tamil). The use of standard evaluation metrics and the transparent reporting of BERT hyperparameters are strengths. However, the manuscript contains internal numerical inconsistencies that undermine confidence in the headline F1 scores, and it does not provide code, predictions, or official evaluation materials to support independent verification. The contribution is a system description for a shared task, acceptable in principle for a workshop venue only if the reporting errors are corrected and the results are made verifiable.","major_comments":[{"comment":"Table 6's predicted-instance counts contradict the precision and recall values in Table 4. From Table 4, the predicted count for each class is recall × support / precision, which gives approximately 23 (Opinionated), 50 (Sarcastic), 24 (Neutral), 305 (Positive), 43 (Substantiated), 84 (Negative), and 15 (None). Table 6 lists these same values but assigned to different labels (Opinionated 305, Sarcastic 84, Neutral 50, Positive 43, Substantiated 15, Negative 23, None 24). The text in Section 3.5 explicitly repeats the erroneous assignment when it states that the Opinionated label received 305 predictions. Because the macro F1 values in Table 4 are derived from the same per-class precision and recall values, this inconsistency makes the paper's internal reporting of the official test results unreliable and leaves the headline F1 scores without corroboration from the published tables. The authors must correct Table 6 and the accompanying discussion, or provide the official confusion matrices and predictions.","section":"Section 3.5, Tables 4 and 6"},{"comment":"Section 3.3 states that TF-IDF and BertTokenizer are applied for the 'Logistic Regression and BERT models,' but Section 3.4 describes XGBoost (XGBClassifier) and BERT, and no Logistic Regression results appear anywhere in the paper. This methodological contradiction must be resolved: either the Logistic Regression experiment should be reported, or the text in Section 3.3 should be corrected to refer to XGBoost. As written, it is unclear which models were actually evaluated.","section":"Sections 3.3 and 3.4"},{"comment":"The manuscript provides no code, no data, no XGBoost hyperparameters, and no link to the official evaluation scores. Given the internal inconsistency between Tables 4 and 6, the authors should make their test-set predictions or confusion matrices available as supplementary material so that the claimed macro F1 scores (0.2835 and 0.2806) can be independently checked. Without such materials, the central comparative claim is not verifiable.","section":"Reproducibility and verifiability"}],"minor_comments":[{"comment":"The test set is described as 'unlabeled,' yet Tables 4 and 5 report support counts for each class. Please clarify that gold labels were obtained from the shared task organizers after submission, or describe how the support counts were derived.","section":"Section 3.1"},{"comment":"The text says the results of both approaches are presented in 'Tables 4 and 6,' but the BERT results appear in Table 5, not Table 6. The reference should be to Tables 4 and 5 for the two models, with Table 6 as a separate summary for XGBoost.","section":"Section 3.5"},{"comment":"There is a typo in 'IF-IDF' which should be 'TF-IDF.'","section":"Section 3.4"},{"comment":"The keyword line has a formatting artifact: 'K ˛eywords' displays an unusual accent; this should be cleaned up.","section":"Abstract"},{"comment":"The conclusion contains a duplicated phrase: 'into into seven multiclasses.'","section":"Section 4"},{"comment":"There are several typos in the figure discussion, including 'corret' instead of 'correct,' 'osition' instead of 'position,' and 'Figure 2 and 3' should be 'Figures 2 and 3.'","section":"Section 3.5"},{"comment":"The sentence 'the main contribution of this paper is summarized as follows' should begin with a capital letter after the preceding period.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a shared-task system description with limited technical novelty, which may be acceptable for a workshop venue. The more serious issue is data integrity: the Table 4/Table 6 predicted-count mismatch is exactly the kind of error that suggests the reported statistics were not carefully checked. I believe the authors should be given the opportunity to correct the tables and provide verification materials, hence major revision rather than rejection. I also note a high number of self-citations, several of which seem tangential to the present work; this does not affect my recommendation but may deserve editorial attention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a DravidianLangTech 2025 system description for seven-way political sentiment classification of Tamil X posts. The only genuinely new artifact is a numeric baseline — XGBoost macro F1 0.2835, BERT 0.2806 — on an externally scored test set. If those numbers survive verification, they give future shared-task participants something to beat. That is the paper's whole contribution, and it is modest but real.\n\nWhat it does well: the authors use the organizer-supplied train/dev/test split, evaluate on an unlabeled test set through the official channel, and report per-class precision/recall/F1 plus confusion-matrix commentary. That is the right experimental skeleton for a shared task. The limitation statement, apart from the one overclaim mentioned below, is honest about the tiny dataset and generalization limits.\n\nNow the soft spots, in proportion. The biggest one is internal consistency. Section 3.3 says TF-IDF was used for “Logistic Regression,” but Section 3.4 and all results are XGBoost. That looks like a leftover sentence, but it should not be in a published paper. More seriously, Table 4 and Table 6 contradict each other. Using precision = TP/predicted and recall = TP/support, Table 4 implies Opinionated had about 23 predicted instances (0.0870 * 46 / 0.1739), while Table 6 says 305. The stress-test note is right: the Table 6 predicted counts are the same multiset as Table 4's derived predicted counts, but assigned to the wrong labels (305 actually matches Positive's derived count). So one of the tables is mislabeled or permuted. The headline macro F1 comes from Table 4, which is internally consistent with its own accuracy and weighted averages, so the contradiction does not prove the F1 numbers are false — but it does mean the paper as published cannot be trusted without the official results, the predictions, or the confusion matrices.\n\nOther issues: no code, data, or predictions are released, so nothing is independently reproducible. The limitation statement says the model demonstrated “strong performance” when the F1 scores are low, which is an overclaim given the macro F1 around 0.28. Heavy self-citation is annoying but not a scientific flaw; the cited works are relevant enough.\n\nBottom line: this paper is for the DravidianLangTech community and shared-task track readers who want a numeric reference point. With the internal contradictions fixed and official scores confirmed, it is a citable system description. As it stands, it needs heavy revision before it should appear. I would still send it to peer review rather than desk-reject, because the shared-task evaluation is real and the flaws are fixable — but I would not cite the headline numbers until the authors clean up the tables and release the predictions.\n\nRecommendation: allow peer review, but require the inconsistencies to be resolved and the raw predictions or official leaderboard scores to be included.","headline":"A low-resource shared-task baseline with plausible but internally inconsistent reported numbers; useful only after the tables are fixed and official scores confirmed.","tokens_in":8462,"tokens_out":1662,"would_cite":false,"duration_ms":17999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tamil political tweets hold best model to 0.28 macro F1 on a seven-way sentiment task.","keywords":["multiclass sentiment analysis","political discourse","Tamil","XGBoost","BERT","low-resource languages","social media","macro F1"],"falsifier":"Sum the predicted counts in the published confusion matrices by class and compare them with the #Predicted column of Table 6 (305, 84, 50, 43, 15, 23, 24). If the sums do not match, or if recomputing macro F1 from a consistent matrix changes the 0.2835 and 0.2806 figures, the central result is not reproducible from the paper itself.","tokens_in":7377,"feed_emoji":"🗳️","tokens_out":7813,"duration_ms":69691,"temperature":0.7,"pith_summary":"This paper tries to establish a baseline for multiclass sentiment analysis of political tweets in Tamil, a low-resource language. It frames the task as seven-way classification over Substantiated, Sarcastic, Opinionated, Positive, Negative, Neutral, and None-of-the-above, using a labeled set of social media posts. The authors train two pipelines, TF-IDF with XGBoost and a fine-tuned BERT, and report macro-F1 scores of 0.2835 and 0.2806 on the held-out test set. They read these numbers as evidence that political discourse is hard to classify automatically and as a floor that future work must beat. If the result holds, it gives the community a concrete reference point for political opinion mining in under-resourced Dravidian languages.","feed_headline":"Tamil political tweets hold best model to 0.28 F1","feed_subtitle":"On 544 held-out posts, XGBoost and BERT both land near 0.28 macro F1, a low floor for low-resource political NLP.","key_machinery":"The machinery is a seven-class supervised pipeline built on a 4,352-post Tamil training set, with a 544-post development set and a 544-post unlabeled test set. One arm encodes text with TF-IDF and trains an XGBoost classifier; the other tokenizes with BertTokenizer and fine-tunes bert-base-uncased for five epochs at batch size 32 and learning rate 1e-5. The load-bearing quantity is macro-averaged F1 over all seven classes, which is what ranks the models; because macro averaging weighs rare classes equally with common ones, the classifiers' poor handling of rare classes drives the low scores.","core_discovery":"The paper's central claim is that on the official test set, XGBoost with TF-IDF features reaches a macro F1 of 0.2835 and BERT reaches 0.2806, a near tie in which XGBoost edges ahead. Both models are strongly biased toward the Positive class: Positive receives by far the most correct predictions, while Opinionated, Sarcastic, Substantiated, None-of-the-above, and much of Negative are frequently absorbed into it. The authors present these numbers as demonstrating the difficulty of contextualized political sentiment and as baseline results for further work on this dataset and language.","pith_inferences":["A reader's inference from the two result tables is that the 0.0029 gap between XGBoost and BERT is smaller than the inconsistency between the reported per-class statistics, so the claimed ordering should not be used as evidence until the numbers are reconciled.","A testable extension the paper leaves implicit is hierarchical classification: first coarse positive/negative/neutral, then fine-grained classes, which could reduce the dominant confusion and is directly checkable on the same test set.","The low scores also suggest a data-limited regime, so data augmentation or pseudo-labeling on the unlabeled test posts might improve rare classes more than changing preprocessing."],"forward_implications":["A future system for this task must clear roughly 0.28 macro F1 on the same seven classes to be a meaningful improvement.","The error pattern points to the classes with the smallest training support, None-of-the-above and Substantiated, and to Positive as the dominant confusion target.","Class-balanced training or cost-sensitive losses are the direct next step suggested by the data, not a new architecture.","The near tie between a bag-of-words-plus-boosted-trees model and a fine-tuned transformer implies that architecture is not the main bottleneck on this dataset."],"supporting_citations":[{"why":"Supplies the Tamil political-tweet dataset with train/dev/test splits and the official class labels used for evaluation.","marker":"(Roy et al., 2025)"},{"why":"Provides the XGBClassifier implementation that forms the first model pipeline.","marker":"(Chang et al., 2022)"},{"why":"Frames the task of sentiment analysis over informal political discourse that the paper extends to multiclass Tamil tweets.","marker":"(Mullen and Malouf, 2006)"},{"why":"Demonstrates prior Tamil tweet sentiment analysis and motivates the feature-extraction and preprocessing choices.","marker":"(Ramanathan et al., 2021)"}],"fun_headline_variants":["Political tweet sentiment: XGBoost and BERT tie at 0.28 F1","Both models flop on political tweets: 0.28 F1 ceiling","Tamil political NLP: best model hits just 0.28 F1","Sentiment analysis of political posts struggles at 0.28 F1","XGBoost beats BERT by hair in political sentiment, still weak"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the published tables all come from one consistent evaluation run of the same test predictions; since the paper's own per-class counts do not fully agree, that premise is not fully established.","fun_headline_variants_meta":{"raw":{"variants":["Political tweet sentiment: XGBoost and BERT tie at 0.28 F1","Both models flop on political tweets: 0.28 F1 ceiling","Tamil political NLP: best model hits just 0.28 F1","Sentiment analysis of political posts struggles at 0.28 F1","XGBoost beats BERT by hair in political sentiment, still weak"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000526,"raw_usage":{"total_tokens":2503,"prompt_tokens":869,"completion_tokens":1634,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1530}},"tokens_in":485,"tokens_out":1634,"duration_ms":11875,"temperature":1.0,"reasoning_tokens":1530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:19:40.542408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sum the predicted counts in the published confusion matrices by class and compare them with the #Predicted column of Table 6 (305, 84, 50, 43, 15, 23, 24). If the sums do not match, or if recomputing macro F1 from a consistent matrix changes the 0.2835 and 0.2806 figures, the central result is not reproducible from the paper itself.","supporting_citations":[],"review_version":1}