{"id":"b71edf6d-cd7f-4cd6-ab4b-7108b5b7d583","arxiv_id":"2505.18995","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A LoRA-tuned Filipino LLM is compared to CalamanCy and found weaker, but the reported numbers and statistical test are internally contradictory.","lead":"This paper fine-tunes SeaLLM-7B with LoRA to create a Filipino language model, FiLLM, and compares it with the existing CalamanCy model. The authors report that CalamanCy beats FiLLM on part-of-speech tagging and dependency parsing, but the paper's own statistics are inconsistent and no code or model is released.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's statistical evidence is internally impossible: a mean difference of 11 with t=0.12 and p=0.03 cannot occur; recomputing from the three task F1 scores gives t≈1.62, p≈0.25, so the central comparison has no valid significance test.","rationale":"I read the paper in good faith. It attempts to fine-tune SeaLLM-7B with LoRA for Filipino tasks and compare it to CalamanCy. For the comparison to be credible, reported scores must be consistent and the significance test valid. The weakest load-bearing point is not merely the unexplained CalamanCy configuration, but the arithmetic of Table 3: the reported t and p are mathematically incompatible with the reported means. This is a correctness defect, not a disagreement with consensus. It means the paper's central comparative claim has no valid statistical support. Other inconsistencies in the results sections compound the problem, and no code or artifacts are provided to disambiguate. I agree partially with the reader's focus on identical evaluation conditions; that is a valid concern, but the internal statistical impossibility is more fundamental because it invalidates the only significance test even if conditions were identical. Under the paper's own numbers, no conclusion 'we reject the null hypothesis' can be drawn. Therefore the verdict of REJECT remains appropriate; release of reproducible evaluation could turn this into a weak but acceptable negative data point, but as written the central claim is not supported.","tokens_in":4989,"tokens_out":4393,"duration_ms":38976,"concrete_test":"Recompute the paired t-test from the actual per-task F1 scores shown in Figure 3 (FiLLM: 89, 89, 73; CalamanCy: 90, 97, 97). The differences are 1, 8, 24; the mean is 11 and the sample SD is 11.79, so t = 11/(11.79/sqrt(3)) ≈ 1.62, df=2, p ≈ 0.25. If the published t=0.12 and p=0.03 are not reproduced, Table 3 and the claim of statistical significance are invalid. To test the F1 numbers themselves, request the evaluation script and per-item predictions for both systems; rerun FiLLM and CalamanCy on identical TLUnified-NER and MergedUD test splits using the official scoring tools (e.g., seqeval for NER/POS and the UD evaluation script for dependency parsing), with identical preprocessing and tokenization, and compare the recomputed F1 scores to Table 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical assertion is that FiLLM is usable but weaker than CalamanCy on NER, POS, and dependency parsing. The only statistical support for that comparison is the paired t-test in Table 3. That table is arithmetically impossible. The three task F1 scores listed in Figure 3/Table 2 are: FiLLM (89, 89, 73) and CalamanCy (90, 97, 97), giving means 83.67 and 94.67 and mean difference 11. The paired t-statistic for these three differences (1, 8, 24) is about 1.62 with df=2, two-tailed p≈0.25, not t=0.12 and p=0.03. With t=0.12, p would be about 0.91; with p=0.03 and df=4, t would need to be near 2.78 or larger. No reasonable correction changes this, and Table 3 also reports n=5 despite only three task scores. Additionally, the body text and conclusion disagree with Table 2: POS recall is given as 0.93 in the text but 0.90 in the table, and the conclusion reports 86/84 for POS and 71 for dependency parsing instead of 89/90 and 73/74. The comparison with CalamanCy also has no stated configuration, test split details, preprocessing, or metric implementation. The internally contradictory statistics alone mean the central comparative claim is unsupported as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FiLLM, a Filipino-language large language model obtained by LoRA fine-tuning of SeaLLM-7B 2.5, and evaluates it on Named Entity Recognition, Part-of-Speech tagging, Dependency Parsing, and Text Summarization using Filipino datasets. The authors report F1 scores of 0.89, 0.89, and 0.73 for FiLLM on NER, POS, and dependency parsing, respectively, compare these against CalamanCy, and use a paired t-test to claim that the performance difference is statistically significant. The manuscript's central claim is that FiLLM is a usable but weaker alternative to CalamanCy for Filipino NLP tasks.","tokens_in":5416,"tokens_out":2288,"duration_ms":22509,"significance":"If the reported results were reliable, the paper would provide a useful data point for low-resource Filipino NLP by demonstrating that memory-efficient LoRA fine-tuning of a multilingual SEA language model yields competitive NER and POS performance, and by benchmarking against an established Filipino pipeline. The authors deserve credit for assembling several Filipino datasets, for applying LoRA to a 7B-parameter model, and for attempting a statistical comparison against an existing system. However, the central empirical claim is not supported as reported: the statistical test in Table 3 is arithmetically impossible, the sample-size and task-count values do not match, and the precision/recall/F1 numbers differ across the table, text, and conclusion. These internal inconsistencies make the current evidence inadequate to support the paper's conclusions, so the potential significance cannot be realized without a complete re-analysis and re-reporting of the experiments.","major_comments":[{"comment":"The paired t-test results in Table 3 are internally impossible. With the reported means 83.67 and 94.67, the mean difference is 11. Across the three F1 scores reported in Figure 3/Table 2, the paired differences are 1, 8, and 24, yielding a paired t-statistic of approximately 1.62 with 2 degrees of freedom and a two-tailed p-value of about 0.25. The reported t-statistic of 0.12 with p = 0.03 cannot occur for any consistent dataset; t = 0.12 would give p ≈ 0.91, and p = 0.03 with df = 4 would require t ≈ 2.78. Since the conclusion \"we reject the null hypothesis\" follows from the p-value, the test as presented is self-contradictory and does not support the claimed significant difference.","section":"Table 3 and §2.5"},{"comment":"Table 3 reports 5 observations for both FiLLM and CalamanCy, but the paper only presents three task-level F1 scores for each model (NER, POS, dependency parsing). No fourth or fifth observation is identified anywhere. The degrees of freedom (4) and the critical value (2.776) are consistent with n = 5, but no actual data with n = 5 is described, so the statistical analysis is not reproducible from the manuscript.","section":"Table 3"},{"comment":"The reported metric values are not self-consistent across the paper. Table 2 gives FiLLM POS recall as 0.90, but the text in §3 states a Recall of 0.93 for POS; the conclusion reports POS precision 86% and recall 84%, and dependency parsing precision/recall of 71%, whereas Table 2 and the text report 0.73/0.74 and 0.73. The NER precision also changes from 0.86 in Table 2 to 86% in the conclusion, which is consistent numerically, but the POS and dependency parsing numbers are not. Because the paper never identifies which numbers are authoritative, the central results cannot be verified.","section":"§3, Table 2, and Conclusion"},{"comment":"The comparison against CalamanCy is not adequately specified. The paper does not state which version of CalamanCy was used, how it was configured, whether it was evaluated on the exact same 80-20 test splits as FiLLM, which preprocessing steps were applied, or how the F1 metrics were computed for each system. Without identical evaluation conditions, the relative F1 scores in Figure 3 have no clear interpretation, and the central claim that CalamanCy outperforms FiLLM is not properly supported.","section":"§2.2 and §3 (Figure 3 comparison)"}],"minor_comments":[{"comment":"The text says \"This formula represents the t-statistic for a paired t-test\" but the formula itself does not appear in the manuscript; please include the actual equation with proper notation.","section":"§2.5"},{"comment":"Table 1 lists datasets for NER, dependency parsing, POS, and summarization, but Figure 1's architecture description additionally mentions the NewsPH-NLI dataset, which is not listed in Table 1; the dataset list should be reconciled.","section":"Table 1 and Figure 1"},{"comment":"There are numerous spelling and formatting inconsistencies, including \"Calamancy\" vs. \"CalamanCy\", \"Name Entity Recogition\" in Table 1, \"DEPPOSSUM\" as a model label in Table 2, and inconsistent capitalization throughout.","section":"Various"},{"comment":"Figure 2 is not described in enough detail for the reader to interpret the summarization results; the axes are labeled but no numerical results, sample sizes, or evaluation protocol are given in the text.","section":"Figure 2"},{"comment":"Several references are incomplete or inconsistently formatted, including missing venue information for [5] and truncated entries in [6]; full bibliographic details should be supplied.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript has the structure of a student project report and appears to have been submitted without a careful internal consistency check. The statistical analysis is not merely incomplete but arithmetically impossible, and the numeric results shift between the table, the text, and the conclusion. These are load-bearing flaws that cannot be fixed by local revision; the experiments would need to be rerun and fully re-reported, including the exact evaluation setup for both models. Even then, the comparison would need to be placed in the context of existing Filipino NLP benchmarks to establish novelty. I would not recommend resubmission to a research venue without a substantially more rigorous and reproducible experimental design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper's headline comparison between FiLLM and CalamanCy is unsupported by its own numbers. The paired t-test in Table 3 cannot be right. With means 83.67 vs 94.67, n=5, and the listed variances, t=0.12 and p=0.03 are arithmetically incompatible. Recomputing from the three task F1 scores (differences 1, 8, 24) gives t≈1.62, p≈0.25. The paper reports n=5 for three tasks, and the conclusion's precision/recall numbers contradict Table 2. This isn't a minor slip; the central comparative claim rests on a statistic that is impossible as reported.\n\nWhat the paper does well is worth saying. The authors chose a relevant base model (SeaLLM-7B) and used standard Filipino datasets, and they were honest enough to state up front that CalamanCy beats FiLLM on most tasks. That honesty is rare and should be credited. The summarization analysis using compression rate and keyword overlap is a reasonable idea, though underdeveloped. The references cover the relevant Filipino NLP literature.\n\nThe soft spots are otherwise extensive. No evaluation code, model checkpoint, or test-split details are provided, so the comparison with CalamanCy is not reproducible. The text changes POS recall from 0.93 to 0.90, and the conclusion's 86/84 and 71 do not match the table's 89/90 and 73. The t-test interpretation is also backward: t=0.12 with a critical value of 2.776 would mean failing to reject the null, not rejecting it. These are load-bearing inconsistencies, not formatting gripes.\n\nWho is this for? A reader hunting for a quick negative data point about LoRA-tuned SeaLLM on Filipino tasks might get a hint, but they cannot trust any specific number. The paper needs a rewrite with corrected statistics, a released artifact, and a fuller description of the evaluation setup before it is useful to anyone.\n\nMy recommendation: reject. This does not deserve referee time in its current state. If the authors fix the arithmetic, release their model and code, and redo the comparison with a proper test, it could become a modest but acceptable workshop paper. Right now it is a cautionary example of why you check the statistics before believing the conclusion.","headline":"The paper's central statistical evidence is internally impossible; the honest comparison with CalamanCy is a good instinct but the execution is too sloppy to trust.","tokens_in":5850,"tokens_out":1654,"would_cite":false,"duration_ms":16184,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FiLLM, a LoRA-tuned SeaLLM-7B 2.5 model, reports F1 0.89 on Filipino NER and POS tagging, 0.73 on dependency parsing, and trails CalamanCy on all three tasks.","keywords":["Filipino NLP","Low-Rank Adaptation","SeaLLM","CalamanCy","Named Entity Recognition","Part-of-Speech Tagging","Dependency Parsing","Text Summarization"],"falsifier":"Run FiLLM and CalamanCy on the same held-out split of the merged UD and TLUnified-NER data with identical tokenization and metric code, and check whether the reported F1 gaps (0.89 versus 0.97 on POS, 0.73 versus 0.97 on parsing) reproduce; a smaller or reversed gap would show the difference came from evaluation setup rather than model capability.","tokens_in":4795,"feed_emoji":"🇵🇭","tokens_out":10645,"duration_ms":85525,"temperature":0.7,"pith_summary":"This paper sets out to show that a Filipino-optimized large language model can be built cheaply by applying Low-Rank Adaptation (LoRA) to the SeaLLM-7B 2.5 base, and to measure whether that model is competitive with CalamanCy, an existing Filipino NLP pipeline. The authors report F1 scores of 0.89 for named entity recognition, 0.89 for part-of-speech tagging, and 0.73 for dependency parsing, against CalamanCy scores of 0.90, 0.97, and 0.97 on the same tasks. The central empirical assertion is therefore that the LoRA-tuned model is usable for entity and tagging work but materially weaker on syntax. The paper also claims a statistically significant difference between the two models under a paired t-test (p = 0.03), and presents text summarization results that show keyword overlap staying high as compression rate rises. If the comparison held up, it would tell Filipino NLP practitioners that a memory-efficient LoRA adapter does not yet close the gap to a purpose-built pipeline.","feed_headline":"FiLLM trails CalamanCy across Filipino tasks","feed_subtitle":"LoRA-tuned SeaLLM-7B scores 0.89 F1 on NER and POS, 0.73 F1 on parsing.","key_machinery":"Low-Rank Adaptation (LoRA), a fine-tuning scheme that keeps the pretrained SeaLLM-7B 2.5 weights frozen and learns two low-rank matrices, LoRA A of shape $D \\times R$ and LoRA B of shape $R \\times D$, whose product is added to the frozen weight updates. This is the mechanism that lets the paper fine-tune a seven-billion-parameter model with limited memory while adapting it to Filipino tasks. The comparison object is CalamanCy, an existing Filipino NLP pipeline, and the evaluation machinery is a paired t-test over the reported F1-style task scores.","core_discovery":"FiLLM, built by LoRA fine-tuning SeaLLM-7B 2.5 on Filipino datasets, achieves average F1 scores of 0.89 for named entity recognition, 0.89 for part-of-speech tagging, and 0.73 for dependency parsing. On the same evaluation, CalamanCy scores 0.90, 0.97, and 0.97 respectively. The authors conclude that CalamanCy outperforms FiLLM, especially on syntactic tasks, and that the difference is statistically significant at the 0.05 level with p = 0.03. They also report that FiLLM's text summarization retains key information even at high compression rates, tying summarization quality to the balance between compression and keyword retention.","pith_inferences":["If the reported numbers are taken at face value, the natural next experiment is to compare the two models sentence-by-sentence on the same held-out split; the dependency-parsing gap suggests FiLLM's LoRA adapter has not learned long-range syntactic structure.","The compression-rate and keyword-overlap correlation is only a proxy for summarization quality; a stricter test would score summaries against references with metrics such as ROUGE or BERTScore, which the paper does not report.","A direct replication would make the comparison airtight: fix a single held-out split, state the CalamanCy configuration, and share the evaluation script; the task-level F1 gaps could then be verified independently."],"forward_implications":["FiLLM can serve as a usable Filipino NER and POS tagger at 0.89 F1 while keeping training memory low through LoRA.","For dependency parsing, FiLLM is not ready to replace CalamanCy, which scores 0.97 against FiLLM's 0.73.","If the paired t-test result holds, the gap between FiLLM and CalamanCy is too large to be dismissed as chance, at least in the paper's evaluation setting.","FiLLM's summarization behavior suggests it can compress Filipino text while retaining a substantial share of keywords, making it a candidate for extractive-style condensation.","The overall result implies that base-model choice and LoRA adaptation alone do not guarantee parity with an established low-resource-language pipeline."],"supporting_citations":[{"why":"Defines the SeaLLM-7B 2.5 base model that FiLLM is built from; the central claim depends on this base.","marker":"[7]"},{"why":"Provides the TLUnified-NER dataset used to evaluate named entity recognition.","marker":"[4]"},{"why":"Supplies the Ugnayan and TRG treebank data used for part-of-speech tagging and dependency parsing.","marker":"[1]"},{"why":"Provides the Filipino hate speech dataset used in the NER evaluation data mix.","marker":"[10]"},{"why":"Supplies the Asian Language Treebank used for text summarization evaluation.","marker":"[8]"},{"why":"Cited as the source for the LoRA fine-tuning approach and the paired t-test procedure.","marker":"[5]"}],"fun_headline_variants":["FiLLM lags CalamanCy on Filipino syntax: 0.73 vs 0.97","CalamanCy dominates Filipino POS tagging over LoRA FiLLM","Filipino NER close, but CalamanCy tops FiLLM in syntax","LoRA fine-tune FiLLM trails CalamanCy in dependency parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes FiLLM and CalamanCy were evaluated under identical conditions, on the same test splits, with the same preprocessing and the same metric implementation, but the paper gives no CalamanCy configuration or evaluation code.","fun_headline_variants_meta":{"raw":{"variants":["FiLLM lags CalamanCy on Filipino syntax: 0.73 vs 0.97","CalamanCy dominates Filipino POS tagging over LoRA FiLLM","Filipino NER close, but CalamanCy tops FiLLM in syntax","LoRA fine-tune FiLLM trails CalamanCy in dependency parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000783,"raw_usage":{"total_tokens":3426,"prompt_tokens":883,"completion_tokens":2543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":499,"tokens_out":2543,"duration_ms":18150,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:21:31.561597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FiLLM and CalamanCy on the same held-out split of the merged UD and TLUnified-NER data with identical tokenization and metric code, and check whether the reported F1 gaps (0.89 versus 0.97 on POS, 0.73 versus 0.97 on parsing) reproduce; a smaller or reversed gap would show the difference came from evaluation setup rather than model capability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SeaLLM-7B 2.5 base model that FiLLM is built from; the central claim depends on this base."},{"cited_title":"These results underscore FiLLM’s reliability in fundamental NLP tasks, making it a valuable tool for text processing applications","cited_arxiv_id":null,"evidence_quote":"Provides the TLUnified-NER dataset used to evaluate named entity recognition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Ugnayan and TRG treebank data used for part-of-speech tagging and dependency parsing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Filipino hate speech dataset used in the NER evaluation data mix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Asian Language Treebank used for text summarization evaluation."},{"cited_title":"Without their guidance and aid during the creation of this research, it would not have been finished","cited_arxiv_id":null,"evidence_quote":"Cited as the source for the LoRA fine-tuning approach and the paired t-test procedure."}],"review_version":1}