{"id":"89b318c5-973a-458b-a89e-f9367a4cc7fe","arxiv_id":"2501.07020","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An open-source Vietnamese social media normalizer adds multitask NSW detection and dictionary lookup, reporting small F1 gains that are partly contradicted by its own table.","lead":"ViSoLex is an open-source toolkit that looks up non-standard Vietnamese words and normalizes social media sentences into standard text. It combines a dictionary, weakly supervised training, and a web interface, with the main research contribution being a multitask model that detects and normalizes informal words at once.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's ViSoBERT p=0.0 Accuracy row contradicts the claimed 'consistent improvement,' so the sole quantitative evidence for the multitask advantage is not self-consistent.","rationale":"The paper's contribution is an open-source repository plus a multitask extension of prior work, and the quantitative case for the extension rests entirely on Table 1. That table contains a direct numerical contradiction: ViSoBERT multitask Accuracy at p=0.0 is lower than single-task Accuracy, yet the Improvement row reports an increase and the prose claims consistent improvements. This is not a matter of theoretical disagreement or external baselines; it is an internal consistency failure in the only evidence offered for the central claim. Because the repository is open source, the check is straightforward: rerun the published configurations and compare. If the table is corrected and the F1 gains persist, the central claim may survive; if the sign error signals broader misreporting, the claim is unsupported. I therefore retain the reader's CONDITIONAL verdict: the resource itself is real and potentially valuable, but the evaluation must be corrected and re-verified before the multitask benefit can be accepted. I only partially align with the reader's weakest assumption: the synthetic diacritic-removal protocol is a genuine external-validity concern, but the internally inconsistent table is more immediately load-bearing because it undermines confidence in every reported number.","tokens_in":55,"tokens_out":7506,"duration_ms":413240,"concrete_test":"Recompute the ViSoBERT p=0.0 Accuracy entry from the released repository: run the provided single-task and multitask configurations on the bundled test split (or inspect the saved evaluation outputs) and compare the resulting accuracy values to Table 1. If the true multitask accuracy is 95.08 or any value below 95.42, the improvement sign in Table 1 is wrong, and the Section 4/6 claims of consistent improvement must be corrected; the multitask-benefit claim then requires re-examination. If the true value is above 95.42, the table contains a transcription error, and the F1 rows should be re-verified the same way before being cited as evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Section 3.2.1's assertion that the multitask approach 'enhances efficiency and performance,' supported quantitatively by Table 1. The most load-bearing problem is that Table 1 cannot be read as printed. For ViSoBERT at p=0.0, the Accuracy row shows Single task = 95.42 and Multitask = 95.08, which is a decrease of 0.34 percentage points, yet the Improvement row prints ↑0.34. Section 4 then states that 'both models see slight improvements' and Section 6 claims 'consistent improvements in F1-score and accuracy across multitask settings.' These statements are directly contradicted by the table's own numbers. Since Table 1 is the only reported quantitative comparison between single-task and multitask normalizers, this sign/direction error in one of the six improvement rows means the numerical basis for the central claim is not internally reliable. It may be a typographical slip, but the paper provides no raw predictions, error bars, or external check that would let a reader determine whether the F1 improvements (for example, the 3.74-point gain for ViSoBERT at p=1) are similarly affected. The evaluation protocol is also narrow, varying only diacritic removal, and the strongest gain appears exactly at p=1, where normalization largely reduces to diacritic restoration; but the immediate blocker is that the printed evidence is self-inconsistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ViSoLex is an open-source repository for Vietnamese social media lexical normalization, providing two services: NSW Lookup and Lexical Normalization. The lexical normalizer is trained with multitask learning (NSW detection plus normalization) under a weak supervision framework with a Rule Attention Network, building on the authors' prior single-task framework. The paper reports an evaluation comparing single-task and multitask versions of BARTpho and ViSoBERT under two diacritic-removal ratios (p=0 and p=1), using F1-score, Integrity Score, and Accuracy. The central claim is that multitask learning consistently improves normalization performance, especially for ViSoBERT at p=1 with a 3.74-point F1 gain.","tokens_in":6089,"tokens_out":1684,"duration_ms":16553,"significance":"If the claims hold, ViSoLex would be a useful open-source resource for Vietnamese NLP, particularly for downstream tasks like sentiment analysis and hate speech detection, where social-media noise is common. The repository's modular design and weak supervision approach address a genuine scarcity of labeled Vietnamese data. However, the paper's significance as a research contribution is currently limited: the evaluation is narrow (synthetic diacritic removal only), the quantitative evidence is internally inconsistent, and no comparison is made to existing normalization systems or external benchmarks.","major_comments":[{"comment":"Table 1 contains a sign error in the ViSoBERT Accuracy row for p=0.0: single-task accuracy is 95.42% and multitask accuracy is 95.08%, a decrease of 0.34 percentage points, yet the Improvement row prints ↑0.34. This directly contradicts the text in Section 4 that \"both models see slight improvements\" and Section 6 that ViSoLex \"achieves consistent improvements in F1-score and accuracy across multitask settings.\" Since Table 1 is the only reported quantitative comparison, this inconsistency undermines the internal reliability of the central claim and must be corrected and re-verified, ideally with raw prediction outputs or a reproducible script.","section":"Table 1, Section 4"},{"comment":"No confidence intervals, significance tests, dataset sizes, or counts of non-standard words (NSWs) are reported. Improvements of 0.08–0.38 percentage points in Accuracy for most conditions may be within normal evaluation variance, especially without any measure of variability across runs or test-set partitions. The paper should report the number of sentences/tokens in the test set, the number of NSW instances, and ideally multiple seeded runs with confidence intervals, so readers can assess whether the claimed multitask advantage is statistically distinguishable from noise.","section":"Section 4"},{"comment":"The evaluation protocol only perturbs a clean corpus by removing diacritics with probability p. Real Vietnamese social media noise includes abbreviations, phonetic spellings, typos, misspellings, and mixed scripts, which this synthetic corruption does not capture. The largest reported gain (ViSoBERT F1 +3.74 at p=1) is essentially a diacritic-restoration scenario, and thus may not transfer to natural NSWs. The authors should evaluate on a natural NSW benchmark such as ViLexNorm or on a held-out set of authentic social media text, or at least provide a qualitative error analysis showing which types of NSWs are handled by the multitask model.","section":"Section 4, Section 6"}],"minor_comments":[{"comment":"Typo: \"with diacritics removal in traning and development dataset\" should read \"training\"; also \"Table 1 demonstrate\" should be \"Table 1 demonstrates.\"","section":"Section 4"},{"comment":"Equation (1) introduces loss-balancing weights α and β, but the paper never reports the values used in the experiments or any sensitivity analysis; please state the chosen hyperparameters and whether they were tuned.","section":"Section 3.2.1"},{"comment":"The contribution of the Rule Attention Network is not isolated experimentally; the reported multitask gains conflate the effect of multitask learning with the effect of weak-supervision rules. An ablation removing the RAN would clarify which component drives the improvement.","section":"Section 3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of a system demonstration/resource announcement than a full research paper. The strongest contribution is the open-source repository, but the evidence for its superiority over the single-task baseline is not yet reliable. Please ensure the authors verify every number in Table 1 and provide reproducibility details; otherwise the paper risks overclaiming. I would also encourage the authors to compare against existing Vietnamese normalization systems or at least report results on the ViLexNorm benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ViSoLex is a genuinely useful open-source resource for Vietnamese social media normalization, but the paper's central quantitative claim is not currently reliable. Table 1 has a direction error that the authors' own prose contradicts: for ViSoBERT at p=0.0, Accuracy drops from 95.42 to 95.08, yet the Improvement row prints ↑0.34. Section 4 then says 'both models see slight improvements' and Section 6 claims 'consistent improvements in F1-score and accuracy across multitask settings.' You can't read the table as printed and also believe those sentences. This may be a typographical slip, but the table is the only reported comparison between single-task and multitask normalizers, so the numerical basis for the main advantage claim is internally inconsistent.\n\nWhat is actually new here is modest but real: the authors extend their own weak supervision framework (Nguyen et al. 2024b) with multitask learning, an NSW dictionary lookup, and a web UI, and they ship it all open-source with a clean modular design. That is an engineering contribution, not a new mechanism. For Vietnamese NLP, which is under-resourced, having a reusable normalization tool with training scripts and a customizable dictionary is valuable. Credit is due for releasing code, data pipelines, and a demonstration interface.\n\nThe soft spots beyond the table error are the usual ones for system papers: no confidence intervals or significance tests, no dataset sizes or NSW counts, and an evaluation that only perturbs a clean corpus by removing diacritics with probability p. That synthetic corruption is not real social media noise, which includes abbreviations, phonetic spellings, typos, and mixed scripts. So even if the multitask gain at p=1 were real, it might not transfer to actual user-generated text. The authors also compare only against their own prior model; external baselines like MoNoise or other Vietnamese normalizers are absent. Given that ViLexNorm and other datasets exist, a deeper comparison would have strengthened the paper.\n\nThe weaknesses are correctable. The tool itself likely works, and the multitask approach may well help; but as printed, the evidence for that is not self-consistent. This paper deserves a serious peer review, not a desk reject, because the resource is timely and reusable. The reviewer should ask the authors to fix the table, report variance or at least dataset details, broaden the evaluation to natural noise, and temper the 'consistent improvements' language. After that, it would be a solid system contribution.","headline":"A useful open-source resource undermined by a self-inconsistent evaluation table — the multitask advantage claim needs fixing before the paper is publishable.","tokens_in":6579,"tokens_out":2108,"would_cite":false,"duration_ms":19716,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding NSW detection as a second task improves Vietnamese lexical normalization, most sharply when diacritics are stripped.","keywords":["lexical normalization","Vietnamese social media","non-standard words","multitask learning","weak supervision","NSW detection","open-source NLP"],"falsifier":"Run the multitask and single-task normalizers on a held-out set of raw, unmodified Vietnamese social media comments (not diacritic-stripped) and compare F1; if the multitask model's F1 is not higher on that natural data, the paper's central claim does not generalize beyond the synthetic corruption.","tokens_in":5629,"feed_emoji":"🛠️","tokens_out":3797,"duration_ms":33487,"temperature":0.7,"pith_summary":"ViSoLex is an open-source system for normalizing non-standard Vietnamese social media text. The paper's central claim is that training the normalizer to simultaneously detect non-standard words (NSWs) improves normalization accuracy, with the clearest gain on heavily degraded text: for ViSoBERT, F1 rises from 72.19% to 75.93% when all diacritics are removed. The system also offers a dictionary lookup service, with unknown NSWs outsourced to GPT-4o and then added to the dictionary. A sympathetic reader would care because informal Vietnamese spelling breaks downstream tools, and a reproducible, extendable normalizer can serve sentiment analysis, hate speech detection, and translation.","feed_headline":"Detecting non-standard words boosts Vietnamese normalization","feed_subtitle":"ViSoBERT's F1 jumps from 72.19% to 75.93% on diacritic-free Vietnamese social media text.","key_machinery":"The central object is a multitask lexical normalizer built on a shared encoder with two task heads: a token-classification head for NSW detection and a sequence-generation head for normalization. A Rule Attention Network acts as a teacher, weighting weak supervision rules derived from an NSW dictionary and regular expressions, and the total loss is $L_{\\mathrm{total}} = \\alpha L_{\\mathrm{norm}} + \\beta L_{\\mathrm{NSW}}$. The weak-supervision loop follows the ASTRA-style self-training idea, letting the model learn from unlabeled data and adapt to the diverse, evolving NSW patterns in social media.","core_discovery":"The discovery, on the paper's own terms, is that multitask learning—coupling NSW detection with lexical normalization in a single weakly supervised model—yields better normalization than normalization alone. The supporting evidence is Table 1: for ViSoBERT, F1 improves from 72.19% to 75.93% at full diacritic removal and from 75.79% to 77.22% on unperturbed text, while BARTpho sees smaller gains of 0.34% and 0.25%. The authors interpret this as evidence that detecting what is non-standard helps the model decide how to standardize it, especially when the input is visually degraded.","pith_inferences":["The evaluation only synthesizes noise by dropping diacritics; real social media noise also includes phonetic spellings, abbreviations, typographic errors, and code-switching. Testing on naturally noisy data would clarify whether the +3.74 point gain transfers.","Because the largest gain appears in ViSoBERT, a model pretrained specifically on Vietnamese social media, the multitask benefit may depend heavily on the pretraining domain; a general-domain encoder may not show the same jump.","A minimally testable extension is to train the same multitask architecture on other low-resource languages where detection labels are easier to obtain than normalized-word labels, and check whether normalization F1 rises similarly.","The GPT-4o expansion loop creates an active-learning feedback cycle: each user query with an unseen NSW proposes a new dictionary entry, and curator approval is the only quality gate, which could be studied systematically."],"forward_implications":["If the multitask gain is real, future versions of ViSoLex can treat NSW detection as a default auxiliary task rather than an optional extra.","The dictionary lookup with a GPT-4o fallback means the repository can grow its coverage without manual entry for every new slang term.","The weakly supervised, modular design implies researchers can retrain the system with their own labeled files, unlabeled data, and a custom NSW dictionary for other languages or domains.","Normalized Vietnamese output should improve downstream tasks such as sentiment analysis and hate speech detection, where non-standard words typically degrade performance."],"supporting_citations":[{"why":"The prior weakly supervised lexical normalization framework that ViSoLex extends with multitask learning; its metrics and comparison are the baseline.","marker":"Nguyen et al. (2024b)"},{"why":"ASTRA, the self-training-with-weak-supervision approach that the paper adapts into the Rule Attention Network and teacher-student setup.","marker":"Karamanolakis et al. (2021)"},{"why":"ViSoBERT, the social-media-pretrained Vietnamese language model used in the multitask normalizer and the model showing the largest F1 gain.","marker":"Nguyen et al. (2023)"},{"why":"BARTpho, the other pretrained sequence-to-sequence Vietnamese model evaluated in the multitask setting.","marker":"Tran et al. (2022)"},{"why":"ViLexNorm, the first Vietnamese lexical normalization corpus, which provides the underlying labeled data for training and evaluation.","marker":"Nguyen et al. (2024c)"}],"fun_headline_variants":["ViSoLex open-source: NSW detection lifts normalization F1 to 75.93%","Multitask learning boosts Vietnamese text normalization F1 to 75.93%","Detecting non-standard words improves Vietnamese social media normalization","ViSoLex: weakly supervised model pairs detection with normalization","How NSW detection helps Vietnamese normalization: F1 reaches 75.93%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that randomly removing diacritics from a clean corpus produces text similar enough to real Vietnamese social media slang that F1 gains on this synthetic test will hold on natural posts.","fun_headline_variants_meta":{"raw":{"variants":["ViSoLex open-source: NSW detection lifts normalization F1 to 75.93%","Multitask learning boosts Vietnamese text normalization F1 to 75.93%","Detecting non-standard words improves Vietnamese social media normalization","ViSoLex: weakly supervised model pairs detection with normalization","How NSW detection helps Vietnamese normalization: F1 reaches 75.93%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001154,"raw_usage":{"total_tokens":4736,"prompt_tokens":851,"completion_tokens":3885,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":3789}},"tokens_in":467,"tokens_out":3885,"duration_ms":28471,"temperature":1.0,"reasoning_tokens":3789,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:48:35.744752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the multitask and single-task normalizers on a held-out set of raw, unmodified Vietnamese social media comments (not diacritic-stripped) and compare F1; if the multitask model's F1 is not higher on that natural data, the paper's central claim does not generalize beyond the synthetic corruption.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ASTRA, the self-training-with-weak-supervision approach that the paper adapts into the Rule Attention Network and teacher-student setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ViSoBERT, the social-media-pretrained Vietnamese language model used in the multitask normalizer and the model showing the largest F1 gain."}],"review_version":1}