{"id":"a85a6a69-8966-4d71-b7da-1fcc6f4337c0","arxiv_id":"2411.13407","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A joint model of XLM-R embeddings and a CNN classifier reaches 82.78% F1 on Vietnamese NLI, but the claimed consistent superiority over fine-tuned baselines is not supported by the paper's own results.","lead":"The authors tested combining four pretrained language models (PhoBERT, XLM-R, mBERT, InfoXLM) with CNN and BiLSTM classifiers for Vietnamese natural language inference, reporting a best F1 of 82.78% on the ViNLI benchmark. The paper matters because it evaluates a simple, resource-efficient recipe for a low-resource language, but the reported gains over fine-tuning are small and inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed consistent superiority is contradicted by the paper's own Table 3 and Section 5; the reported gain percentages cannot be reproduced from the reported data.","rationale":"The reader's weakest_assumption concerns whether the Huynh et al. baselines were produced under comparable conditions. That is a legitimate methodological worry, but a more decisive problem exists regardless of baseline provenance: the paper's own Table 3 does not show consistent superiority. The only way to recover two of the abstract's three margins is to mix dev-set best-model F1 with different baseline rows; no consistent test-set pairing yields +6.58%, +19.08%, and +0.94%. Section 5's explicit admission that the approach 'does not outperform the previous experiment' is an in-text limitation that directly contradicts the abstract. Therefore the central claim is unsupported on internal evidence, independent of any external reproducibility check. Because this confirms rather than changes the existing REJECT verdict, no verdict adjustment is needed. I agree with the reader's emphasis on baseline comparability as a contributing weakness, but the internal arithmetic is the load-bearing issue.","tokens_in":17839,"tokens_out":7040,"duration_ms":62241,"concrete_test":"Recompute, directly from Table 3, every paired ΔF1 between each joint model and its same-CLM, same-label Huynh et al. baseline, separately for dev and test. If the largest same-CLM test-set gain is about +1.47 points (XLM-R three-label) and mBERT joint models are negative on both three- and four-label test sets, then the abstract's 'consistently superior' claim and its three stated margins are falsified by the paper's own data. This arithmetic check settles the concern without requiring new experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—that the joint approach is 'consistently' superior to fine-tuned PhoBERT, mBERT, and XLM-R by +6.58%, +19.08%, and +0.94% F1—is not supported by the paper's own results. In Table 3, the best joint model (XLM-R + CNN, three-label test F1 = 82.78) exceeds the Huynh et al. baselines by +6.91 points over PhoBERT, +17.95 over mBERT, and +1.47 over XLM-R. These margins match neither the abstract's numbers nor any consistent test-set pairing. Same-CLM comparisons are mixed: on three-label test, PhoBERT-CNN improves by only +0.08 F1 and XLM-R-CNN by +1.47, while mBERT-CNN is worse by −0.91. On four-label test, XLM-R-CNN and mBERT-CNN are both worse than their Huynh et al. baselines. Section 5 explicitly states: 'this approach does not outperform the previous experiment.' Thus the headline claim fails on internal evidence. Baseline comparability—the reader's concern—is secondary, because even taking Table 3 at face value there is no consistent superiority. This is a correctness risk in the central argument, not a disagreement with prior consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a joint architecture for Vietnamese natural language inference (NLI) in which a frozen or lightly trained transformer-based contextualized language model (PhoBERT, XLM-R, mBERT, or InfoXLM) produces sentence-pair representations that are then classified by a CNN or BiLSTM. On the ViNLI dataset, the authors report a best three-label test F1 of 82.78% for XLM-R with a CNN, compare against the fine-tuned baselines of Huynh et al. (2022), and also compare with non-contextualized word embeddings and a one-shot ChatGPT baseline. The paper claims in the abstract that the joint approach 'consistently demonstrated superior performance' over fine-tuned PhoBERT, mBERT, and XLM-R by margins of +6.58%, +19.08%, and +0.94% F1, respectively.","tokens_in":18101,"tokens_out":3287,"duration_ms":35014,"significance":"If the central claim were supported, the paper would offer a practically useful result: a lightweight classifier on contextualized embeddings can match or exceed full fine-tuning of strong multilingual language models for Vietnamese NLI, at lower computational cost. The paper also contains useful descriptive material, including per-label and per-topic accuracy breakdowns, confusion matrices, and a comparison between contextualized and non-contextualized embeddings. However, the headline superiority claim is contradicted by the paper's own results and by its own statement in Section 5, and the baseline comparisons are not controlled experiments. The empirical contribution is therefore not yet established at the level required for publication.","major_comments":[{"comment":"The central claim of consistent superiority is not supported by the paper's own data. The abstract reports gains of +6.58%, +19.08%, and +0.94% F1 over PhoBERT, mBERT, and XLM-R, but Table 3 shows that the best joint model (XLM-R + CNN, three-label test) exceeds the Huynh et al. baselines by 6.91, 17.95, and 1.47 F1 points, respectively. These margins match no consistent pairing in the table. More importantly, same-CLM comparisons are mixed: on the three-label test set, PhoBERT-CNN improves by only 0.08 F1, XLM-R-CNN by 1.47, and mBERT-CNN is worse by 0.91; on the four-label test set, XLM-R-CNN and mBERT-CNN are both worse than the corresponding baselines. Section 5 explicitly states that 'this approach does not outperform the previous experiment.' The abstract and Section 5 must be reconciled, and the claim of consistent superiority should be removed or substantially qualified.","section":"Abstract and Section 5, Table 3"},{"comment":"The baselines in Table 3 are taken from Huynh et al. (2022) and were not rerun in the authors' own pipeline. The authors do not report whether the train/dev/test splits, tokenization, preprocessing (including VnCoreNLP segmentation), training hyperparameters, or evaluation scripts match those of Huynh et al. Differences below 1 F1 point, such as the 0.08-point improvement of PhoBERT-CNN over PhoBERT, are within the range one would expect from implementation differences or random seed variation. Without identical conditions or multiple runs with variance estimates, any claimed improvement over the baselines is not attributable to the joint architecture.","section":"Section 4.3, Table 3"},{"comment":"The metric definitions in Equations (1)-(4) define Accuracy, Precision_M, Recall_M, and F1_M over i in {1, 2, 3}, stated to correspond to entailment, contradiction, and neutral. Yet Table 3 reports results for both three-label and four-label settings. For the four-label setting, either the 'Other' class is included without a corresponding definition, or the metrics are computed over only three labels while the model was trained on four labels. The paper must state explicitly how the macro-average is computed for four labels. If the equations are used as written for four-label results, those results are not correctly defined.","section":"Section 4.3, Equations (1)-(4)"},{"comment":"The comparison with ChatGPT is based on only 100 randomly sampled test pairs, but the paper does not report any variance or confidence interval for the ChatGPT accuracies and F1 scores. Given the small sample size, a difference of 0.42 vs. 0.80 accuracy has a wide uncertainty interval, and the claim that Vietnamese NLI is 'a tricky task' for one-shot learning should be stated accordingly. Additionally, Table 5 reports F1 values for ChatGPT and the joint models but the number of labels and the exact prompting protocol are only partially described in the appendix; the appendix prompt also lists 'neutral' twice instead of the three label alternatives intended. These issues should be corrected before the comparison is used as evidence.","section":"Section 5, ChatGPT comparison and Table 5"}],"minor_comments":[{"comment":"The phrase 'contextualized work presentations' is likely a mistranslation of 'contextualized word representations' and should be corrected.","section":"Abstract"},{"comment":"The sentence 'This approach was first suggested by Sennrich et al. in their research on sentence summarization with neural attention models' misattributes a paper on subword translation to sentence summarization; the reference to Sennrich et al. (2016) is the correct type of work, but the description should be fixed.","section":"Section 3.2"},{"comment":"Reference [36] for word2vecVN points to VnCoreNLP (Vu et al., 2018), which is duplicated from reference [31]; the correct source for the word2vecVN embeddings should be found and cited.","section":"References"},{"comment":"The caption 'Three Image Sequences' in Figure 3 is unhelpful; the subfigures should be individually described or the caption should state that they are confusion matrices for the three selected models.","section":"Figure 3"},{"comment":"The statement 'we have to check with or without dropout, and the results that should have dropout that can increase 0.4 to 0.5 in accuracy' is grammatically unclear; please state the dropout values tested, the selected value for each model, and the direction of the effect.","section":"Section 4.2"},{"comment":"The Vietnamese ChatGPT prompt contains an error in the label list: it says 'trung lập' twice and omits 'mâu thuẫn' (contradiction); the English translation repeats 'neutral' and omits 'contradiction'. This affects the validity of the ChatGPT evaluation if the model received the incorrect prompt.","section":"Appendix 7"}],"recommendation":"reject","confidential_remarks":"The manuscript's central empirical claim is internally inconsistent with its own Table 3 and Section 5, and the baseline comparisons are not controlled, so the contribution cannot be accepted in its present form. The language quality and organization also fall well below the standard of a serious journal, but that alone would not be decisive. The paper's strongest asset is the descriptive error analysis and the contextualized-vs-non-contextualized comparison; if the authors rework the claims, rerun baselines under identical conditions, and correct the metric definitions, a more narrowly scoped empirical paper could be viable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward empirical sweep: four frozen CLM encoders, two neural classifiers, three- and four-label ViNLI, plus a small ChatGPT comparison and per-label error analysis. The best number, XLM-R + CNN at 82.78 F1 on three-label test, is plausible, and the error analysis is a useful reference for anyone working on Vietnamese NLI.\n\nBut the central claim in the abstract—consistent superiority over fine-tuning by the stated margins—is contradicted by the paper's own results. Table 3 shows mBERT-CNN is 0.91 points worse than the fine-tuned mBERT baseline on three-label test, and on four-label both XLM-R-CNN and mBERT-CNN are worse. Section 5 openly says 'this approach does not outperform the previous experiment.' The reported gain percentages in the abstract cannot be reproduced from Table 3. That is a load-bearing flaw, not a cosmetic one.\n\nTwo additional problems: Equations (1)-(4) define macro F1 over three labels, yet four-label results are reported without a matching definition. And the baselines are taken from Huynh et al. (2022) without rerunning under the same pipeline; no variance or significance tests are reported, and no code or data are released. Even if you take Table 3 at face value, there is no consistent superiority, so the baseline comparability concern is secondary.\n\nIf reframed as a comparative study that documents when the joint approach helps and when it hurts, the empirical content could be salvageable. As written, the paper's headline claim is demonstrably false, so it is not a reliable research contribution. A serious editor could desk reject on the internal contradiction alone.\n\nI would not send this to peer review in its current form; it would need a full rewrite of the claims, a careful handling of the metric definitions, and ideally re-run baselines. For a reading group, it's a decent example of how an abstract can overstate results, but not much else.","headline":"Empirical sweep is useful reference, but the abstract's superiority claim fails against the paper's own Table 3 and Section 5.","tokens_in":18673,"tokens_out":3116,"would_cite":false,"duration_ms":30681,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Freezing a large language model and adding a small neural network matches full fine-tuning for Vietnamese natural language inference.","keywords":["Transformer","Contextualized Language Model","Neural Network","Natural Language Inference","Vietnamese","ViNLI","PhoBERT","XLM-R"],"falsifier":"Re-run the fine-tuned baselines (PhoBERT, mBERT, XLM-R) on the same ViNLI splits with the same tokenization, learning rate 1e-5, and evaluation metric, and compare three-label test F1; if fine-tuned XLM-R reaches or exceeds 82.78% F1, the claimed advantage of the frozen joint model disappears.","tokens_in":17628,"feed_emoji":"🧠","tokens_out":7323,"duration_ms":64766,"temperature":0.7,"pith_summary":"This paper tries to establish that Vietnamese natural language inference can be handled well by a cheap joint architecture: freeze a transformer-based contextualized language model (PhoBERT, mBERT, XLM-R, or InfoXLM), take its last hidden layer as a contextual embedding of the premise–hypothesis pair, and let a small CNN or BiLSTM classify. On the ViNLI benchmark the best configuration, XLM-R with a CNN, reaches 82.78% F1 on the three-label test set, which the authors report as higher than the fine-tuned PhoBERT, mBERT, and XLM-R baselines. Most joint models land within roughly ±2.3% of those baselines, so the advantage is concentrated in the best configuration. The paper also shows contextualized embeddings beat static word vectors by a wide margin, and that one-shot ChatGPT performs far worse. If correct, the finding matters because it offers an efficient alternative to expensive full fine-tuning for languages with scarce resources.","feed_headline":"Joint model hits 82.78% F1 on Vietnamese NLI","feed_subtitle":"Freezing XLM-R and adding a small CNN beats fine-tuned PhoBERT, mBERT, and XLM-R at lower cost","key_machinery":"The load-bearing mechanism is the frozen CLM plus neural classifier pipeline. The CLM produces a sequence of contextualized vectors for the [CLS]-prefixed, [SEP]-separated premise–hypothesis input; the last hidden layer is converted to a fixed 1024-dimension representation and fed either to a CNN (convolution, ReLU, max-pooling, one branch per label) or to a two-layer BiLSTM. The CNN's sliding window extracts features from the matrix-form CLM output, which the paper says explains its consistent edge over BiLSTM in these experiments.","core_discovery":"The central discovery is that a frozen encoder plus a task-specific neural classifier is a strong recipe for Vietnamese NLI. After concatenating premise and hypothesis with [CLS] and [SEP], the authors feed the pair through the CLM at its maximum input length—no truncation—and pass the last hidden layer through a linear layer to a fixed 1024-dimensional space, followed by a CNN with four per-label branches (or a two-layer BiLSTM); the branch outputs are concatenated, dropped out, and linearly projected to label logits. The best result, 82.78% F1 (three-label test) with XLM-R + CNN, exceeds the fine-tuned PhoBERT, mBERT, and XLM-R numbers reported in [12]. The authors read this as evidence that the joint approach captures the contextual understanding of large models without updating their weights.","pith_inferences":["If the pattern transfers to other low-resource languages, freezing a large multilingual encoder and training only a small head could become a default efficient recipe for NLI when labeled data and GPU memory are scarce.","The reported gaps (e.g., +0.94% F1 over XLM-R) are within the range that could flip across random seeds; variance estimates and significance tests would be needed to confirm the advantage is real.","The per-label and per-topic error analyses are model-specific, so ensembling joint models built on different CLMs could push accuracy beyond any single configuration.","Because the encoder is frozen, the approach is compatible with distillation or data augmentation on the small head, which the paper only names as future work."],"forward_implications":["The best joint model, XLM-R combined with CNN, achieves 82.78% F1 on the three-label ViNLI test set, higher than the fine-tuned PhoBERT, mBERT, and XLM-R baselines reported in [12].","Contextualized embeddings from CLMs outperform static word vectors (word2vecVN, fasttext, w2v_cc_300d) by roughly a factor of two in accuracy, so vector-space inputs are a poor choice for Vietnamese NLI.","CNN heads generally beat BiLSTM heads across CLM embeddings, indicating the matrix-form CLM output favors convolutional feature extraction over recurrent sequence modeling.","One-shot ChatGPT reaches only about half the F1 of the joint models, suggesting Vietnamese NLI is still hard for few-shot generative inference.","Since only the small classifier is trained, joint models cut memory needs compared with fine-tuning large encoders, supporting applications with limited GPU resources."],"supporting_citations":[{"why":"Supplies the ViNLI dataset and the fine-tuned PhoBERT, mBERT, and XLM-R baseline scores the paper compares against.","marker":"[12]"},{"why":"Provides PhoBERT, the Vietnamese-specific pre-trained model used as one of the four contextualized language models.","marker":"[20]"},{"why":"Defines the BERT architecture and supplies the multilingual mBERT model used as an embedding source and baseline.","marker":"[21]"},{"why":"Establishes the Transformer architecture underlying all the contextualized language models used.","marker":"[22]"},{"why":"Provides VnCoreNLP, the word segmenter applied to Vietnamese text before PhoBERT tokenization.","marker":"[31]"},{"why":"Provides XLM-R large, the multilingual model whose joint CNN configuration yields the best reported F1.","marker":"[32]"},{"why":"Provides InfoXLM, the fourth CLM used for embedding extraction in the joint models.","marker":"[33]"},{"why":"Defines the BiLSTM architecture used as the recurrent classifier in the joint models.","marker":"[34]"},{"why":"Supplies the macro-F1 and accuracy formulas used to evaluate all models.","marker":"[37]"}],"fun_headline_variants":["Frozen XLM-R + CNN beats fine-tuning for Vietnamese NLI","82.78% F1: frozen XLM-R + CNN outperforms fine-tuning","Freezing XLM-R beats fine-tuning for Vietnamese NLI","No fine-tuning needed: joint CLM+CNN tops Vietnamese NLI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed superiority over fine-tuning assumes that the fine-tuned baseline numbers taken from [12] were produced under conditions comparable to the authors' own runs—same data splits, tokenization, and training settings—because those baselines were not re-run in this study.","fun_headline_variants_meta":{"raw":{"variants":["Frozen XLM-R + CNN beats fine-tuning for Vietnamese NLI","82.78% F1: frozen XLM-R + CNN outperforms fine-tuning","Freezing XLM-R beats fine-tuning for Vietnamese NLI","No fine-tuning needed: joint CLM+CNN tops Vietnamese NLI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3083,"prompt_tokens":962,"completion_tokens":2121,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2041}},"tokens_in":578,"tokens_out":2121,"duration_ms":14531,"temperature":1.0,"reasoning_tokens":2041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:26:23.260825+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the fine-tuned baselines (PhoBERT, mBERT, XLM-R) on the same ViNLI splits with the same tokenization, learning rate 1e-5, and evaluation metric, and compare three-label test F1; if fine-tuned XLM-R reaches or exceeds 82.78% F1, the claimed advantage of the frozen joint model disappears.","supporting_citations":[{"cited_title":"In: Proceed- ings of the 29th International Conference on Computational Linguistics, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the ViNLI dataset and the fine-tuned PhoBERT, mBERT, and XLM-R baseline scores the paper compares against."},{"cited_title":"In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp","cited_arxiv_id":null,"evidence_quote":"Provides PhoBERT, the Vietnamese-specific pre-trained model used as one of the four contextualized language models."},{"cited_title":"In: Proceedings of naacL-HLT, vol","cited_arxiv_id":null,"evidence_quote":"Defines the BERT architecture and supplies the multilingual mBERT model used as an embedding source and baseline."},{"cited_title":"In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vish- wanathan, S.V.N., Garnett, R","cited_arxiv_id":null,"evidence_quote":"Establishes the Transformer architecture underlying all the contextualized language models used."},{"cited_title":"In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp","cited_arxiv_id":null,"evidence_quote":"Provides XLM-R large, the multilingual model whose joint CNN configuration yields the best reported F1."},{"cited_title":"In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp","cited_arxiv_id":null,"evidence_quote":"Provides InfoXLM, the fourth CLM used for embedding extraction in the joint models."},{"cited_title":"Neural Comput","cited_arxiv_id":null,"evidence_quote":"Supplies the macro-F1 and accuracy formulas used to evaluate all models."}],"review_version":1}