Pith. sign in

REVIEW 4 major objections 5 minor 21 references

A Large-Scale Benchmark for Vietnamese Sentence Paraphrases

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces ViSP, the first large-scale Vietnamese sentence-paraphrase dataset, with over 1.2 million original–paraphrase pairs built by combining LLM generation with human validation, and positions it as a benchmark for…

desk verdict First large-scale Vietnamese paraphrase dataset, but Table 4's pair counts don't match its own averages, so the 1.2M headline needs reconciliation before I'd trust the resource. read the letter →

arxiv 2502.07188 v1 pith:V76AMI4O submitted 2025-02-11 cs.CL

Sang Quang Nguyen , Kiet Van Nguyen This is my paper · ORCID
classification cs.CL
keywords Vietnameseparaphrasegenerationdatasetbenchmarklargelanguagemodelslow-resourceNLPhumanevaluationsequence-to-sequence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish ViSP as the first large-scale benchmark for Vietnamese sentence paraphrasing, filling a gap in a language that has been under-served in paraphrase research. It argues that a hybrid pipeline—automatic generation by the Gemini language model guided by a few-shot prompt, followed by majority-vote human filtering—can produce a dataset of more than 1.2 million pairs at a quality level suitable for training and evaluation. The authors further claim that this resource makes possible, for the first time, a systematic comparison of paraphrase methods for Vietnamese, from simple augmentation techniques to fine-tuned sequence-to-sequence models and large language models. A sympathetic reader would care because ViSP could become the standard testbed for Vietnamese paraphrase generation and a template for building similar resources in other low-resource languages.

What carries the argument

The load-bearing mechanism is the dataset-construction pipeline: sentence collection from UIT-ViQuAD, UIT-ViNewsQA, ALQAC, and ViNLI; preprocessing and topic classification with Gemini; a few-shot prompt chosen through a BLEU-4 comparison between Gemini and human annotators; generation of k paraphrases per sentence via the formalized operation M_f(s, k) = {p1, ..., pk}; and a three-annotator, two-of-three majority vote applying a four-rule checklist (semantic equivalence, fluency and grammar, style and tone consistency, no contradictions). The human-validation stage is what converts raw LLM output into what the authors call a high-quality benchmark, and the reported average error rate of 4.49% across the dataset is the quantitative evidence for that conversion.

What would settle it

Draw a stratified random sample of accepted ViSP pairs, stratifying by source corpus, topic, and generation round, and have fresh annotators who were not involved in the original validation apply the Section 2.5 checklist. If the semantic-equivalence error rate in the sample significantly exceeds the reported 4.49% average, or if the error rate varies substantially across rounds or sources, then the validation step did not uniformly guarantee quality across the dataset.

Watch

Extended reading notes

Core claim

The central claim is that ViSP is the first large-scale, human-verified Vietnamese sentence-paraphrase corpus, containing 1,178,942 original–paraphrase pairs drawn from openly available Vietnamese news and Wikipedia sources. The dataset is generated by Gemini using a few-shot prompt that was selected after beating human annotators on a 300-sentence BLEU-4 comparison, and each pair is then vetted by three annotators with a two-out-of-three True vote required for inclusion. The paper reports that human evaluators rate the retained paraphrases above 4.7 out of 5 across informativeness, relevance, fluency, and coherence, and it presents baseline experiments showing that monolingual Vietnamese models such as BARTpho-wordlarge outperform multilingual models on this benchmark, while still falling short of human performance.

Load-bearing premise

The dataset's quality rests on the assumption that the single few-shot prompt chosen by a 300-sentence BLEU-4 comparison against human annotators generalizes to all 1.2 million generated pairs, and that the two-of-three human vote reliably removes every invalid paraphrase.

Editorial extensions

If this is right

  • ViSP gives Vietnamese NLP a large training set for paraphrase generation, enabling models like BARTpho and ViT5 to be fine-tuned and evaluated on a common benchmark rather than on translated or question-only data.
  • The paper's baselines establish that monolingual Vietnamese models, especially BARTpho-wordlarge, currently produce the most lexically faithful paraphrases, while multilingual mBART retains better semantic similarity at smaller output counts.
  • The dataset's topic and length breakdowns allow controlled evaluation of how paraphrase quality varies across domains and sentence structures, which can guide targeted improvements for difficult cases such as metaphor and metonymy.
  • Human performance scores, with BERTScore of 88.30 versus the best model's 86.17 on Test300, quantify the remaining gap and give future work a concrete target.
  • Because ViSP is sourced from public Vietnamese corpora and released for research, it can support downstream tasks the authors name, including machine translation, question answering, retrieval-augmented generation, and sentence-similarity models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same generation-plus-human-filter pipeline could be transferred to other low-resource languages, provided a strong few-shot LLM and a small panel of native speakers are available; the 4.49% error rate suggests the approach is viable but not noise-free.
  • The paper does not assess per-prompt or per-source error rates; an implication is that the dataset's quality may be uneven across topics or the six generation rounds, so downstream users may want to stratify by source when training or evaluating.
  • One untested extension is to use ViSP as a training set for a dedicated Vietnamese paraphrase model, which the authors list as future work; if such a model closes the gap to human BERTScore, that would confirm the benchmark's utility as a training resource rather than only an evaluation set.
  • Since all pairs come from news and Wikipedia text, the benchmark likely under-represents conversational and specialized domains; a reader should not assume ViSP covers informal Vietnamese or technical jargon without additional curation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ViSP, a Vietnamese sentence-paraphrase dataset, and reports that it contains 1.2 million original–paraphrase pairs collected from news, Wikipedia, law, and other domains. The construction pipeline uses Gemini with a few-shot prompt selected via a small BLEU-4 comparison against human annotators, followed by a three-annotator majority-vote validation. The paper also presents an extensive empirical comparison of fine-tuned sequence-to-sequence models, data augmentation baselines, and several large language models on the resulting train/validation/test splits, using BLEU, ROUGE, BERTScore, and diversity metrics.

Significance. If the resource claims are accurate, ViSP would be a substantial new benchmark for Vietnamese paraphrasing, filling a clear gap: existing Vietnamese resources such as ViQP cover only questions, and TaPaCo includes only 962 Vietnamese pairs. The human validation step, the topic diversity, and the breadth of baselines (BARTpho, ViT5, mBART, mT5, and multiple LLMs) are all valuable components for future research. The paper also makes the dataset publicly available, which supports reproducibility and follow-up work. However, the central size claim is currently undermined by an internal inconsistency in the reported statistics, and several supporting evaluations lack statistical rigor.

major comments (4)
  1. [Table 4 / Section 2.6.1]
  2. [Section 2.3 / Table 1 / Section 2.4]
  3. [Section 2.5 / Section 2.6.2]
  4. [Table 8 / Table 12]
minor comments (5)
  1. [Table 8] The Back Translation row shows BERTScore 21.60 on the Test set, which is inconsistent with the neighboring scores of approximately 79; this is likely a typographical error and should be corrected.
  2. [Section 2.3] The roles of the 'generation group' and 'evaluation group' are confusingly described; it is unclear which annotators produced the human reference paraphrases used in Table 1 and which only evaluated them.
  3. [Appendix C] Figure 6 is referenced in the text but does not appear in the manuscript; the error distribution figure should be included or the reference removed.
  4. [Table 4 footnote] The phrase 'all possible combinations' is ambiguous; a precise definition of how sentence pairs are formed is needed, especially given the inconsistency described above.
  5. [Throughout] There are numerous citation and formatting issues, including inconsistent citation styles and misspelled names (e.g., 'Y ang' in the references); these should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ViSP's construction relies on external human validation, not on self-referential derivation.

full rationale

ViSP is a dataset-construction paper. The claimed derivation chain is: collect Vietnamese sentences from external corpora, preprocess, generate candidate paraphrases with Gemini under a few-shot prompt, filter by a 2-of-3 human majority vote, and then report statistics and model evaluations. The central claim, '1.2M original–paraphrase pairs,' is a descriptive dataset statistic, not a quantity predicted from a fitted model. The selection of the few-shot prompt in Section 2.3 uses BLEU-4 against human-written paraphrases on 300 sentences; that is an engineering choice and is not renamed as a prediction, and the final dataset quality is gated by the independent human validation in Section 2.5. The paper's self-citations (UIT-ViQuAD, ViQP) are used as data sources or related work, not as load-bearing uniqueness arguments. No equation in the paper defines the reported dataset statistics in terms of the evaluation metrics, and no fitted parameter is later called a prediction. The reader-flagged concern about Table 4 (Pair counts inconsistent with Original × Avg. paraphrase per original) is a substantive correctness and consistency issue in the descriptive statistics, but it is not circularity: the pair count is not derived from inputs in a way that makes the claim true by construction. Similarly, the Limitations section's admission that CoT/ToT prompting was not compared is a scope limitation, not a circular step. Overall, the paper's central resource is externally human-validated and the derivations are self-contained, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and rests on data curation choices rather than mathematical axioms. No free parameters or invented entities are introduced; the load-bearing assumptions are the reliability of BLEU-4 as a prompt-selection criterion and the sufficiency of majority human voting for paraphrase validity.

assumptions (3)
  • domain assumption Majority vote of two out of three annotators correctly identifies valid paraphrases.
    Used in Section 2.5 to filter all 1.2M pairs; if annotators miss semantic errors, the 'high quality' claim fails.
  • domain assumption BLEU-4 against human references is a valid proxy for paraphrase generation quality when comparing Gemini to humans and when selecting the few-shot prompt.
    Section 2.3 uses BLEU-4 win rates to conclude Gemini outperforms humans and to select the prompt for the full corpus; this is a metric assumption, not a semantic equivalence check.
  • domain assumption The four source corpora are representative of general Vietnamese and cover diverse topics.
    Section 2.1 collects from ViQuAD, ViNewsQA, ALQAC, and ViNLI; the topic distribution is dominated by health and society, which may limit generality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Large-Scale Benchmark for Vietnamese Sentence Paraphrases." pith.science (2026). https://pith.science/paper/V76AMI4O

@misc{pith2026250207188,
  author       = {Pith},
  title        = {Pith review of: A Large-Scale Benchmark for Vietnamese Sentence Paraphrases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V76AMI4O}},
  note         = {Machine review of arXiv:2502.07188}
}
read the original abstract

This paper presents ViSP, a high-quality Vietnamese dataset for sentence paraphrasing, consisting of 1.2M original-paraphrase pairs collected from various domains. The dataset was constructed using a hybrid approach that combines automatic paraphrase generation with manual evaluation to ensure high quality. We conducted experiments using methods such as back-translation, EDA, and baseline models like BART and T5, as well as large language models (LLMs), including GPT-4o, Gemini-1.5, Aya, Qwen-2.5, and Meta-Llama-3.1 variants. To the best of our knowledge, this is the first large-scale study on Vietnamese paraphrasing. We hope that our dataset and findings will serve as a valuable foundation for future research and applications in Vietnamese paraphrase tasks.

Figures

Figures reproduced from arXiv: 2502.07188 by the authors.

Figure 1
Figure 1. The overview process of creating our dataset [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Word clouds illustrating the most frequent words for each topic in the ViSP dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. BLEU-4 and BERTScore across different topics. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: BLEU-4 and BERTScore across different lengths. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of sentence source in the ViSP dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Error rate distribution across different paraphrase verification rule. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 16 canonical work pages

  1. [1]

    UIT-ViQuAD(Nguyen et al., 2020b) This machine reading comprehension dataset includes over 23,000 human-generated question-answer pairs. These pairs are derived from 5,109 passages extracted from 174 Vietnamese Wikipedia articles, providing a rich source of information and ensuring a diverse range of topics and contexts

    work page

  2. [2]

    These pairs are based on a collection of 4,416 Vietnamese healthcare news articles, with answers being textual spans directly taken from the corresponding articles

    UIT-ViNewsQA(Van Nguyen et al., 2022) This corpus consists of 22,057 question-answer pairs created by crowd-workers. These pairs are based on a collection of 4,416 Vietnamese healthcare news articles, with answers being textual spans directly taken from the corresponding articles

    work page 2022

  3. [3]

    Each pair is carefully reviewed for clarity and accuracy, making it an essential resource for testing question answering models in the legal domain

    ALQAC (Nguyen et al., 2023a): The ALQAC dataset contains thousands of multiple-choice question- answer pairs, sourced from Vietnamese legal documents. Each pair is carefully reviewed for clarity and accuracy, making it an essential resource for testing question answering models in the legal domain

    work page

  4. [4]

    These pairs are extracted from more than 800 online news articles, offering a substantial and varied dataset for natural language inference tasks

    ViNLI (Huynh et al., 2022) The ViNLI corpus comprises over 30,000 human-annotated premise- hypothesis sentence pairs. These pairs are extracted from more than 800 online news articles, offering a substantial and varied dataset for natural language inference tasks. Figure 5 shows the distribution of original sentence sources in the ViSP dataset, with UIT-V...

    work page 2022

  5. [5]

    A higher ENT-4 value suggests more varied and creative paraphrases

    ENT-4 (Shannon, 1948): This is the entropy of 4-grams, reflecting the diversity and unpredictability of word combinations. A higher ENT-4 value suggests more varied and creative paraphrases

    work page 1948

  6. [6]

    A lower Jaccard value indicates less overlap with the source, and hence greater paraphrase diversity

    Jaccard (Jaccard, 1901): This score measures the lexical overlap between the original sentence and its paraphrase. A lower Jaccard value indicates less overlap with the source, and hence greater paraphrase diversity. B.2 Human Eval We conduct manual evaluations where human reviewers assess the quality of paraphrased sentences. Each paraphrase is evaluated...

    work page 1901

  7. [12]

    A higher BLEU-4 value indicates greater syntactic and lexical alignment with the reference

    BLEU-4 (Papineni et al., 2002): This metric measures the precision of 4-grams between the generated paraphrase and the reference. A higher BLEU-4 value indicates greater syntactic and lexical alignment with the reference

    work page 2002

  8. [13]

    A higher ROUGE-2 value reflects better preservation of key content from the reference

    ROUGE-2 (Lin, 2004): This metric calculates the recall of bigrams (2-grams) in the generated paraphrase compared to the reference. A higher ROUGE-2 value reflects better preservation of key content from the reference

    work page 2004

Show all 21 references
  1. [14]

    A higher BERTScore implies stronger semantic similarity and fidelity to the reference text

    BERTScore (Zhang et al., 2019): This measure uses contextual embeddings to compare each token in the generated paraphrase with those in the reference. A higher BERTScore implies stronger semantic similarity and fidelity to the reference text

    2019

  2. [15]

    Higher values indicate more diverse and less repetitive paraphrases

    DIST-1 and DIST-2 (Li et al., 2016): These metrics capture the distinctiveness of unigrams and bigrams, respectively. Higher values indicate more diverse and less repetitive paraphrases

    2016

  3. [18]

    INF (Informativeness): How well does the paraphrase retain the original meaning?

  4. [19]

    REL (Relevance): To what extent are the important facts and details preserved?

  5. [20]

    FLU (Fluency): How fluent and natural does the sentence sound?

  6. [21]

    Năm nay là lần đầu tiên Đại lễ săn thỏ Phục sinh được tổ chức kể từ 2017

    COH (Coherence): How well do the sentence parts fit together to form a coherent whole? C Paraphrase Verification Checklist To ensure the quality and accuracy of the paraphrased sentences in our dataset, we implemented a verification process where annotators assessed whether ea...

    2017

  7. [2002]

    In Australian Joint Conference on Artificial Intelligence, pages 24–35

    Experiments in query paraphrasing for infor- mation retrieval. In Australian Joint Conference on Artificial Intelligence, pages 24–35. Springer. Appendix In this section, we provide supplementary information to support the main content of this paper. This includes additional d...

  8. [2005]

    LAMP-TR-125 CS-TR-4754 UMIACS-TR-2005-57, University of Maryland, Col- lege Park, MD

    A paraphrase-based approach to machine translation evaluation. LAMP-TR-125 CS-TR-4754 UMIACS-TR-2005-57, University of Maryland, Col- lege Park, MD. Yves Scherrer. 2020. TaPaCo: A corpus of sentential paraphrases for 73 languages. In Proceedings of the Twelfth Language Resourc...

    2005

  9. [2013]

    Computa- tional Linguistics, 39(2):301–353

    Statistical metaphor processing. Computa- tional Linguistics, 39(2):301–353. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual translation with exten- sible multilingual pretraining and finetuning....

    2020 arXiv

  10. [2018]

    Delphine Bernhard and Iryna Gurevych

    Generating natural language adversarial exam- ples. Delphine Bernhard and Iryna Gurevych. 2008. An- swering learners’ questions by retrieving question paraphrases from social q&a sites. Rahul Bhagat and Eduard Hovy. 2013. What Is a Para- phrase? Computational Linguistics, 39(3...

    2008 arXiv

  11. [2020]

    In International Conference on Machine Learn- ing, pages 4411–4421

    Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference on Machine Learn- ing, pages 4411–4421. PMLR. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila...

    2024 arXiv

  12. [2022]

    ACM Trans

    New vietnamese corpus for machine reading comprehension of health news articles. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 21(5). Peter Wallis. 1993. Information retrieval based on para- phrase. In Proceedings of pacling conference. Cite- seer. Jason Wei, Xuezhi Wang, ...

    1993 arXiv

  13. [2024]

    Advances in Neural Information Processing Systems, 36

    Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Y oav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint...

    2019 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.