{"id":"81807a00-2be9-4812-8d2a-c2e18426bd7c","arxiv_id":"2504.21017","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ViQA-COVID is a new Vietnamese COVID-19 reading comprehension dataset with 6,444 question-answer pairs, the first for Vietnamese with multi-span answers, benchmarked at 85.97% F1 by XLM-R large.","lead":"Vietnamese researchers introduce ViQA-COVID, a machine reading comprehension dataset with 6,444 question-answer pairs over 537 COVID-19 passages, the first multi-span extraction dataset for Vietnamese. The paper benchmarks five multilingual BERT-style models, with XLM-R large reaching about 86% F1 on the test set, and promises public release.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Passage splitting with max length 384/256 and stride 128 (Sec. 4.2) does not guarantee every gold answer span is in one window; boundary-crossing spans are lost, so Table 4 EM/F1 may be invalid.","rationale":"The reader's weakest_assumption called out the passage-splitting mechanism as a secondary concern, alongside the missing inter-annotator agreement and the Table 1/3 inconsistency. This stress-test focuses on the split mechanism because it is a concrete, testable correctness risk that directly undermines the reported benchmark numbers, which are part of the central claim. The paper's novelty claim (first multi-span extraction MRC dataset for Vietnamese) is supported by the cited related work, and the annotation process includes cross-checking, so the dataset itself may be valuable even if the evaluation protocol is flawed. The recommended verdict remains CONDITIONAL: the paper should not be rejected outright, but the authors should release the dataset, report annotation agreement, and either fix the windowing to guarantee answer-span containment or analyze how many examples are affected. The verdict is UNCHANGED from the reader's CONDITIONAL because the additional concern reinforces rather than shifts the original decision.","tokens_in":8516,"tokens_out":6611,"duration_ms":62101,"concrete_test":"Re-implement the Sec. 4.2 windowing exactly (RDRSegmenter tokenization; feature length 384 for XLM-R, 256 for PhoBERT; stride 128; document offset at multiples of stride) and, for each of the 6,444 QA pairs, check whether every gold answer span is fully contained in at least one feature window. If any pair has an uncovered span, rerun XLM-R large with a corrected scheme that guarantees containment (e.g., add a feature starting at the span start or use a large-enough overlap), and compare the resulting test EM/F1 with Table 4. Also recompute Table 3 from the actual passage-length distribution to determine whether the 537/612 discrepancy is a typo or reflects extra split features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that passages longer than the model maximum are split into features of length 384 (PhoBERT 256) with stride 128, and that overlap handles answers at split positions. This does not preserve all answer spans. With windows starting at 0, 128, 256, ... (for 384-token features), a ground-truth span that begins in one window and ends in the next, e.g. tokens 250-400, is contained in neither [0,383] nor [256,639]. The 128-token overlap only covers the boundary region, not the full span. The paper reports no check that every gold span is a substring of some feature, no discarded-example count, and no special handling of multi-span answers that straddle boundaries. Since 475 of the numbers in Table 3 are in the >=512-token class, a substantial share of passages are split, so this is not a corner case. If any gold spans are unreachable by the input representation, the reported EM/F1 in Table 4 are not valid estimates of model performance on ViQA-COVID as claimed. The inconsistency between Table 1's 537 passages and Table 3's 612-token row total compounds the uncertainty about whether the described preprocessing was applied correctly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ViQA-COVID, an extractive machine reading comprehension dataset for Vietnamese focused on COVID-19, constructed from CDC Vietnam and other reputable sources. The dataset contains 6,444 question-answer pairs over 537 passages, with roughly 20% of answers requiring multiple spans, and it is claimed to be the first multi-span extraction MRC dataset and the first COVID-19 MRC dataset for Vietnamese. The authors also report fine-tuning experiments with mBERT, PhoBERT-base/large, and XLM-R-base/large, concluding that XLM-R-large achieves the best test performance with 85.97% F1 and 72.00% EM. The paper includes dataset statistics, a description of the annotation process, an error analysis, and a discussion of the difficulty profile.","tokens_in":8717,"tokens_out":5793,"duration_ms":55055,"significance":"If the dataset and benchmark results are reliable, ViQA-COVID fills a real gap: there is no public COVID-19 MRC benchmark for Vietnamese, and a multi-span extraction dataset for Vietnamese would be a useful resource for low-resource NLP and health-domain QA. The paper provides a substantial annotation effort, documents question-type and answer-type distributions, reports experiments with four pretrained model families, and includes an error analysis that gives a concrete picture of the remaining challenges. The claimed contribution is therefore significant for the Vietnamese NLP community. The main value depends on the dataset being released and on the annotation quality and evaluation pipeline being verifiable, which the current manuscript does not fully establish.","major_comments":[{"comment":"The passage-splitting scheme does not guarantee that every gold answer span is contained in at least one input feature. With maximum feature length 384 (PhoBERT: 256) and stride 128, a span such as tokens 250 to 400 falls neither in window [0, 383] nor in window [256, 639]; the overlap region is only 128 tokens wide, so spans longer than the stride that straddle a split boundary are unreachable. The paper states that overlap handles answers at split positions, but it reports no check that every gold span is a substring of some feature, no count of skipped or truncated examples, and no special handling for multi-span answers whose individual spans cross boundaries. Since Table 3 indicates 475 passages in the \"greater than or equal to 512 tokens\" class, this is not a corner case; if any gold spans in the test set are unreachable, the EM/F1 values in Table 4 are not valid estimates of performance on ViQA-COVID as claimed.","section":"Section 4.2 and Section 4.4"},{"comment":"The passage totals in Tables 1 and 3 are inconsistent. Table 1 reports 537 passages (284 train + 139 dev + 114 test = 537), whereas Table 3's length distribution sums to 612 (335 train + 151 dev + 126 test = 612). Because Section 4.2 motivates the splitting procedure directly from Table 3, this discrepancy makes the preprocessing pipeline unverifiable. The authors should reconcile the two tables and state explicitly whether Table 3 counts original passages, split input features, or some other unit. Until this is resolved, the relationship between the reported dataset statistics and the actual experimental setup is unclear.","section":"Section 3.2, Tables 1 and 3"},{"comment":"No inter-annotator agreement is reported. The annotation process is described as creation and cross-checking by three CDC analysts with advice from two CDC experts, but no quantitative consistency measure, such as span-level agreement, Cohen's kappa, or a small-scale double-annotation study, is provided. Since the ground-truth spans are the basis for all reported benchmark scores, and since multi-span annotation is inherently more subjective than single-span annotation, the reliability of the labels is unverified. The authors should add an agreement statistic or a carefully described verification study; otherwise, the model scores in Table 4 cannot be distinguished from scores on noisy labels.","section":"Section 3.1"},{"comment":"The evaluation protocol for non-span and multi-span answers is underspecified. Table 1 shows that 10 to 12 percent of answers are non-span, and roughly 20 percent are multi-span, but the paper never states how EM and F1 are computed for these cases under the B/I/O tagging approach. For example, is a predicted span on an unanswerable question scored as 0, and how are partially overlapping sets of predicted spans aggregated into a single F1? Without this definition, the aggregate numbers in Table 4 cannot be reproduced or meaningfully compared with other MRC benchmarks. The authors should provide the exact scoring formula used for non-span and multi-span predictions.","section":"Section 4.3 and Section 5"}],"minor_comments":[{"comment":"The manuscript contains numerous grammatical and typographical errors, such as \"After two years of appearance,\" \"a answer can include multi-span,\" and \"Data was encrypted sensitive information.\" A thorough language edit would improve clarity.","section":"Abstract and throughout"},{"comment":"Section 4.2 says the models' maximum input feature length is 512 tokens, while Section 4.4 states that the maximum feature length used is 384 (PhoBERT 256). These statements should be reconciled, and the choice of 384/256 with stride 128 should be justified in terms of answer length and model capacity.","section":"Sections 4.2 and 4.4"},{"comment":"The paper says the dataset will be \"publicly release[d] soon,\" but no URL or repository is provided. For a resource paper, a release link or a concrete availability statement is expected, and without public access the central contribution cannot yet be used or independently checked.","section":"Section 3.2 and Section 6"},{"comment":"The experimental results are reported from what appears to be a single run per model. No confidence intervals, multiple-seed standard deviations, or significance tests are given, so the smaller performance gaps, such as XLM-R-base versus XLM-R-large on multi-span F1 (77.83 vs. 79.10 on the test set), may not be statistically meaningful. Adding variance across seeds or a paired significance test would strengthen the comparative claim.","section":"Section 5 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The dataset is not yet publicly available, so the central deliverable cannot be inspected. Given that the contribution is a resource paper, I would weigh release readiness and the resolution of the annotation-quality and preprocessing issues heavily in the next round."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the resource: ViQA-COVID, the first multi-span extraction MRC dataset for Vietnamese, COVID-19 domain, 6,444 question-answer pairs over 537 passages. That claim survives contact with the cited prior work – UIT-ViQuAD and UIT-ViNewsQA are single-span, UIT-ViWikiQA is sentence-level, ViMMRC is multiple-choice. So the dataset fills a real gap, and the annotation process, with CDC analysts and two experts, is a solid effort. The error analysis is also honest and useful; the four error groups are the kind of qualitative breakdown that helps people actually use the dataset.\n\nThe soft spots are not fatal, but they are real. The most glaring is the internal inconsistency: Table 1 says 537 passages (284/139/114 train/dev/test) while Table 3 sums to 612 passages (335/151/126). That is a 75-passage difference and it's not explained. Maybe Table 3 counts passage chunks after splitting, but then the paper should say so. As is, it undermines trust in the statistics.\n\nNo inter-annotator agreement is reported, which for a dataset paper is a standard but important omission. Without it, we don't know how noisy the gold spans are. The dataset is also not released yet – only \"will be released soon\" – which makes the benchmark numbers unreproducible.\n\nOn the stress-test about passage splitting: the specific example given (a span from token 250 to 400 with 384-length, stride-128 windows) is fine because the window [128, 511] contains it. However, the general worry is valid: with max length 384 and stride 128, any span longer than about 257 tokens is not guaranteed to fit inside a single window, and the paper does not check whether any gold spans are lost. Given that 475 of the passages are ≥512 tokens, this is not a corner case. The reported EM/F1 may still be correct, but the paper doesn't demonstrate that.\n\nThe benchmark itself is single-run, no seed variance, no human performance estimate – so take the absolute numbers with a grain of salt. The ordering of models (XLM-R large > XLM-R base > PhoBERT > mBERT) is unsurprising and probably robust.\n\nBottom line: the dataset is a useful contribution and the novelty is real, but the paper needs revision before the benchmark numbers are treated as authoritative. Fix the passage count discrepancy, report IAA, release the data, and verify or explicitly handle span preservation across splits. I'd send it to review, not desk reject, but I'd ask for those changes.\n\nRecommendation: engage with it, but read the statistics carefully.","headline":"A genuinely new Vietnamese multi-span COVID-19 MRC dataset, but the paper's own statistics don't add up and the benchmark numbers rest on unverified preprocessing.","tokens_in":9338,"tokens_out":2540,"would_cite":false,"duration_ms":23536,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces ViQA-COVID, a 6,444-question-answer Vietnamese dataset for COVID-19 machine reading comprehension, and reports that XLM-R large is the strongest tested model.","keywords":["Vietnamese","machine reading comprehension","COVID-19","multi-span extraction","question answering","benchmark dataset","XLM-R","sequence tagging"],"falsifier":"Re-annotate a random sample of the passages with independent annotators and compute inter-annotator agreement on the answer spans; if agreement is low (especially on multi-span answers), the reported benchmark numbers cannot be trusted as measures of model capability.","tokens_in":8282,"feed_emoji":"🦠","tokens_out":6109,"duration_ms":51862,"temperature":0.7,"pith_summary":"The paper claims to fill two gaps at once: no COVID-19 machine reading comprehension dataset existed for Vietnamese, and no Vietnamese dataset required models to extract multiple answer spans from one passage. ViQA-COVID is presented as the first multi-span extraction MRC dataset for Vietnamese, built from public health reports and news passages, with 6,444 question-answer pairs over 537 passages and about 21% of answers composed of multiple spans. The authors benchmark five models and report that XLM-R large reaches 85.97% F1 and 72.00% exact match on the test set, with single-span answers much easier than multi-span ones. The value, if the dataset is sound, is a reusable evaluation and training resource for Vietnamese health-domain question answering.","feed_headline":"6,444 Vietnamese COVID-19 QA pairs target multi-span answers","feed_subtitle":"The first Vietnamese multi-span extraction MRC dataset gives models a harder, more realistic health-domain reading test.","key_machinery":"The load-bearing mechanism is the multi-span extraction setup: instead of predicting a single start–end interval, the models are trained with a sequence-tagging head that labels each token B (begin), I (inside), or O (outside) an answer span, so multiple answers can be recovered from B and O tokens. Because most passages are longer than the 384/256-token model input limit, the input pipeline splits passages into overlapping features with a stride of 128, which is what the paper relies on to keep answer spans intact. This combination—B/I/O tagging plus sliding-window splitting—is what the paper uses to make a long-passage, multi-answer Vietnamese benchmark tractable for BERT-style encoders.","core_discovery":"The central claim is that ViQA-COVID is a valid, reusable benchmark for Vietnamese COVID-19 machine reading comprehension, and the first Vietnamese multi-span extraction MRC dataset. The dataset contains 6,444 question-answer pairs over 537 passages, with 21.0–21.3% of answers multi-span, 10–12% non-span (unanswerable), and the rest single-span. On this benchmark, XLM-R large outperforms the other four tested models, achieving 72.00% exact match and 85.97% F1 on the test set, confirming that cross-lingual pretraining transfers to Vietnamese health texts better than Vietnamese-only PhoBERT variants. The paper also analyzes error types, showing that multi-span questions and long sequences of dates, places, or people are the main sources of failure.","pith_inferences":["If ViQA-COVID is released and maintained, it could become a standard low-resource Vietnamese reading benchmark, and its 21% multi-span share might push Vietnamese models toward span-set decoding rather than single-interval predictions.","The error analysis suggests that enumerations of dates, places, and people are the hardest cases; an extension would be a dataset split that deliberately oversamples those question types to stress-test models.","The same annotation pipeline of public-health reports plus expert-advised questions could transfer to other Vietnamese health topics, such as dengue or influenza, without redesigning the dataset format.","Because a non-Vietnamese-specific XLM-R large wins despite the competition, a testable follow-up is whether a Vietnamese-specific model trained with a span-set objective would close the multi-span gap."],"forward_implications":["Vietnamese health-domain question-answering systems can now be trained and evaluated on a COVID-19-specific benchmark instead of relying only on general-domain Vietnamese datasets.","Multi-span extraction becomes a measurable sub-task in Vietnamese NLP, with about 21% of answers requiring multiple spans, so progress on that specific challenge can be tracked.","The reported model ranking (XLM-R large best, then XLM-R base, then PhoBERT variants, then mBERT) gives practitioners a clear baseline hierarchy for future work on Vietnamese MRC.","Because the dataset includes unanswerable questions (about 10–12%), it also supports evaluation of a model's ability to abstain rather than hallucinate an answer."],"supporting_citations":[{"why":"Provides the Vietnamese single-span MRC dataset and annotation-process template that ViQA-COVID's construction builds on.","marker":"[15]"},{"why":"Supplies the sequence tagging approach (B/I/O) used for all multi-span answer extraction in this paper.","marker":"[20]"},{"why":"Introduces XLM-R, the pretrained model family that achieves the best benchmark scores.","marker":"[2]"},{"why":"Introduces PhoBERT, the Vietnamese-specific pretrained model used as a strong baseline.","marker":"[13]"},{"why":"Earlier COVID-19 MRC dataset that motivates the domain and establishes the question-answering setup.","marker":"[12]"},{"why":"Another COVID-19 QA dataset, used as prior work that ViQA-COVID extends to the multi-span and Vietnamese setting.","marker":"[21]"}],"fun_headline_variants":["First Vietnamese multi-span MRC dataset for COVID-19","6,444 QA pairs push Vietnamese MRC beyond single-span","New dataset tests Vietnamese reading with multi-span answers","XLM-R large leads on novel Vietnamese COVID-19 QA benchmark","ViQA-COVID: first multi-span extraction dataset in Vietnamese"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's ground truth is taken as correct without a reported inter-annotator agreement measure, so if the manual answer spans are noisy or inconsistent, the model scores are not a valid measure of reading ability.","fun_headline_variants_meta":{"raw":{"variants":["First Vietnamese multi-span MRC dataset for COVID-19","6,444 QA pairs push Vietnamese MRC beyond single-span","New dataset tests Vietnamese reading with multi-span answers","XLM-R large leads on novel Vietnamese COVID-19 QA benchmark","ViQA-COVID: first multi-span extraction dataset in Vietnamese"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1519,"prompt_tokens":911,"completion_tokens":608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":535}},"tokens_in":527,"tokens_out":608,"duration_ms":5202,"temperature":1.0,"reasoning_tokens":535,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:38:15.255395+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the passages with independent annotators and compute inter-annotator agreement on the answer spans; if agreement is low (especially on multi-span answers), the reported benchmark numbers cannot be trusted as measures of model capability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Vietnamese single-span MRC dataset and annotation-process template that ViQA-COVID's construction builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sequence tagging approach (B/I/O) used for all multi-span answer extraction in this paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces XLM-R, the pretrained model family that achieves the best benchmark scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces PhoBERT, the Vietnamese-specific pretrained model used as a strong baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier COVID-19 MRC dataset that motivates the domain and establishes the question-answering setup."},{"cited_title":"Rapidly Bootstrapping a Question Answering Dataset for COVID-19","cited_arxiv_id":"2004.11339","evidence_quote":"Another COVID-19 QA dataset, used as prior work that ViQA-COVID extends to the multi-span and Vietnamese setting."}],"review_version":1}