{"id":"1a4c52b0-f0d3-42e9-b8f4-6084a374665f","arxiv_id":"2601.03792","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A dual-model consistency filter and a 94.2% expert-approved sample back VietMed-MCQ, a 3,190-question Vietnamese Traditional Medicine quiz where Qwen2.5 models beat Vietnamese-specialized models by 7.2%.","lead":"This paper releases VietMed-MCQ, a 3,190-question multiple-choice benchmark for Vietnamese Traditional Medicine built by having one large language model generate questions and a second model filter out disagreements. It also finds Chinese-trained models outperform Vietnamese-specialized models on the quiz, attributing the edge to shared Sino-Vietnamese medical concepts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Qwen-family filtering may bias the benchmark: Qwen's +7.21% edge could reflect selection on Qwen-student agreement rather than cross-lingual transfer.","rationale":"The paper has two intertwined claims: (1) VietMed-MCQ is a usable, high-quality evaluation resource; (2) the benchmark results reveal cross-lingual conceptual transfer from Chinese-prior models. The first claim has independent support: a stratified 500-item expert review with 94.2% acceptance and substantial inter-rater agreement, plus a transparent dual-model pipeline. The second claim is the least secure. The filter's retention rule uses Qwen2.5-32B as the arbiter, and the same model family then tops the leaderboard. This is an internal design confound, not a dispute with external consensus. It directly threatens the headline mechanism, not just the dataset's raw quality. The reader's weakest assumption—that teacher-student agreement is a valid quality filter—captures this, and the reader's rationale explicitly names the selection-bias problem. I agree with the conditional verdict: the resource may still be useful, but the explanatory claim is unsupported until the selection-bias check is run. A conditional acceptance, rather than rejection, is appropriate because the concern is concrete and addressable, and the dataset's human-validated subset provides partial evidence of item quality. The proposed test on the rejected items (or a non-Qwen control) would settle whether the Qwen advantage is an artifact of filter selection. If the advantage persists on the control set, the cross-lingual interpretation becomes substantially more credible. Until then, the benchmark's model rankings should be read as 'accuracy on a Qwen-filtered test set,' not as evidence for cross-lingual transfer. No change to the reader's conditional verdict is needed; the condition should explicitly include this control.","tokens_in":12230,"tokens_out":3950,"duration_ms":42423,"concrete_test":"Benchmark all seven models on the 269 rejected candidates (or on a control set generated with a non-Qwen student, e.g., Llama-3.1-8B) using the same zero-shot protocol and option shuffling. Compute Qwen2.5-7B minus the best Vietnamese model with McNemar's test and bootstrap CIs. If the +7.21% advantage shrinks to non-significance or reverses on the rejected/control set, the cross-lingual transfer claim is a filter-selection artifact; if the advantage persists, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 retains a question only when Qwen2.5-32B's blind answer matches the teacher (\\hat a = a). Section 5.2 then attributes Qwen2.5-7B's +7.21% accuracy advantage over Vietnamese models to cross-lingual conceptual transfer. But the evaluation set is not a random sample of VTM knowledge; it is the subset on which Qwen2.5-32B agreed with Llama-3.1-70B. The 500-question human validation does not address this: it estimates whether accepted items are correct, not whether the retained subset is free of model-family selection. Only 269/3,459 candidates were rejected, so the selection pressure appears small, yet those 269 are precisely the cases where Qwen's prior diverged; if they disproportionately contain items on which non-Qwen models would excel, the ranking is biased before any model is run. The paper's own Limitations section concedes the evidence check is 'a necessary but insufficient quality gate' and cannot verify semantic entailment or reasoning correctness. No experiment separates 'truth-correlated agreement' from 'bias-correlated agreement.' Thus the central explanatory claim—that Chinese priors dominate Vietnamese specialization—is confounded by using Qwen as the validator.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents VietMed-MCQ, a synthetic multiple-choice question benchmark for Vietnamese Traditional Medicine. The pipeline extracts contexts from textbooks, uses Llama-3.1-70B as a teacher to generate question-answer-evidence tuples, and filters candidates by requiring Qwen2.5-32B as a student to independently produce the same answer and a substring-based evidence check. Of 3,459 candidates, 3,190 survive. A stratified human validation of 500 items reports 94.2% acceptance (Fleiss' kappa = 0.82). Benchmarking seven models shows Qwen2.5-7B at 62.01% accuracy, outperforming Vietnamese-specialized models, a result the authors attribute to cross-lingual transfer from Chinese/TCM knowledge. The paper also provides difficulty distribution, error analysis, and limitations.","tokens_in":12485,"tokens_out":8264,"duration_ms":70233,"significance":"If the results are robust, VietMed-MCQ would be a valuable resource for evaluating culturally specific medical NLP, and the cross-lingual transfer finding would be of broader interest. The paper's strengths include the public release of code and data, a small human validation with reported inter-annotator agreement, randomized option shuffling during evaluation, and a candid limitations section. However, the central explanatory claim—that Chinese language priors drive performance—is currently confounded by the Qwen-based filtering step and by uncontrolled model comparisons. The benchmark itself appears usable, but the stated conclusions about cross-lingual transfer need substantially stronger evidence before they can be accepted.","major_comments":[{"comment":"The consistency filter uses Qwen2.5-32B as the Student, and the retained dataset thus excludes the 269 candidates (7.78%) on which Qwen disagreed with the Teacher. The subsequent evaluation then reports that Qwen2.5-7B outperforms Vietnamese models by a large margin. This is not a neutral test: the benchmark has been conditioned on Qwen-family judgment. The 500-question human validation (§4.2) only assesses the correctness of retained items; it does not test whether filtering introduces a family-dependent bias. The paper itself notes in §6 that an ablation isolating the student filter would strengthen the claims, but no such experiment is reported. Please report rankings on the full candidate set, or on subsets produced by alternative student models (e.g., a non-Qwen student), and show that the Qwen advantage persists. Without this, the cross-lingual transfer conclusion is confounded.","section":"Section 3.3 / Section 5.2"},{"comment":"The reported '+7.21% accuracy' over the best Vietnamese model is arithmetically inconsistent. Table 3 gives Qwen2.5-7B 62.01%, VinaLlama 55.17%, and Vistral 54.80%. The gap over the best Vietnamese model (VinaLlama) is 6.84 percentage points; 7.21 is the gap over Vistral only. The abstract repeats the 7.21 figure without specifying the baseline. Moreover, the cross-lingual transfer interpretation is not supported by a controlled comparison: VinaLlama and Vistral differ from Qwen in base architecture, pretraining corpus, and instruction-tuning recipe, and Llama-3-8B (which lacks Chinese priors) also outperforms both Vietnamese models. Please correct the number and test matched models (e.g., the same base with and without Vietnamese specialization) or explicitly discuss the confounds.","section":"Section 5.2 / Table 3 / Abstract"},{"comment":"The quality claim for the dataset rests on a weak automatic gate. Eq. (3) only checks substring containment; as the authors state in §6, it 'cannot verify semantic entailment... cannot assess reasoning correctness... cannot check evidence sufficiency.' Only 500 of 3,190 questions (15.7%) received human review, and this review was post-hoc—no items were removed (see the conflicting statements in §4.2). Consequently, 2,690 questions are supported only by dual-model agreement, which is not a gold standard. The abstract and conclusion describe the dataset as 'high-quality' and 'clinically verified' without qualification. Please either expand human validation to a larger sample (including rejected candidates) or qualify the quality claims in the abstract and conclusion to reflect the actual evidence level.","section":"Sections 3.3, 4.2, and 6"}],"minor_comments":[{"comment":"Fleiss' kappa = 0.82 is labeled 'substantial'; under Landis and Koch, 0.81–1.00 is 'almost perfect.' Correct the terminology.","section":"Section 4.2"},{"comment":"The text says 'Questions receiving at least three Accept votes... were retained' but immediately adds 'no questions were removed based on validation results.' Clarify whether the validation was used to filter or was purely post-hoc.","section":"Section 4.2"},{"comment":"'Clinically verified' and 'expert approval' overstate the role of the four student evaluators; suggest 'human-validated' with evaluator composition specified.","section":"Abstract / Section 1"},{"comment":"Difficulty thresholds are 'not reproducible' because they were adjusted to target proportions. Provide the final thresholds or an automated classifier to make the difficulty labels reproducible.","section":"Section 4.3, footnote 2"},{"comment":"The sentence 'The proportion of Hard questions (26.3' is truncated; complete the sentence.","section":"Section 6"},{"comment":"The qualitative error analysis lacks methodological detail (how many examples, selection criteria). Add a brief protocol.","section":"Section 5.4"},{"comment":"AfriMedQA appears in the text but AfriMed-QA in the reference list; align the spelling.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code will be useful to the community, and the authors are transparent about limitations. However, the headline cross-lingual transfer claim needs stronger evidence, and the reported effect size should be corrected. I recommend major revision with the specific experiments outlined above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the dataset is the contribution, not the explanation. VietMed-MCQ gives the field a genuinely missing artifact—3,190 MCQs for Vietnamese Traditional Medicine with a real human validation sample (500 items, 94.2% accept, kappa 0.82). That is a step up from most synthetic medical QA efforts, and the pipeline (RAG + teacher-student consensus) is cheap enough that others can copy it into similar low-resource niches. The finding that Qwen2.5-7B beats VinaLlama and Vistral by ~7 points is worth reporting; it is a fresh empirical datapoint.\n\nThe soft spots are real but mostly fixable. The consistency gate is, by the authors' own admission, a 'necessary but insufficient' quality check: the substring evidence test cannot verify entailment, and the student filter only checks agreement between two LLMs. The bigger worry—which the stress-test note gets right—is that Qwen2.5 is both the student filter and the winning family. The 269 rejected items are exactly the cases where Qwen diverged from Llama, so the surviving set is not a neutral sample of VTM knowledge. That said, 7.78% rejection is small, and the human validation did pass 94% of the retained set, so the selection pressure is limited. The cross-lingual transfer claim, however, needs a capability-matched control (e.g., a Chinese-prior model without Vietnamese instruction tuning, or an ablation that removes the student filter) before I'd trust it.\n\nOther minor issues: the validation protocol says 'at least two Accept votes' in Appendix A.2 but 'at least three' in Section 4.2; difficulty labels were manually rebalanced with non-reproducible thresholds; the source corpus is never named; and the paper cites no URL for the promised code/data, so 'publicly available' is unverifiable.\n\nWho is this for? Anyone building or evaluating LLMs for low-resource, culturally specific medical domains. The dataset itself is valuable even if the interpretation is not. It deserves peer review—serious referees will push for the ablations and controls, but the artifact should be published. I'd bring it to reading group and likely cite it as a resource, with the caveat about the filter.","headline":"A useful new benchmark for an underserved domain, but the headline cross-lingual claim is not yet backed because the evaluation set was filtered by the winning model family.","tokens_in":13045,"tokens_out":1861,"would_cite":true,"duration_ms":17349,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VietMed-MCQ: a 3,190-question benchmark for Vietnamese traditional medicine, built by a teacher-student consistency filter, reports 94.2% expert approval and shows Chinese-prior LLMs beating Vietnamese-specialized models by +7.21% accuracy.","keywords":["Vietnamese Traditional Medicine","multiple-choice benchmark","synthetic data generation","consistency filtering","cross-lingual transfer","retrieval-augmented generation","LLM evaluation","low-resource medical NLP"],"falsifier":"Take a fresh random sample of 500 questions drawn only from the 2,690 questions that were never human-reviewed, and have the same medical expert and four students evaluate them with the paper's rubric. If the approval rate falls well below 94.2% (or inter-rater agreement drops), the consistency gate is not selecting for correctness as strongly as claimed, and the benchmark's quality and the validity of the cross-lingual accuracy gap would be called into question.","tokens_in":12078,"feed_emoji":"🩺","tokens_out":2761,"duration_ms":28276,"temperature":0.7,"pith_summary":"The paper introduces VietMed-MCQ, the first comprehensive multiple-choice benchmark for Vietnamese Traditional Medicine, containing 3,190 questions generated from textbooks via retrieval-augmented generation and filtered by a teacher-student consistency check. A human expert and four students approved 94.2% of a 500-question sample, with substantial inter-rater agreement. Benchmarking seven open-source models, the authors find that general-purpose models with strong Chinese-language priors (Qwen2.5) outperform Vietnamese-specialized models by 7.21 accuracy points, suggesting that cross-lingual conceptual transfer from Traditional Chinese Medicine outweighs language-specific fine-tuning for culturally rooted medical knowledge. All models still struggle with complex diagnostic reasoning, with the best reaching only about 64% accuracy.","feed_headline":"Chinese-prior LLMs top Vietnamese models on VTM quiz","feed_subtitle":"A new 3,190-question benchmark shows cross-lingual transfer beats language-specific tuning in traditional medicine.","key_machinery":"The teacher-student consistency filter: a 70B teacher model (Llama-3.1-70B) generates a question, four options, answer, reasoning, and a quoted evidence span from a textbook context; a 32B student model (Qwen2.5-32B) answers the same question independently with the context provided. A candidate question is kept only if the student's answer matches the teacher's and if the evidence string is a literal substring of the source context. This dual-consensus gate filtered out 7.78% of candidates (269 of 3,459) without human labeling, and the remaining 3,190 questions form the benchmark.","core_discovery":"The paper claims to establish both a usable evaluation resource and a counterintuitive finding: models trained with Chinese-language medical content transfer conceptual knowledge to Vietnamese Traditional Medicine better than models specifically tuned for Vietnamese. The evidence is the accuracy gap on VietMed-MCQ: Qwen2.5-7B-Instruct achieves 62.01% versus 55.17% for VinaLlama and 54.80% for Vistral, with the gap persisting in 3-shot settings. The authors attribute this to shared Sino-Vietnamese terminology (e.g., Am Duong for Yin-Yang) and argue that conceptual overlap between VTM and TCM enables cross-lingual transfer that dominates simple language adaptation.","pith_inferences":["The positional bias (50.3% correct answers at option B) was mitigated by shuffling during evaluation, but the dataset's internal statistics may still reward models that exploit generation artifacts; a direct test is comparing models on shuffled vs. unshuffled answer distributions.","The cross-lingual transfer claim might be confounded by model recency and architecture: Qwen2.5-7B is a newer model than the Llama-2-based VinaLlama, so a matched-architecture comparison would isolate the Chinese-prior effect from generational improvements.","Because the 500-question human validation was a post-hoc quality check that removed no questions, the quality of the 2,690 never-human-reviewed questions rests entirely on the consistency gate; a fresh validation sample from that unexamined subset would test whether the 94.2% approval rate generalizes.","The author's hypothesis that conceptual overlap drives transfer could be tested by evaluating the same Qwen models on a Vietnamese non-medical domain where no such overlap exists, predicting smaller or absent transfer gains."],"forward_implications":["If the benchmark is sound, it provides the first standardized evaluation tool for Vietnamese Traditional Medicine, enabling future comparisons of LLMs on culturally specific medical knowledge.","The +7.21% advantage of Chinese-prior models implies that, for low-resource languages with shared conceptual roots, pretraining on a related high-resource language can matter more than fine-tuning on the target language.","The teacher-student consistency gate, despite its substring-check limitation, offers a scalable, low-cost way to filter hallucinated synthetic medical data—useful for other low-resource domains.","The ceiling of ~64% accuracy on VietMed-MCQ indicates that current open-source models lack the diagnostic reasoning needed for reliable VTM clinical support, setting a concrete target for improvement.","The paper's own observation that MCQ formats may overestimate reasoning suggests that free-response variants could show even lower model competence, reinforcing the need for harder evaluations."],"fun_headline_variants":["Chinese-prior LLMs beat Vietnamese-tuned models on VTM MCQ","VTM quiz: Chinese-prior LLMs outclass Vietnamese-tuned","Cross-lingual transfer wins in VTM: Chinese-prior LLMs top","New VTM benchmark: Chinese-prior LLMs outperform Vietnamese","On 3,190 VTM questions, Chinese-prior models lead"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The automatic consistency gate is a valid quality filter: the paper assumes that when the student model independently gives the same answer as the teacher, the question is correct—an assumption the paper itself calls 'necessary but insufficient' because the substring evidence check cannot verify reasoning, entailment, or evidence sufficiency, and because human validation removed no questions from the final dataset.","fun_headline_variants_meta":{"raw":{"variants":["Chinese-prior LLMs beat Vietnamese-tuned models on VTM MCQ","VTM quiz: Chinese-prior LLMs outclass Vietnamese-tuned","Cross-lingual transfer wins in VTM: Chinese-prior LLMs top","New VTM benchmark: Chinese-prior LLMs outperform Vietnamese","On 3,190 VTM questions, Chinese-prior models lead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000709,"raw_usage":{"total_tokens":3031,"prompt_tokens":745,"completion_tokens":2286,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2190}},"tokens_in":489,"tokens_out":2286,"duration_ms":14442,"temperature":1.0,"reasoning_tokens":2190,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T12:12:44.697010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of 500 questions drawn only from the 2,690 questions that were never human-reviewed, and have the same medical expert and four students evaluate them with the paper's rubric. If the approval rate falls well below 94.2% (or inter-rater agreement drops), the consistency gate is not selecting for correctness as strongly as claimed, and the benchmark's quality and the validity of the cross-lingual accuracy gap would be called into question.","supporting_citations":[],"review_version":1}