{"id":"0251a93e-a2c7-4c8a-8e50-be4a5b7343a8","arxiv_id":"2411.17270","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A simplified HPSG neural parser with PhoBERT reaches 82.34 F on Vietnamese constituency parsing, but the result relies on randomly permuted training trees and no released code.","lead":"The authors adapt a neural HPSG parser from English to Vietnamese by swapping in PhoBERT and XLM-RoBERTa encoders, and they report 82.34% F-score for constituency parsing on VietTreebank plus a 89.04 private-test F-score on the VLSP 2023 shared task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claims for VietTreebank/VnDT rest on training data randomly permuted without linguistic constraints; without an unpermuted-data HPSG baseline, the improvement cannot be attributed to the architecture.","rationale":"The reader's weakest_assumption identifies the random permutation of non-compliant trees as the load-bearing premise. I agree: the paper's VietTreebank and VnDT results are the primary evidence for the transfer of the HPSG architecture, and those results come from training on modified data while all baselines use the original data. This is a textbook confound. The concern is concrete and verifiable: an ablation with the original unpermuted data would settle whether the permutation is responsible for the improvement. I considered whether the VLSP 2023 result (89.04 F1) is more load-bearing, since that treebank was not permuted; however, that result uses the parser's own head rules to generate dependency annotations, and the gain over Stanza is only 0.31 F1 with no significance testing, so it is weaker evidence overall. The permutation issue affects the most substantial and reproducible-looking gains (1.79 F1 over the best self-attentive baseline and a higher UAS), so it is the single most important threat to the central claim. The paper itself admits the lack of linguistic constraints and the labels-preserved arc permutation, which strengthens the likelihood that the modification changes the task in a way that is not equivalent to training on the original treebank. The proposed test is one that the authors can run with their existing code and data, and it directly addresses the gap. The reader's CONDITIONAL verdict is appropriate: the paper is a useful empirical report but the SOTA claim should remain conditional on the release of an unpermuted-data ablation. No new objection beyond the reader's is raised, so the verdict should be UNCHANGED.","tokens_in":12223,"tokens_out":6162,"duration_ms":58146,"concrete_test":"Retrain the same HPSG Neural Parser (PhoBERT_large, POS tags, λ=0.9, identical hyperparameters) on the original VTB/VnDT training and development sets, handling the ~1,000 non-HPSG-compliant tree pairs by excluding them (or by a deterministic, linguistically motivated repair) rather than random permutation. Evaluate on the identical test splits and compare F1 and UAS against the permuted-data model. If the original-data model achieves an F1 within, say, 0.5 points and a UAS within 0.5 points of the permuted model, the SOTA claim is robust to the modification. If the permuted model is significantly better, the reported gains are an artifact of the permutation, and the paper should be reframed accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline results on VietTreebank and VnDT (F1 82.34, UAS 85.73) are produced by an HPSG parser trained on a modified corpus: roughly 1,000 constituent/dependency tree pairs that violated simplified HPSG rules were 'randomly permuted ... not bound by linguistic constraints' (§5.2). The baselines in Tables 1 and 2 (Self-Attentive, PhoNLP, Biaffine) were trained on the original, unmodified treebanks. Because the training distributions differ, the comparison is not controlled: any gain could stem from the permutation itself, which may act as a form of data augmentation or introduce systematic biases that coincidentally favor the HPSG decoder's inductive biases, rather than from the simplified HPSG formalism. The paper provides no ablation training the HPSG parser on the original, unpermuted data (or with non-compliant trees excluded). The claim in §5.2 that the permutation's 'impact ... was minimal' is based on tuning the loss weight λ, not on comparing permuted versus unpermuted training data, so it does not address the confound. The lower LAS reported by the authors is attributed to 'arc permutation without changing the original labels,' which acknowledges that the modification alters dependency structure in a linguistically unprincipled way; the higher UAS reported alongside could therefore be an artifact of the same modification rather than evidence of HPSG's effectiveness. Without an unpermuted-data baseline, the central claim that the simplified HPSG architecture transfers to Vietnamese is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts the joint-span HPSG parser of Zhou and Zhao (ACL 2019) to Vietnamese by replacing its word encoders with PhoBERT and XLM-RoBERTa and by modifying the training data. Because roughly 15% of the constituent/dependency tree pairs in VietTreebank (VTB) and VnDT do not satisfy the simplified HPSG constraints, the authors randomly permute the non-compliant pairs in the training and development sets without linguistic constraints. They report a constituency F1 of 82.34 on the VTB test set, a dependency UAS of 85.73 on VnDT, and an F1 of 89.04 on the VLSP 2023 private test, claiming state-of-the-art results. For the VLSP 2023 treebank, which has no dependency annotation, the authors design head-percolation rules and use ClearNLP to convert constituency trees into dependency trees. The paper's central claim is that this simplified-HPSG parser outperforms prior Vietnamese parsers when trained on the modified corpora.","tokens_in":12540,"tokens_out":5357,"duration_ms":50924,"significance":"If the reported results were obtained under controlled conditions, the paper would provide a useful data point that the Zhou-Zhao joint-span HPSG architecture transfers to Vietnamese when combined with PhoBERT-large, and it would support the authors' suggestion that treebank construction should pay more attention to simplified HPSG constraints. The paper has several strengths: the test sets are left unaltered, the models are run five times, baseline systems are replicated in some cases, and the authors are transparent about the non-linguistic nature of their data modifications and head rules. Code and toolkit links are provided. The weakness is that the headline comparisons are not controlled: the HPSG parser is trained on randomly permuted data while the baselines are trained on the original treebanks, and no ablation separates the effect of the data modification from the effect of the architecture. The VLSP 2023 claim also rests on an unvalidated constituency-to-dependency conversion. These issues are fixable with additional experiments, but they are load-bearing for the stated state-of-the-art conclusions.","major_comments":[{"comment":"The HPSG parser is trained on data in which roughly 1,000 non-compliant tree pairs were \"randomly permuted ... not bound by linguistic constraints,\" while the Self-Attentive, Biaffine, and PhoNLP baselines were trained on the original treebanks. This changes the training distribution, so the reported F1 and UAS gains cannot currently be attributed to the simplified HPSG architecture rather than to the permutation itself. The λ sweep in Figure 3 only varies the loss weight and does not compare permuted versus unpermuted training data, so the statement in §5.2 that the permutation's impact was \"minimal\" is not supported. I request an ablation training the same HPSG parser on (a) the original, unpermuted data and (b) data with non-compliant pairs removed, reporting F1, UAS, and LAS for both conditions.","section":"§5.2, Tables 1–2"},{"comment":"The VLSP 2023 result depends on head-percolation rules written \"by individuals with a non-linguistic engineering background\" and on a ClearNLP conversion, but no validation of the resulting dependency trees is reported. Because the HPSG parser requires dependency supervision and the 89.04 private-test F1 exceeds the Stanza parser by only 0.31 points, the quality of this conversion could be decisive for the claimed advantage. Please compare the converted dependencies against an existing rule set or gold-standard dependencies, or provide a manual error analysis of the converted trees.","section":"§5.2, Figure 4, Table 3"},{"comment":"The headline numbers are presented without variance, even though the models are run five times. Additionally, Table 2 marks a result as \"significantly different\" when the lowest of five HPSG runs is compared with the replicated PhoNLP score using an unspecified paired t-test; selecting the minimum run before testing is not a valid significance procedure. Please report mean ± standard deviation over all five seeds for every row, and use a proper paired test (e.g., bootstrap or matched-seed comparison) for claims of statistical significance.","section":"Tables 1 and 2"},{"comment":"The paper is inconsistent about which data were permuted: §3.1 says \"samples from the training and development sets were adjusted\" for the VTB and VnDT corpora, while §5.2 says \"our modifications were limited to the training and development subsets of the VnDT dataset.\" This distinction matters because it determines whether the VTB constituency F1 comparison is also confounded by the modified training data. Please clarify whether the constituency trees in the VTB training set were modified or left untouched.","section":"§3.1 vs. §5.2"}],"minor_comments":[{"comment":"The text says the system \"scored zero on categories like Nby and ADJb,\" but Figure 5(a) shows Nby at 8.00% and ADJb at 38.79%; please correct the mismatch.","section":"§5.4, Figure 5(a)"},{"comment":"The sentence \"Nguyễn et al. [25] introduce our project on developing a Vietnamese lexicon\" is confusing, because reference [25] is a lexicon paper and not the current project; please rephrase.","section":"§3.1, reference [25]"},{"comment":"The URL path contains a duplicated suffix \"run_constituency .py .py\"; please fix the typo.","section":"Footnote 6"},{"comment":"The head-percolation rules are printed as one long, low-resolution rule string; please provide a readable table of the rules or move them to an appendix.","section":"Figure 4"},{"comment":"The caption says predicted POS tags were generated with the VnCoreNLP toolkit, while §5.2 describes a fine-tuned Stanza tagger with PhoBERT-large; please clarify which tagger produced the POS tags used in Table 1 and Table 2.","section":"Table 1, caption and §5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is more like a workshop or system-description contribution than a journal article in its current form, but the core issue is experimental control rather than a fatal flaw. If the authors can provide an unpermuted-data ablation, validate the VLSP 2023 conversion, and report proper variance and significance measures, the empirical claims would become assessable. I would not recommend acceptance without those additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one-sentence take: this is an honest attempt paper with a real confound at the center of its SOTA claim.\n\nWhat is actually new: applying Zhou and Zhao's joint-span HPSG parser to Vietnamese with PhoBERT/XLM-R; documenting that ~15% of VTB/VnDT tree pairs violate simplified HPSG; and showing 89.04 F1 on the VLSP 2023 private test, a modest but real win over Stanza. The evaluation itself looks genuine: untouched test set, five runs, paired t-test against a replicated PhoNLP baseline, and a thoughtful analysis of tag-wise errors. That part deserves credit.\n\nThe soft spot is the one the authors themselves describe in §5.2: to make the training data HPSG-compliant, they randomly permuted about 1,000 constituent/dependency tree pairs 'not bound by linguistic constraints.' Test sets were untouched, so the comparison to baselines trained on the original treebanks is not apples-to-apples. The gain could come from the permutation acting as data augmentation or from distribution shift that happens to favor the HPSG decoder. The text claims the impact 'was minimal' but that conclusion is drawn from tuning λ, not from comparing permuted vs unpermuted training data. There is no ablation, no release of the modified corpus, no code. Without those, the 82.34 F1 and 85.73 UAS cannot be attributed to simplified HPSG.\n\nOther soft spots: Table 1 has no variance bars, so we don't know if the 82.34 vs 80.55 gap is meaningful across runs; the VLSP head-percolation rules were written by engineers, not linguists, which the authors admit; and the lower LAS is dismissed rather than explained. These are real but secondary.\n\nOverall: the paper is a reasonable research report of an attempted transfer, and it is unusually candid about its own limitations. The central SOTA claim is currently unsupported, but the failure is fixable: retrain on the original treebanks (or with the non-compliant trees excluded), release the modified data, and report variance. As is, I would not cite the SOTA numbers, but I would engage with the paper as a serious empirical study that deserves referee time. My call: send to peer review, require the ablation and data release before accepting.","headline":"Honest transfer attempt whose headline SOTA numbers are confounded by unconstrained random permutation of training data; fixable with an ablation and data release.","tokens_in":13074,"tokens_out":1711,"would_cite":false,"duration_ms":16325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simplified HPSG neural parser, adapted to Vietnamese with PhoBERT, is claimed to reach 82.34% constituency F1 on VietTreebank and 89.04% on the VLSP 2023 private test, with lower LAS attributed to label-preserving random permutations…","keywords":["Vietnamese parsing","HPSG","constituency parsing","dependency parsing","PhoBERT","VietTreebank","VnDT","VLSP 2023"],"falsifier":"Retrain the same parser using linguist-corrected versions of the roughly one thousand tree pairs that were randomly permuted: if LAS does not rise above the reported 78.42 or F1 drops below 82.34, the paper's account of the repair's effect is falsified.","tokens_in":12027,"feed_emoji":"📊","tokens_out":14053,"duration_ms":114442,"temperature":0.7,"pith_summary":"This paper tries to establish that a simplified, joint-span Head-Driven Phrase Structure Grammar parser, originally built for English, can be made to work for Vietnamese by replacing its encoder with PhoBERT or XLM-RoBERTa and by repairing the roughly 15% of VietTreebank and VnDT tree pairs that violate the grammar's constraints. The repair is done by randomly permuting the offending training and development trees, keeping the original dependency labels untouched because no linguistic expert was consulted. The experiments report an 82.34% constituency F-score on the VietTreebank test set, an 89.04% F-score on the VLSP 2023 private test (above the Stanza baseline's 88.73%), and a UAS of 85.73 on VnDT, with lower LAS attributed to the label-preserving permutation. The paper concludes that simplified HPSG should receive more linguistic-expert attention when Vietnamese treebanks are built.","feed_headline":"89.04% F1: simplified HPSG parser beats Stanza on Vietnamese","feed_subtitle":"The same joint-span architecture reaches 82.34% constituency F1 on VietTreebank and 85.73 UAS on VnDT.","key_machinery":"The load-bearing object is the joint-span simplified HPSG tree: a constituent-style tree in which every span carries both a syntactic category and a head word, so dependency arcs are embedded in the same span structure. The parser scores spans and head-arcs with a biaffine attention model over self-attentive token representations built from PhoBERT or XLM-RoBERTa, character, word, and POS embeddings, and decodes with dynamic programming. The paper's adaptation also adds a pre-processing repair: roughly one thousand non-compliant constituency/dependency tree pairs in the training and development splits are randomly permuted so that every pair satisfies the simplified HPSG constraints. That repair, plus the pretrained Vietnamese encoders, is what carries the claimed transfer from Penn Treebank to Vietnamese.","core_discovery":"The paper claims that a joint-span simplified HPSG neural parser, originally built for the Penn Treebank, can be transferred to Vietnamese by swapping in PhoBERT or XLM-RoBERTa encoders and by repairing the ~15% of VietTreebank/VnDT tree pairs that violate simplified HPSG rules through random permutation of training and development items. In the authors' experiments this adaptation reaches an 82.34% constituency F-score on the VietTreebank test set with predicted POS tags, above the self-attentive PhoBERT-large baseline's 80.55%, and an 89.04% F-score on the VLSP 2023 private test, slightly above Stanza's 88.73%. On dependency parsing it reports a UAS of 85.73 on VnDT, higher than the PhoNLP baseline, while LAS stays lower, which the authors attribute to keeping original dependency labels untouched during permutation. The paper's conclusion is that simplified HPSG deserves more linguistic-expert attention when building Vietnamese treebanks.","pith_inferences":["If the random-permutation repair is as benign as the development curves suggest, the same preprocessing could be tried on other languages whose treebanks pair constituency and dependency annotations, with the 15% violation rate as a natural diagnostic.","The paper's LAS result points to a direct follow-up: re-run the same architecture on the permuted spans but with linguist-corrected head and dependency labels; if LAS rises without hurting UAS, the label-preserving choice is the cause.","Because the VLSP 2023 head rules were written by non-linguists, an expert-written version of those rules is a testable way to see whether the HPSG parser's margin over Stanza widens further on out-of-domain data.","A more skeptical probe would compare per-tree error patterns on the original versus permuted training trees; if the parser performs worse on sentences whose trees were permuted, the repair is introducing a systematic bias rather than a neutral fix."],"forward_implications":["With PhoBERT-large and predicted POS tags, the joint-span HPSG parser reaches 82.34% constituency F1 on the VietTreebank test set, above the self-attentive PhoBERT-large baseline's 80.55%.","On the VLSP 2023 private test the same model scores 89.04% F1, beating the Stanza baseline's 88.73%, while the two are nearly tied on the public test (86.05% vs 85.87%).","On VnDT dependency parsing the parser reports UAS 85.73 with PhoBERT-large, above the PhoNLP baseline's 84.95, but lower LAS, which the authors attribute to keeping original dependency labels while randomly permuting arcs.","Tuning the constituency/dependency loss weight to 0.9 on the development set indicates that the random-permutation repair did not disrupt the balance between the two parsing tasks."],"supporting_citations":[{"why":"Supplies the joint-span simplified HPSG parser architecture and constraints that the paper modifies for Vietnamese.","marker":"[13]"},{"why":"Supplies PhoBERT, the Vietnamese pretrained encoder used for token representation and for the POS tagger.","marker":"[7]"},{"why":"Supplies XLM-RoBERTa, the alternative cross-lingual encoder tested in the experiments.","marker":"[8]"},{"why":"Provides the self-attentive constituency parser baselines with pre-training that the HPSG parser is compared against.","marker":"[29]"},{"why":"Provides the Stanza constituency parser baseline and the PhoBERT-tuned POS tagger used in the VLSP 2023 setup.","marker":"[14]"},{"why":"Supplies the PhoNLP dependency parser baseline whose LAS and UAS the paper compares on VnDT.","marker":"[30]"},{"why":"Supplies VnCoreNLP, the tool that generated the predicted POS tags used in the VietTreebank/VnDT experiments.","marker":"[28]"},{"why":"Supplies the VietTreebank corpus, the constituency dataset used for training and evaluation.","marker":"[16]"},{"why":"Defines the VLSP 2023 Vietnamese constituency parsing shared task and its private test set.","marker":"[15]"}],"fun_headline_variants":["Simplified HPSG parser beats Stanza on Vietnamese","89% F1: simplified HPSG tops Vietnamese parsing","Vietnamese HPSG parser: 82% F1, SOTA on constituency","Neural HPSG parser for Vietnamese outperforms baselines","HPSG parser for Vietnamese: new SOTA in constituency parsing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that randomly permuting the roughly one thousand non-compliant tree pairs in the training and development sets, without linguistic review, produces valid training trees rather than noise that quietly trains the parser to accept ungrammatical structures.","fun_headline_variants_meta":{"raw":{"variants":["Simplified HPSG parser beats Stanza on Vietnamese","89% F1: simplified HPSG tops Vietnamese parsing","Vietnamese HPSG parser: 82% F1, SOTA on constituency","Neural HPSG parser for Vietnamese outperforms baselines","HPSG parser for Vietnamese: new SOTA in constituency parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3269,"prompt_tokens":1042,"completion_tokens":2227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":2137}},"tokens_in":658,"tokens_out":2227,"duration_ms":14898,"temperature":1.0,"reasoning_tokens":2137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:19:08.690733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same parser using linguist-corrected versions of the roughly one thousand tree pairs that were randomly permuted: if LAS does not rise above the reported 78.42 or F1 drops below 82.34, the paper's account of the repair's effect is falsified.","supporting_citations":[{"cited_title":"Head-Driven Phrase Structure Grammar Parsing on Penn Treebank","cited_arxiv_id":null,"evidence_quote":"Supplies the joint-span simplified HPSG parser architecture and constraints that the paper modifies for Vietnamese."},{"cited_title":"PhoBERT: Pre-trained language models for Vietnamese","cited_arxiv_id":null,"evidence_quote":"Supplies PhoBERT, the Vietnamese pretrained encoder used for token representation and for the POS tagger."},{"cited_title":"Unsupervised Cross-lingual Representation Learning at Scale","cited_arxiv_id":null,"evidence_quote":"Supplies XLM-RoBERTa, the alternative cross-lingual encoder tested in the experiments."},{"cited_title":"An Empirical Study for Vietnamese Constituency Parsing with Pre-training","cited_arxiv_id":null,"evidence_quote":"Provides the self-attentive constituency parser baselines with pre-training that the HPSG parser is compared against."},{"cited_title":"Stanza: A Python Natural Language Processing Toolkit for Many Hu- man Languages","cited_arxiv_id":null,"evidence_quote":"Provides the Stanza constituency parser baseline and the PhoBERT-tuned POS tagger used in the VLSP 2023 setup."},{"cited_title":"PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and de- pendency parsing","cited_arxiv_id":null,"evidence_quote":"Supplies the PhoNLP dependency parser baseline whose LAS and UAS the paper compares on VnDT."},{"cited_title":"VnCoreNLP: A Vietnamese Natural Language Processing Toolkit","cited_arxiv_id":null,"evidence_quote":"Supplies VnCoreNLP, the tool that generated the predicted POS tags used in the VietTreebank/VnDT experiments."},{"cited_title":"Building a Large Syntactically-Annotated Cor- pus of Vietnamese","cited_arxiv_id":null,"evidence_quote":"Supplies the VietTreebank corpus, the constituency dataset used for training and evaluation."},{"cited_title":"VLSP 2023 chal- lenge on Vietnamese Constituency Parsing","cited_arxiv_id":null,"evidence_quote":"Defines the VLSP 2023 Vietnamese constituency parsing shared task and its private test set."}],"review_version":1}