{"id":"d2870357-831b-4c56-b6ca-477489e181f0","arxiv_id":"2505.10507","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"With tuned design choices, word aligner based label projection matches or surpasses marker based methods, and ensembling translate-train with translate-test improves and stabilizes cross-lingual token classification.","lead":"This paper shows that small choices in word alignment based label projection strongly affect cross-lingual transfer for token classification, and that carefully tuned word aligners match or beat marker based methods. It also introduces an ensemble of translate-train and translate-test that improves accuracy and makes results less sensitive to those choices.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unquantified discontinuous-span projections could undermine the robustness claim for verb-final languages; Section 6 admits the issue without measuring its frequency or effect.","rationale":"The paper is a solid empirical contribution: it systematically isolates WA design choices, separates validation from test, releases code, and includes ablations over word aligners, MT models, and encoder models. The identified weakness is real and load-bearing but tractable. The span mapper in Section 2.1.1 relies on a single-continuous-span assumption that the authors themselves flag in Section 6 as violated for German and Dutch xSID examples, yet they do not measure how often this occurs or quantify its effect. The per-language results in Table 15 are consistent with the failure mode: in xSID T-Train, WA underperforms Codec by 2.6 F1 on German and 4.4 F1 on Dutch, while beating Codec on several other languages. If a large fraction of verb-final target slots are discontinuous, the central 'robustness' and 'matches or surpasses marker-based' claims are overstated for that language family, even if the averaged scores still look competitive. The proposed concrete test would settle the matter directly: quantify the discontinuity rate and retrain with a discontinuous-capable mapper. Because the authors already have the pipeline and code, this is a feasible condition rather than a demand for new data. I would therefore move the verdict from ACCEPT to CONDITIONAL, with the condition being the quantification and ablation described above.","tokens_in":26637,"tokens_out":11626,"duration_ms":122417,"concrete_test":"Using the released pipeline, compute the discontinuity rate on the German and Dutch xSID T-Train splits: for each gold labeled span in the source, collect its AccAlign-aligned target tokens and count how many source spans form more than one maximal contiguous segment in the translated target; do the analogous count for T-Test projections. Then implement a discontinuous-span-aware variant of the Section 2.1.1 mapper (label each maximal contiguous segment with the source span label, resetting B/I at each segment) and retrain XLM-R on the resulting T-Train data. If German and Dutch T-Train F1 move by less than about 1 point, the continuity assumption is not the driver of the observed gaps and the concern does not land; if the gaps to Easy/Codec close substantially, the paper's robustness claim needs qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The span-mapping algorithm in Section 2.1.1 assumes every labeled source span projects to a single continuous target span: the first source label is placed on the first aligned target token and the last source label on all remaining tokens in that span. The Limitations (Section 6) admit that this assumption fails for some German and Dutch xSID examples because sentence-final verbs split one slot into discontinuous spans, but the paper never quantifies how often this happens. This matters because the central claim is robustness: if discontinuous projections are frequent in verb-final languages, every affected span is systematically mislabeled in T-Train and T-Test, and the averaged F1 scores in Table 2 can hide per-language degradation. Table 15 shows the expected signature: in xSID T-Train, WA lags Codec by 2.6 F1 on German and 4.4 F1 on Dutch, well above the 0.9-point average lag, while WA wins on several non-verb-final languages. Without a count of discontinuous projections or an ablation that repairs them, the 'matches or surpasses marker-based' and 'robust' claims are not fully established for the SOV language family.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper revisits word-aligner (WA) based label projection for translation-based cross-lingual transfer (XLT) on token classification tasks. The authors systematically investigate three low-level design choices: the span-mapping algorithm, filtering strategies for discarding incomplete alignments, and pre-tokenization of the translated sentence. Using MasakhaNER 2.0 and xSID (29 languages, English as source), they show that these choices substantially affect translate-train (T-Train) and especially translate-test (T-Test) performance. With configurations selected on validation data, they report that WA-based projection matches or outperforms the marker-based methods Easy and Codec. They further propose ETT, an ensemble that averages the class logits of a T-Train and a T-Test model after projecting the latter back to the original target sentence, and show that ETT improves over both components and reduces sensitivity to the pre-tokenization choice. Additional experiments vary the base encoder/decoder model, the word aligner, and the MT system.","tokens_in":26830,"tokens_out":14074,"duration_ms":128916,"significance":"The paper is a valuable, carefully executed empirical study. It directly challenges the recent claim that marker-based label projection is inherently superior to WA-based projection, and it provides a clear recipe for configuring WA-based pipelines. The evaluation is broad and transparent: two benchmarks, 29 languages, per-language tables, multiple base models, aligners, and MT systems, and the code and data are publicly released. The proposed ETT ensemble is simple and effective, and the finding that it reduces sensitivity to design choices is practically important. The main weaknesses are an overgeneralized robustness claim and an unquantified limitation regarding discontinuous projected spans; both are addressable in revision. In its current form, the evidence supports the average 'WA matches or surpasses' claim, but the robustness claim requires either additional experiments or rephrasing.","major_comments":[{"comment":"The claim that ETT 'reduces sensitivity to low-level WA design choices' (abstract, contribution 3) and 'mitigates the observed performance variations making suboptimal filtering strategies less influential' (Limitations) is not fully supported by the experiments. Table 3 demonstrates robustness only for the pre-tokenization choice (WS-TOK vs LS-TOK) in T-Test. No experiment shows ETT with different filtering strategies (e.g., NO-FILT versus RSTR-TGT, which differ by 9.2 points for T-Test in Table 1) or with different span-mapping algorithms. Please either add an ETT ablation over the filtering strategies from Table 1, or qualify the claims so that 'low-level design choices' refers specifically to the pre-tokenization (and, where shown, MT model) dimensions.","section":"Abstract; Section 4.2, Table 3; Section 6"},{"comment":"The span-mapping algorithm assumes that every labeled source span projects to a single continuous target span, and the Limitations state that this fails for some German and Dutch xSID examples because sentence-final verbs split a slot into discontinuous spans. The paper does not quantify how often this occurs, nor does it provide an ablation that repairs discontinuous projections. This matters because Table 15 shows that WA-based T-Train lags Codec by 2.6 F1 on German and 4.4 F1 on Dutch, well above the 0.9 average lag, a pattern consistent with systematic mislabeling of affected spans. I ask the authors to report the frequency of discontinuous spans and, ideally, to evaluate a repair strategy (e.g., splitting the projected span) so that the impact on the 'matches or surpasses' and robustness claims for verb-final languages can be assessed.","section":"Section 2.1.1; Section 6; Table 15"}],"minor_comments":[{"comment":"The row headers 'AccAlginnof t' and 'Awesomenof t' are formatting errors and should read 'AccAlign (no ft)' and 'Awesome (no ft)'.","section":"Table 4"},{"comment":"'per-tokenization experiments' should be 'pre-tokenization experiments'.","section":"Appendix G"},{"comment":"The section header 'Ensembling-Translate-Train' does not match the 'Ensemble-Train-Test' (ETT) terminology used in the main text and should be renamed.","section":"Tables 14 and 15"},{"comment":"The WA configuration is selected on validation data while the marker-based baselines use published defaults; the discussion should explicitly acknowledge this asymmetry when interpreting the 'matches or surpasses' claim, since it is favorable to WA.","section":"Section 3 and Section 4.2"},{"comment":"Differences are reported as means over three seeds without significance testing; for Table 2, where several differences are 0.3-0.6 F1 points with standard deviations of 0.4-0.7, a paired test or a comment on effect size would strengthen the comparative claims.","section":"Results throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is strong and the central findings are likely correct. The two major comments are addressable: the robustness claim needs either an added ablation (ETT with different filtering strategies) or rephrasing, and the discontinuous-span issue needs a quantitative footnote or a repair ablation. I do not see circularity or missing attribution. With these additions, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a well-executed empirical study that does something overdue: it treats word alignment label projection as a method with design choices, not a fixed baseline. The main result—that an optimized WA pipeline (specific span mapping, filters, pre-tokenization) matches or beats marker-based Easy/Codec on token-level XLT—contradicts the prior claim and is backed by two benchmarks, 29 languages, multiple base models, aligners, and MT systems. The ETT ensemble (averaging T-Train and T-Test logits after projecting them to the target sentence) gives a consistent gain and also makes the pipeline much less sensitive to WA details; that robustness finding is the most useful part of the paper.\n\nThe evaluation is broad and honestly reported: per-language tables, standard deviations over three seeds, code and data released. The authors also openly flag the main assumption of their span mapping, that a labeled span maps to one continuous span. As they note, German and Dutch xSID examples can split a slot into discontinuous spans because of sentence-final verbs. The problem: they do not quantify how often this happens or what it costs. The stress-test reading of Table 15 suggests this is not a rounding error: in xSID T-Train, WA lags Codec by 2.6 F1 on German and 4.4 F1 on Dutch, while WA wins on several non-verb-final languages. That is exactly the pattern you would expect if the continuity assumption is hurting. So the paper's \"robust\" claim is a bit too strong as written; it is robust on average, across the 29 languages, but not specifically for SOV languages.\n\nOther caveats are minor: the WA configuration is tuned on validation while the marker baselines use published defaults (this asymmetry favors the WA side, though the effect is probably moderate); there are no significance tests (but standard deviations are given); and the T-Test results depend on language-specific pre-tokenization, which is a legitimate but extra moving part.\n\nOverall, the central finding holds up: WA projection is not inherently worse than marker-based projection, and ETT is a simple, reproducible improvement. The paper deserves a serious referee. I would recommend asking the authors to add a count of discontinuous projections per language and an ablation that repairs or skips them, and to discuss the German/Dutch results explicitly. That would turn a known limitation into a measured one.\n\nYes, I'd take this to reading group, and I'd cite it.","headline":"A solid empirical study showing that with proper tuning, word alignment label projection matches marker-based methods, and the ETT ensemble is a simple, robust gain; the main caveat is the unquantified discontinuous-span problem in verb-final languages.","tokens_in":27366,"tokens_out":1969,"would_cite":true,"duration_ms":18677,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimized word alignment matches or beats marker-based label projection, and an ensemble of the two translation directions outperforms it.","keywords":["cross-lingual transfer","token classification","label projection","word alignment","translate-train","translate-test","ensemble","slot labeling"],"falsifier":"Take a language with verb-final word order, or any pair where reordering splits spans, annotate gold target spans, and measure how often a single source span's aligned target tokens form two or more disconnected chunks. Run word-aligner-based translate-test and ETT only on that subset: if the F1 gap versus marker-based projection is large on discontinuous examples while small on continuous ones, the continuity assumption, not alignment quality, is what carries the result.","tokens_in":26428,"feed_emoji":"🔀","tokens_out":12463,"duration_ms":116468,"temperature":0.7,"pith_summary":"To do translation-based cross-lingual transfer on token-level tasks, a system has to move labels from the original sentence onto its translation; this paper studies the word-alignment route for that label projection. It claims that earlier comparisons against tag-based (marker) projection were unfair to word aligners, because big gains come from three details: projecting whole labeled spans instead of token-by-token labels, filtering alignments that look incomplete, and choosing how the translated sentence is pre-tokenized. With those details set well, word aligners match or beat marker-based projection on both translate-train and translate-test over 29 languages, and the paper's ensemble of the two translation directions (ETT), which averages the class logits returned on each token, outperforms the marker-based projection and shrinks the effect of the low-level choices. A sympathetic reader should care because word-alignment projection needs no special fine-tuning of the translation model, so the result makes translation-based transfer cheaper and its behavior more predictable.","feed_headline":"Ensembling both translation directions beats tag-based projection","feed_subtitle":"Averaging train- and test-side logits improves token classification across 29 languages and reduces alignment errors.","key_machinery":"The argument rests on four mechanisms. The first is span-based projection: rather than moving token labels one by one along alignments, it takes a labeled span in the source sentence, maps the BIO start tag of the first source token onto the first aligned target token, maps the continuation tag across all intervening aligned target tokens, and thereby enforces continuous well-formed spans in the translation. The second is a set of filters that discard or shrink projections whose alignments look incomplete: source tokens with no target link, target spans that are discontinuous, or span counts and types that do not match, with a dedicated restricted-target rule for single-token spans in translate-test. The third is pre-tokenization of the translated sentence with a language-specific tokenizer rather than plain whitespace. The fourth is ETT, which projects the translate-test class logits back into the target sentence and averages them with translate-train logits token-wise. These mechanisms carry the argument because they convert noisy alignments into conservative, well-formed spans and then let the two translation directions correct each other.","core_discovery":"Contrary to the conclusions of recent marker-based projection papers, optimized word-alignment projection delivers transfer performance at least on par with inserting and preserving tags around labeled spans, and occasionally better: on named-entity recognition it slightly exceeds the marker methods in translate-train, and on slot labeling it trails by less than a point. Translate-test, which translates the target sentence to English for prediction, is the sensitive setting: a language-specific pretokenizer adds up to 12.6 points on the NER benchmark, and a filtering rule that restricts single-token source spans to single target tokens recovers about 9 points over applying no filter. Translate-train, in contrast, is largely indifferent to these choices. The ETT ensemble, averaging the class logits of translate-train and translate-test for every target token, improves average F1 by about 3.1 points over translate-train and 1.5 points over translate-test on the two benchmarks, and it cuts the loss from using whitespace instead of language-specific tokenization from about 9.9 points to 1.4.","pith_inferences":["The logit-averaging trick in ETT should generalize to ensembling more than two projections, say several aligners or tokenizers at translate-test time, and would likely compress design-sensitivity further at the cost of extra translation compute; the paper only demonstrates the two-component version.","The continuity assumption sets a ceiling: for verb-final languages or free-word-order pairs where discontinuous target spans are common, ETT would need a projection that can emit multiple target spans per source span; the paper notes the phenomenon for German and Dutch but does not build that extension.","Because translate-train is insensitive to the filtering choices, a practical recipe for low-resource languages is to fix one reasonable translate-train configuration and spend all tuning budget on translate-test, then combine with ETT.","Benchmark reports of translation-based transfer should state label-projection details; otherwise, differences between word-aligner and marker-based systems are not interpretable, which is the source of the contradiction with earlier results."],"forward_implications":["Translate-test should receive the tuning effort: language-specific pre-tokenization and restricted-target filtering buy up to 12.6 points on the NER benchmark, while translate-train choices barely move the score.","ETT lifts average F1 over translate-train by about 3.1 points and over translate-test by about 1.5 points, and reduces the gap between good and bad pre-tokenization choices from roughly 9.9 to 1.4 points.","Across the reported experiments, the comparison holds with different translation engines, different word aligners, and different downstream encoders, so the result is not tied to a single model choice.","Word-aligner projection avoids the tag-preservation fine-tuning or constrained decoding that marker-based methods require, so the matched performance makes the cheaper option the default."],"supporting_citations":[{"why":"Marker-based Easy method; the translate-train baseline this work re-evaluates and claims to match or beat.","marker":"(Chen et al., 2023)"},{"why":"Marker-based Codec with constrained decoding; the translate-train and translate-test baseline this work matches or beats.","marker":"(Le et al., 2024)"},{"why":"AccAlign aligner used for the main word-alignment label projection.","marker":"(Wang et al., 2022)"},{"why":"Awesome aligner and the fine-tuning scheme used as the alternate word aligner in ablations.","marker":"(Dou and Neubig, 2021)"},{"why":"MasakhaNER 2.0 named entity recognition benchmark covering 18 languages used for evaluation.","marker":"(Adelani et al., 2022)"},{"why":"xSID slot-labeling benchmark covering 11 languages used for evaluation.","marker":"(van der Goot et al., 2021)"},{"why":"Prior study of translate-train and translate-test to low-resource languages; supplies the training recipe this paper follows and the observation that translate-test is the more sensitive route.","marker":"(Ebing and Glavaš, 2024)"},{"why":"Introduced ensembling of translate-train and translate-test predictions; the starting point for ETT.","marker":"(Oh et al., 2022)"},{"why":"XLM-R, the multilingual encoder used as the base model for downstream fine-tuning.","marker":"(Conneau et al., 2020)"},{"why":"NLLB 3.3B, the machine translation model used to produce the translated training and test data.","marker":"(Team et al., 2022)"}],"fun_headline_variants":["Tuned alignments match marker-based cross-lingual transfer","Ensembling translations beats tag projection for token tasks","Word alignment rivals markers when optimized for transfer","Alignment tuning closes gap with marker-based projection","ETT ensemble outperforms marker projection in XLT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one labeled span in the original sentence always corresponds to one continuous labeled span in the translation; the authors concede this fails for some German and Dutch examples, where sentence-final verbs split a slot into disjoint pieces, so frequent discontinuous projections would break the projection and the ensemble.","fun_headline_variants_meta":{"raw":{"variants":["Tuned alignments match marker-based cross-lingual transfer","Ensembling translations beats tag projection for token tasks","Word alignment rivals markers when optimized for transfer","Alignment tuning closes gap with marker-based projection","ETT ensemble outperforms marker projection in XLT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1367,"prompt_tokens":1068,"completion_tokens":299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":225}},"tokens_in":684,"tokens_out":299,"duration_ms":3731,"temperature":1.0,"reasoning_tokens":225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:08:02.081244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a language with verb-final word order, or any pair where reordering splits spans, annotate gold target spans, and measure how often a single source span's aligned target tokens form two or more disconnected chunks. Run word-aligner-based translate-test and ETT only on that subset: if the F1 gap versus marker-based projection is large on discontinuous examples while small on continuous ones, the continuity assumption, not alignment quality, is what carries the result.","supporting_citations":[],"review_version":1}