{"id":"35417403-5135-45b6-a791-52a3de0676d4","arxiv_id":"1909.00444","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A discriminative alignment module, trained on 1.7K to 4.9K labeled sentence pairs and plugged into a Transformer MT model, outperforms FastAlign and attention baselines by 11 to 27 F1 points and improves projected NER.","lead":"This paper trains a neural word alignment model on a few thousand labeled sentence pairs and shows it beats standard unsupervised aligners by large margins on English-Chinese and English-Arabic. The alignments also improve a downstream named-entity recognition task in Chinese, making the approach useful for building datasets in low-resource languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training and evaluation share the same GALE gold alignments that footnote 5 admits systematically deviate from the protocol; the reported 11-27 point F1 gains may partly encode annotator-specific noise rather than true alignment quality.","rationale":"The strongest_claim requires DiscAlign to produce objectively better alignments, not merely better matches to GALE's particular annotation decisions. The most load-bearing condition is the validity of the gold labels, because the same labels supervise and evaluate the model. The paper itself, in footnote 5, admits systematic deviations from the alignment protocol, which is a self-identified limitation that the review must weigh. If those deviations are systematic, the supervised model can score high by emulating them, while the unsupervised baselines cannot, so the reported F1 gains could be biased upward. The human annotation experiment is real supporting evidence but is not decisive: untrained annotators may share natural alignment preferences with GALE annotators without matching the protocol, and their scores were computed against the same possibly biased gold. Other concerns, such as missing code, lack of significance testing, and the unsupported ablation claim, affect reproducibility and confidence but would not, if fixed, explain away genuine alignment gains. The gold-label issue is the one that could invalidate the central empirical claim while leaving the experimental setup intact. The proposed re-annotation test would settle the concern: either the DiscAlign margin persists on independent gold labels, or it collapses. Because the reader already assigned CONDITIONAL and this is an addressable concern rather than a fatal flaw, the reader's verdict should remain unchanged, typically subject to the condition that independent validation of the gold labels be provided.","tokens_in":1023,"tokens_out":2403,"duration_ms":72883,"concrete_test":"Sample 100-200 sentences from each GALE test split and have two trained annotators re-annotate them following the published GALE alignment protocol, or alternatively apply the corrections implied by the systematic deviations described in footnote 5. Recompute the F1 scores in Tables 2 and 3 for DiscAlign, FastAlign, and average attention against the corrected gold labels. If DiscAlign's margin over FastAlign drops by more than about 5 F1 points on the corrected labels, the headline improvement is substantially an artifact of shared noisy training and evaluation labels; if the margin persists, the concern is resolved. Optionally rerun the Section 5 NER projection with the corrected alignments to check whether the downstream gain survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires GALE gold alignments to be a faithful measure of alignment quality. Section 4.2 acknowledges mismatched bitext and formatting errors, and footnote 5 adds that a bilingual speaker identified several systematic deviations from the alignment protocol by GALE annotators. These same labels supervise DiscAlign (via the binary cross-entropy loss in Section 3) and evaluate every intrinsic F1 score in Tables 2 and 3. A supervised model can inflate reported F1 by learning the systematic deviations of the annotation protocol, while FastAlign and average attention never see labels during training, so they cannot. Thus part of the margin may reflect fitting label noise rather than genuinely better alignment. The human annotation experiment in Section 6 partially mitigates this: untrained L2 annotators score close to DiscAlign against GALE gold (73.92 vs 73.31 F1), so the labels are not arbitrary. However, those annotators received no GALE protocol training and were not tested for agreement with that protocol, so their agreement does not prove the systematic deviations are harmless. The downstream NER claim in Table 4 is also evaluated on OntoNotes NER spans, not independently re-annotated alignments, so label-induced bias that correlates with named entities would propagate. This is the assumption whose failure would explain the headline gains without an actual improvement in alignment quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiscAlign, a supervised discriminative alignment module integrated into a Transformer-based NMT system. The model projects contextual encoder and decoder states into a shared space via a three-layer MLP, computes pairwise dot-product similarity, applies a 3x3 convolution over the resulting alignment matrix, and trains each source-target pair with binary cross entropy against gold alignments. It is initialized from a pretrained MT model and finetuned on 1,687 Arabic and 4,871 Chinese annotated sentence pairs from GALE. The authors report intrinsic F1 gains of roughly 11-27 points over FastAlign and averaged-attention baselines on English-Chinese and English-Arabic test sets, an extrinsic English-to-Chinese NER projection experiment in which DiscAlign improves projected-data NER F1 by about 14 points over FastAlign, and a small human annotation study in which untrained L2 Chinese speakers annotate alignment at 4.4 sentences per minute with F1 comparable to DiscAlign.","tokens_in":11850,"tokens_out":5577,"duration_ms":57905,"significance":"If the empirical claims are reliable, this is a useful and practical contribution. The paper shows that a relatively simple supervised alignment layer, trained on only a few thousand labeled sentence pairs, can substantially outperform unsupervised aligners, and it validates the improvement on a downstream NER projection task. The two-language evaluation, the ablation showing that labeled alignment data matters more than additional unlabeled bitext, and the explicit human annotation feasibility study are notable strengths. The authors are also transparent about known quality problems in the GALE data. The main weaknesses are that the intrinsic evaluation relies on the same GALE labels used for training and that no uncertainty quantification is provided for the headline differences; these need to be addressed before the central claims can be fully accepted.","major_comments":[{"comment":"The GALE alignments used both to train DiscAlign and to compute every intrinsic F1 score are acknowledged to contain mismatched bitext, formatting errors, and several systematic deviations from the alignment protocol by GALE annotators. Because FastAlign and averaged attention never see these labels during training, a supervised model can inflate its reported F1 by learning annotator-specific regularities rather than true alignment quality. The human annotation experiment partially mitigates this concern, but those annotators were not trained on the GALE protocol and were evaluated against the same GALE gold labels, so it does not establish that the systematic deviations are harmless. I would like to see an evaluation on a small independently re-annotated clean test set, or an error analysis isolating the protocol-deviation cases, to demonstrate that the 11-27 point gains are not partly an artifact of fitting label noise.","section":"Section 4.2, footnote 5; Tables 2 and 3"},{"comment":"The headline F1 improvements rest on single point estimates with no variance information and no significance testing. The decision thresholds alpha are tuned on a 100-sentence validation subset, and the paper does not report the number of random restarts or seeds used. Given the small test sets (315 and 636 sentences), some of the reported gaps could be within noise. Please report confidence intervals via bootstrapping or across multiple seeds, and provide significance tests for the comparisons against FastAlign.","section":"Section 4.3, Tables 2 and 3"},{"comment":"The feasibility claim for manual alignment elicitation rests on only six annotators, with average scores reported but no per-annotator breakdown, no inter-annotator agreement, and no indication of how many sentences were annotated or how sentence lengths varied across the two partitions. Since this experiment is used in the conclusion to support a call for further annotation efforts, please report annotator-level precision/recall/F1, agreement statistics, and the amount of data annotated.","section":"Section 6, Table 7"}],"minor_comments":[{"comment":"The binary cross entropy loss is written as a sum over i<N, j<M, which excludes the last source and target positions; the intended range is presumably i<=N and j<=M, and the notation should be corrected.","section":"Section 3, loss function"},{"comment":"The precision-recall curves are presented without threshold values on the axes or a legend, making it difficult for the reader to connect specific alpha values to the plotted operating points.","section":"Figure 2(b)"},{"comment":"The Arabic data is described only as sourced from 'local resources'; specifying the corpus or release would improve reproducibility.","section":"Section 4.2"},{"comment":"The sentence 'The data was pre-processed in following the same procedure as Section 4' is missing a word; it should read 'pre-processed following the same procedure'.","section":"Section 5"},{"comment":"The annotation speed of 4.4 sentences per minute is reported without a measure of sentence length or token count, which makes the speed figure difficult to interpret across datasets.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely publishable after revision. The most serious issue is the dependence of the intrinsic evaluation on the same noisy GALE labels used for training; the NER projection and human annotation experiments provide partial support but do not fully resolve the concern. I would not reject, but I would ask for additional validity evidence and uncertainty quantification before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a real contribution. The authors build a discriminative alignment module on top of a Transformer MT model, train it on a few thousand gold-aligned sentences, and show large F1 gains over FastAlign and averaged attention on English-Chinese and English-Arabic. They also demonstrate downstream NER projection gains and run a small human annotation experiment. That is more than most alignment papers do, and the core idea—shared MLP projection, dot-product link matrix, convolutional smoothing—is simple and sensible. The improvements are consistent across two typologically different language pairs and across evaluation conditions, which makes me believe the effect is real rather than a fluke of one dataset.\n\nThe soft spot is exactly where the stress-test note points: the GALE gold alignments are used both to train the model and to compute every intrinsic F1 score, and footnote 5 admits that a bilingual speaker found systematic deviations from the alignment protocol in that gold data. A supervised model can learn those deviations, while FastAlign and attention never see the labels, so part of the 11–27 point margin could reflect fitting annotator bias rather than genuinely better alignment. The human annotation experiment partially mitigates this: untrained L2 annotators score about as well as DiscAlign against the same gold, which suggests the labels are not arbitrary. But those annotators were not trained to the GALE protocol and their agreement does not prove the systematic deviations are harmless. The NER projection results are on OntoNotes spans, not on re-annotated alignments, so label-induced bias could propagate there too.\n\nOther issues are more minor. There is no significance testing or variance reporting, thresholds are tuned on a 100-sentence dev subset, and code is not released. The ablation claim about convolution is stated but not shown in a table. None of this is fatal. The central argument holds up in broad terms: supervised alignment with a modest amount of labeled data does beat unsupervised baselines, and it does help projection. The GALE gold quality issue is real and should be addressed in revision, ideally by re-evaluating on a cleanly re-annotated subset or by analyzing where the systematic deviations occur and whether they explain part of the gap.\n\nThis paper deserves a serious referee. It is a within-subfield result with practical value, and the limitations are addressable rather than structural. I would conditionally accept after a round that includes significance testing, a robustness check on the gold labels, and ideally code release. For a reader, the paper is worth reading for the architecture and the projection results, with a skeptical eye on the absolute F1 numbers.","headline":"Solid, useful supervised alignment paper with a real contribution, but the headline F1 gains rest on GALE gold labels that the authors themselves admit are noisy, so the numbers should be read with caution.","tokens_in":727,"tokens_out":970,"would_cite":true,"duration_ms":20711,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A discriminative alignment module trained on a few thousand gold labels beats FastAlign and attention by 11 to 27 F1, and the better alignments lift projected Chinese NER.","keywords":["word alignment","discriminative alignment","neural machine translation","annotation projection","named entity recognition","low-resource NLP","Transformer"],"falsifier":"Re-annotate a random sample of the GALE Chinese and Arabic test sentences with independent, strictly protocol-checked annotators and compare DiscAlign, FastAlign, and attention on those cleaned labels; if the reported 11 to 27 point F1 advantage shrinks or disappears, the gains are partly artifacts of learning noisy gold labels rather than true alignment quality.","tokens_in":11418,"feed_emoji":"🔗","tokens_out":8334,"duration_ms":58150,"temperature":0.7,"pith_summary":"The paper argues that word alignment can be recovered as a first-class supervised task inside a neural machine translation model, rather than read off from attention. With only 1,687 Arabic and 4,871 Chinese human-annotated sentence pairs, their discriminative alignment module reaches 82.74 F1 on English-Arabic and 73.41 F1 on English-Chinese test sets, beating FastAlign by 27 and 11 points, respectively, and far exceeding averaged Transformer attention. The paper then shows these gains are not just intrinsic: Chinese NER trained on tags projected with the better alignments scores about 14 points higher than tags projected with FastAlign. This matters because alignment is the bottleneck for cross-lingual dataset projection, which lets speakers of high-resource languages build labeled data in low-resource languages.","feed_headline":"Discriminative alignment beats FastAlign by up to 27 F1","feed_subtitle":"Trained on fewer than 5K gold-labeled sentence pairs, the model also lifts projected Chinese NER by 14 F1.","key_machinery":"The central object is a trainable alignment matrix built from cross-lingual contextual representations: source and target states are mapped through a shared three-layer tanh network into a common space, and their dot-product similarity gives an N by M link matrix. A learned 3x3 convolution over this matrix conditions each alignment decision on its neighbors; the convolved scores pass through a sigmoid and are trained with binary cross-entropy against gold alignments. Without the convolution, F1 stays below 50, so the convolution is what turns independent pairwise similarities into coherent alignment decisions.","core_discovery":"The paper's model, DiscAlign, treats every source-word and target-word pair as a binary classification decision, supervised directly by gold alignments. Encoder and decoder hidden states of a pretrained Transformer are projected into a shared space with a three-layer feed-forward network; their dot products form an alignment score matrix; a 3x3 convolution makes each decision depend on neighboring alignment decisions; and a sigmoid converts the scores to probabilities. The model is pretrained on unlabeled bitext and fine-tuned on the gold alignments. On GALE test data, it scores 73.41 F1 on Chinese and 82.74 F1 on Arabic, versus 62.02 and 56.11 for FastAlign and 40.65 and 13.42 for averaged attention. Projecting English OntoNotes NER tags through the model's alignments yields a Chinese NER model at 51.76 F1 versus 37.77 with FastAlign projection.","pith_inferences":["Editorial inference: if the gold alignments are systematically noisy, part of the measured gain may be the model learning the annotators' protocol, including its deviations; re-annotating a test sample under a stricter protocol would isolate true alignment quality.","Editorial inference: the authors' focus on NER spans is a special case, and the same projection pipeline is task-agnostic, so part-of-speech tags, semantic roles, or coreference labels could be projected with comparable expected gains.","Editorial inference: the large impact of labeled data over bitext suggests that collecting a few thousand alignment labels for a new language pair may be more cost-effective than adding millions of parallel sentences.","Editorial inference: because untrained L2 speakers reached near-system F1 on the same sentences, a modest amount of annotator training on recall could plausibly make human-elicited alignments a scalable training resource."],"forward_implications":["Even roughly 5,000 labeled alignments can beat unsupervised alignment by double digits in typologically divergent language pairs, with F1 gains of 11 for Chinese and 27 for Arabic.","Higher alignment quality transfers to downstream projection: projected Chinese NER improves by about 14 F1 over FastAlign projection, and projected data helps most when gold data is scarce, such as only 500 sentences.","Labeled alignment data matters more than unlabeled bitext: halving labeled data costs 9.44 F1, while halving pretraining bitext costs only 3.69 F1.","Because the alignment module is architecture-neutral and works with subword units, it can be applied to other encoder-decoder architectures, and its decision threshold can tune precision versus recall for different tasks."],"supporting_citations":[{"why":"Defines the generative IBM alignment models that FastAlign reparameterizes and that the paper's discriminative model replaces.","marker":"(Brown et al., 1993)"},{"why":"Provides FastAlign, the main unsupervised baseline and the macro-F1 scorer used in all intrinsic evaluations.","marker":"(Dyer et al., 2013)"},{"why":"Supplies the Transformer encoder-decoder whose hidden states the alignment module builds on and whose multi-head attention yields the attention baseline.","marker":"(Vaswani et al., 2017)"},{"why":"Documents that NMT attention does not match gold alignments, motivating a dedicated alignment module.","marker":"(Koehn and Knowles, 2017)"},{"why":"Analyzes where attention and alignment diverge, the pattern the paper extends to the Transformer.","marker":"(Ghader and Monz, 2017)"},{"why":"Provides the GALE Chinese-English gold alignments used for training and intrinsic evaluation.","marker":"(Li et al., 2015)"},{"why":"Provides the GALE Arabic-English gold alignments used for training and intrinsic evaluation.","marker":"(Li et al., 2013)"},{"why":"Defines the annotation-projection protocol that the NER experiment evaluates.","marker":"(Yarowsky et al., 2001)"},{"why":"Supplies the OntoNotes English and Chinese NER data used for projection and for evaluating projected NER models.","marker":"(Weischedel et al., 2013)"}],"fun_headline_variants":["DiscAlign: +27 F1 over FastAlign on word alignment","Small data, big gains: DiscAlign tops alignment baselines","Cross-lingual alignment: DiscAlign lifts F1 by 27 points","DiscAlign: 27 F1 gain over FastAlign with <5K labeled pairs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the GALE gold alignments used for both training and evaluation are accurate enough to count as ground truth, even though the paper itself reports systematic deviations from the annotation protocol and formatting errors in that data.","fun_headline_variants_meta":{"raw":{"variants":["DiscAlign: +27 F1 over FastAlign on word alignment","Small data, big gains: DiscAlign tops alignment baselines","Cross-lingual alignment: DiscAlign lifts F1 by 27 points","DiscAlign: 27 F1 gain over FastAlign with <5K labeled pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1376,"prompt_tokens":852,"completion_tokens":524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":468,"tokens_out":524,"duration_ms":4699,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:52:16.045598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the GALE Chinese and Arabic test sentences with independent, strictly protocol-checked annotators and compare DiscAlign, FastAlign, and attention on those cleaned labels; if the reported 11 to 27 point F1 advantage shrinks or disappears, the gains are partly artifacts of learning noisy gold labels rather than true alignment quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the generative IBM alignment models that FastAlign reparameterizes and that the paper's discriminative model replaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides FastAlign, the main unsupervised baseline and the macro-F1 scorer used in all intrinsic evaluations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Analyzes where attention and alignment diverge, the pattern the paper extends to the Transformer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the annotation-projection protocol that the NER experiment evaluates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the OntoNotes English and Chinese NER data used for projection and for evaluating projected NER models."}],"review_version":1}