{"id":"8acc14ca-4d9b-4b12-9803-38debbe6dfa0","arxiv_id":"2412.11187","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Decoder self-attention heads attending the target-side antecedent are the most influential for pronoun disambiguation, and fine-tuning them yields up to 5 percentage points improvement on contrastive tests.","lead":"This paper examines which attention heads in context-aware machine translation models matter for choosing the correct pronoun, and tests whether forcing the most useful heads to focus on the antecedent improves accuracy. The authors find that fine-tuning a few selected heads raises pronoun disambiguation accuracy by up to 5 percentage points without hurting translation quality, but the selection and evaluation procedure may inflate the gains.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2 also caps total non-antecedent attention at 1−C, so the reported gain may come from off-target suppression; a same-sized control-set intervention is needed before the causal claim can stand.","rationale":"The reader's weakest assumption identifies essentially the same problem: the intervention is not clean, and the paper discards outcome data that contradict the preferred interpretation. My analysis strengthens this by deriving the off-target effect explicitly from Eq. 2: regardless of what the pre-softmax scores for non-X tokens are, the post-softmax mass on all non-X tokens becomes 1−C. This is not a minor side effect; at C=0.99 it means the head is effectively disabled for every token outside the relation of interest. The same issue propagates into the fine-tuning experiment through Eq. 10, so the headline 'improvement solidified into the parameters' cannot be attributed to the antecedent relation without a control. The paper's own statement in Section 5 that losses at C=0.99 may come from decreased attention to other relations is an explicit flag of this confound. I do not see this as a reason to reject the paper outright: the empirical fine-tuning result may still be useful, and the qualitative finding that decoder self-attention heads matter is plausible. But the strong causal claim needs the control test, which is exactly the conditionality the reader already set. Therefore the reader's CONDITIONAL verdict remains appropriate, and I would not move it.","tokens_in":19524,"tokens_out":4986,"duration_ms":50193,"concrete_test":"Using the released code, run the Modifying Heads intervention on the same OpusMT en-de models with C=0.99 but replace the antecedent set X with a matched-size control set of randomly sampled non-antecedent context tokens, repeated over at least 100 random draws per example (or with a same-size set from the current sentence). Compare the resulting ContraPro accuracy against the Modified Accuracy column in Table 6 for d-6-4, d-6-6, d-6-7, and c-5-1. If the control intervention reproduces a comparable fraction (e.g., more than half) of the accuracy gain, the improvement is attributable to suppressing all other attention rather than to attending the antecedent relation. As a second check, rerun the Table 6 fine-tuning with the same control target set and see whether the tuned-head gain persists; if it does, the relation-specific mechanism is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that selected heads are underutilized for the antecedent relation and that tuning them to attend it produces the +5 pp ContraPro gain—is entangled with a large off-target effect in the intervention itself. In Eq. 2, only the pre-softmax scores H_{i,j} for j∈X are changed; the pre-softmax scores for j∉X are preserved. But because the modified X-logits are set to a value derived from S_out = sum_{k∉X} exp(H_{i,k}), the softmax denominator becomes S_out/(1−C). Consequently, the total post-softmax attention mass on all tokens outside X is exactly 1−C, so setting C=0.99 forces the head to allocate only 1% of its attention to every non-antecedent token combined, regardless of their original scores. The paper explicitly discards accuracy losses at C=0.99 as 'possibly resulting from the decreased attention scores for other token-to-token relations, not investigated in this work' (Section 5), which admits the intervention is not relation-specific. The same confound carries into head tuning: Eq. 10 freezes outside pre-softmax scores from an unmodified model copy, but post-softmax outside scores still shrink through the same denominator mechanism. Thus matching the modified-head accuracy in Table 6 does not validate the antecedent-relation mechanism. Without a control intervention that redirects the same amount of attention to non-antecedent tokens, the observed improvements do not uniquely support the underutilized-heads interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates which attention heads in context-aware machine translation models (OPUS-MT en-de and NLLB-200, fine-tuned on IWSLT 2017) are responsible for pronoun disambiguation in English-to-German and English-to-French. Using the ContraPro and LCPT contrastive datasets, the authors measure per-head attention to five pronoun-antecedent or pronoun-pronoun relations, correlate those attention scores with disambiguation correctness, and then intervene by modifying pre-softmax attention scores (Eq. 2) to force a head to spend a fraction C of its attention on a relation of interest. They categorize heads as responsive or non-responsive, identify several heads that appear to be underutilized, and fine-tune selected heads to reproduce the modified attention behavior. The paper reports that fine-tuning the most promising heads improves ContraPro accuracy by up to about 5 percentage points without reducing BLEU (Table 6).","tokens_in":19837,"tokens_out":5943,"duration_ms":51592,"significance":"If the causal claims were substantiated, this would be a useful contribution to interpretability for machine translation and to targeted, head-level fine-tuning: the paper provides a clear experimental pipeline, public code, and a transparent mathematical derivation in Appendix A. The finding that decoder self-attention heads matter most for target-side context integration is plausible and consistent with prior work. However, the central claim that the intervention and fine-tuning improve disambiguation by strengthening the antecedent relation is not uniquely supported, because the intervention also imposes a hard cap on all non-antecedent attention, and the evaluation loop selects heads on the same contrastive test set used for the final measurement. The paper also explicitly discards accuracy changes that contradict its expected pattern. These issues prevent the current version from validating the underutilized-head mechanism, though they are addressable with control experiments and held-out evaluation.","major_comments":[{"comment":"The Modifying Heads intervention is not relation-specific. As derived in Appendix A, setting the post-softmax mass on the target subset X to C forces the total post-softmax mass on all tokens outside X to be exactly 1−C. At C=0.99, every non-antecedent token combined receives only 1% of the head's attention, regardless of its original score. The paper itself concedes this in Section 5 by discarding accuracy losses at C=0.99 as 'possibly resulting from the decreased attention scores for other token-to-token relations, not investigated in this work.' Consequently, the observed accuracy gains at C=0.99 may stem from off-target suppression rather than from increased attention to the antecedent relation, and the 'underutilized heads' interpretation is not uniquely supported. A control intervention that redirects the same total mass C to a matched set of non-antecedent tokens is needed to separate the two mechanisms.","section":"Section 3.3, Eq. (2)"},{"comment":"The head-tuning objective reproduces the same off-target effect as the modifying-head intervention. The target pre-softmax scores are taken from Eq. (2), and although the outside pre-softmax scores are frozen from the unmodified model, the softmax denominator still rescales so that the post-softmax attention to all non-antecedent tokens is capped at 1−C. Thus the fine-tuned heads also learn to suppress non-antecedent attention, and the accuracy improvements in Table 6 are equally consistent with that suppression mechanism. The claim that tuning validates the antecedent-attention role requires a control head trained to allocate the same total mass to a non-antecedent subset; without it, the mechanism remains indistinguishable.","section":"Section 6, Eq. (10) and Table 6"},{"comment":"The head-selection and evaluation loop is circular. Heads are selected for fine-tuning based on their modification gains on ContraPro (Section 5), and the final accuracy in Table 6 is reported on the same ContraPro dataset. Using CTXPRO to construct a training set does not break the loop because the head choice itself is still based on ContraPro performance. This makes the reported improvements partly self-fulfilling. An evaluation on a held-out contrastive split or an independent pronoun-disambiguation test set is required to support the claim that the improvement is genuine and generalizes.","section":"Sections 5 and 6"},{"comment":"The paper's classification into 'attending and positively responsive,' 'attending and negatively responsive,' and similar categories relies on selectively discarding opposing evidence. Section 5 states that losses at C=0.99 and gains at C=0.01 are ignored. This one-sided treatment makes the categories depend on the authors' prior rather than on a consistent behavioral rule. The full modification curves in Appendix G should be used to define categories symmetrically, and the significance of the reported accuracy differences (e.g., the 1.6 percentage point gain for head e-6-1) should be assessed with appropriate statistical tests.","section":"Section 5"}],"minor_comments":[{"comment":"Head e-6-1 is described as 'negatively responsive for all models and positively responsive for the context-aware-3 model,' but the category definitions in Section 5 are mutually exclusive; a head that increases accuracy when modified to 0.99 cannot be negatively responsive by the given definition. Please clarify whether the head should instead be labeled fully responsive for that model.","section":"Section 5.1"},{"comment":"There is a typo: 'OpisMT' should be 'OpusMT'.","section":"Appendix D"},{"comment":"The notation X d and Xd (or X^d and X_d) is visually indistinguishable in the printed equations, making the derivation of Eq. (2) difficult to follow. Please use distinct typography (e.g., calligraphic X for the full key set and italic X for the subset of interest).","section":"Section 3.3 and Appendix A"},{"comment":"No measure of variance or significance is reported for the tuned-head accuracy improvements. Given that some differences are small (e.g., 80.68 vs 80.78 for the context-aware-1 model with c-5-1), reporting multiple seeds or confidence intervals would strengthen the claims.","section":"Table 6"},{"comment":"The description says the method 'preserves the pre-softmax attention scores H for all other target tokens,' but post-softmax scores for those tokens are not preserved; they are rescaled by the denominator. Please rephrase to avoid misleading the reader into thinking the intervention has no off-target effect.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical foundation and the authors are transparent about their methods, but the selective discarding of contradictory modification results and the absence of a control intervention are serious concerns for a causal-interpretability claim. The fixes—control experiments, held-out evaluation, and symmetric reporting of modification curves—are within the scope of a revision. The paper may also benefit from a clearer statement that the 'underutilized heads' interpretation is only one of several explanations for the observed intervention effects."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time: this is a careful, well-scoped look at where pronoun-disambiguation context enters a single-encoder Transformer, plus a clever fine-tuning recipe. The qualitative map is useful. The central causal claim—that some heads are underutilized and tuning them to attend the antecedent raises ContraPro accuracy by up to 5 points—is not supported by the intervention as designed, because the intervention has a large off-target effect.\n\nWhat's genuinely new: the paper measures five pronoun-antecedent relations (source and target sides, across encoder self-attention, cross-attention, and decoder self-attention) in two architectures and two language directions, and finds that decoder self-attention heads, especially in higher layers, are the most causally relevant for contrastive disambiguation. That's a solid, reproducible map. The head-tuning method—freezing all weights except the Q and K projections of a single head and using an MSE loss on pre-softmax scores—is clearly described, and the code and data pipeline are released.\n\nThe load-bearing flaw is in the Modifying Heads intervention. Equation 2 does not only change attention to the antecedent relation. It renormalizes the softmax denominator so that the head's total attention to every token outside that relation is exactly 1−C. At C=0.99, all non-antecedent tokens together receive 1% of the head's attention, no matter what their original scores were. The accuracy gain in that condition could therefore come from suppressing attention to distractors, not from increasing attention to the antecedent. The paper's own Section 5 discards accuracy losses at C=0.99 and gains at C=0.01 as 'possibly resulting from the decreased attention scores for other token-to-token relations, not investigated in this work.' That is the confound, named. The fine-tuning target in Eq. 10 inherits it, so matching the modified-head accuracy in Table 6 does not validate the antecedent-relation mechanism.\n\nSecondary issues: heads are selected for fine-tuning on the same ContraPro benchmark used for final evaluation, which inflates the reported gain; there are no significance measures; and the discarded outcomes should be reported and explained. The qualitative finding that decoder heads matter is likely robust, but the 'underutilized heads' interpretation is not uniquely supported.\n\nThis deserves a serious referee: the map and the tuning method are worth building on, but the paper needs a control intervention (redirecting the same amount of attention to non-antecedent tokens) and a held-out contrastive set before the causal claim can stand.","headline":"A useful map of where pronoun disambiguation lives in single-encoder MT, but the causal claim about underutilized heads is undercut by an off-target intervention and same-benchmark head selection.","tokens_in":20351,"tokens_out":4957,"would_cite":false,"duration_ms":38949,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A few attention heads carry pronoun disambiguation in context-aware machine translation, and tuning them to attend the antecedent raises accuracy by up to 5 percentage points without BLEU loss.","keywords":["attention heads","pronoun disambiguation","context-aware machine translation","contrastive evaluation","ContraPro","Transformer interpretability","head fine-tuning","target-side context"],"falsifier":"Run the same head-tuning pipeline on randomly selected heads or on heads that were classified as non-attending and non-responsive: if those heads produce the same or similar ContraPro gains, the improvement is not specific to the identified disambiguation heads. Alternatively, hold the attention mass on all other token relations fixed while strengthening only the target relation and check whether the accuracy gain survives; if it disappears, the reported effect is an artifact of global attention redistribution.","tokens_in":19309,"feed_emoji":"🧠","tokens_out":6255,"duration_ms":47691,"temperature":0.7,"pith_summary":"This paper tries to show that pronoun disambiguation in context-aware machine translation is not spread evenly across the Transformer but is carried by a small set of attention heads, and that those heads can be improved on purpose. Working with single-encoder models (OPUS-MT English-to-German and NLLB-200 English-to-German and English-to-French), the authors measure how much each head attends five pronoun-to-antecedent relations, correlate those scores with correct contrastive choices, and then artificially raise or lower the attention assigned to a relation. They find heads that attend the target-side antecedent relation and respond to intervention, plus heads that are underutilized: the model would do better if those heads attended more. Fine-tuning the most promising heads to attend the target-side antecedent relation improves ContraPro accuracy by up to 5 percentage points while leaving BLEU essentially unchanged. The point of the exercise is that interpretability findings can be converted into parameter-level fixes.","feed_headline":"Fine-tuning a few attention heads lifts pronoun accuracy by 5 points","feed_subtitle":"Decoder self-attention heads that track the antecedent can be strengthened without lowering BLEU.","key_machinery":"The carrying mechanism is the Modifying Heads intervention. For a chosen head and a relation Y to X, the formula replaces the pre-softmax scores of tokens in X so that their post-softmax attention sum equals a chosen value C, while keeping the pre-softmax scores of all other tokens unchanged. This lets the authors test, head by head, whether forcing more or less attention onto a pronoun-antecedent relation changes disambiguation accuracy. The complementary machinery is head tuning: freeze all parameters except the Q and K projections of a selected head and train it with an MSE loss toward the modified pre-softmax scores, thereby solidifying the intervention into the model weights.","core_discovery":"The central claim is that in context-aware Transformer translation, a handful of decoder self-attention heads causally support pronoun disambiguation by attending to the target-side antecedent, and some of those heads are underused. Evidence comes from three complementary measurements: average attention scores, point-biserial correlation between a head's attention on a relation and the model being correct, and a controlled modification of pre-softmax attention scores. The strongest result is that fine-tuning selected heads (for example head d-6-4 on the target-pronoun-to-target-antecedent relation) raises ContraPro accuracy from 81.46% to 86.42% in the sentence-level OPUS-MT model and produces similar gains in context-aware models, with no BLEU drop. The paper also concludes that target-side context is more impactful than source-side context, that the most relevant heads sit in higher decoder layers, and that some heads show the same disambiguation behavior across English-to-German and English-to-French in the multilingual model.","pith_inferences":["The paper's causal reading assumes the Modifying Heads intervention changes only the target relation's attention mass; because the paper itself discards accuracy losses at C=0.99 as off-target effects, a cleaner test would verify that attention on other token relations is unchanged when one relation is strengthened.","If the tuning result transfers from contrastive ranking to generative decoding, the same method could be applied to other context-dependent phenomena such as deixis, ellipsis, and lexical cohesion, where contrastive test sets exist.","The head-tuning recipe uses gold target context; in a real system the context is the model's own output, so error propagation may reduce the measured gains unless the tuned heads are trained on predicted context.","A direct falsifier would be to fine-tune the same number of randomly chosen heads: if random heads produce similar ContraPro gains, the improvement is not specific to the identified disambiguation heads."],"forward_implications":["If the identified heads are genuinely responsible, context-aware MT improvements can be targeted at a few decoder heads rather than requiring full-model retraining.","Target-side context, meaning the previously generated target sentence, is the main carrier of disambiguation information; source-side antecedent relations matter less.","Underutilized heads mean current models have latent disambiguation capacity that parameter-level tuning can unlock without sacrificing translation quality.","The overlap between improvements from pairs of heads is below 30%, suggesting the heads play complementary roles, so tuning several together could compound the gain.","In multilingual models, some heads exhibit the same disambiguation behavior across language pairs, hinting at shared cross-lingual mechanisms."],"supporting_citations":[{"why":"Supplies ContraPro, the English-to-German contrastive test set on which disambiguation accuracy is measured.","marker":"(Müller et al., 2018)"},{"why":"Supplies LCPT, the English-to-French contrastive pronoun test set used for the second language direction.","marker":"(Lopes et al., 2020)"},{"why":"Defines the context-aware MT setting and contrastive evaluation approach that the paper builds on.","marker":"(Bawden et al., 2018)"},{"why":"Defines the Transformer and multi-head attention architecture whose heads are analyzed and modified.","marker":"(Vaswani et al., 2017)"},{"why":"Provides the CTXPRO toolset used to extract pronoun-antecedent training examples for head tuning.","marker":"(Wicks and Post, 2023)"},{"why":"Supplies the multilingual NLLB-200 distilled model, one of the two pretrained models studied.","marker":"(NLLB Team et al., 2022)"},{"why":"Supplies the OPUS-MT en-de model, the other pretrained model analyzed and tuned.","marker":"(Tiedemann and Thottingal, 2020)"}],"fun_headline_variants":["Tuning a few attention heads boosts pronoun accuracy by 5 points","Underused attention heads hold key to better pronoun translation","5-point pronoun gain from fine-tuning specific attention heads","Attention heads that track antecedents boost translation accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions depend on the intervention being clean: changing the pre-softmax scores for one head-relation pair is assumed to alter only that relation's attention mass while leaving the rest of the model's behavior intact, yet the paper acknowledges off-target effects at C=0.99.","fun_headline_variants_meta":{"raw":{"variants":["Tuning a few attention heads boosts pronoun accuracy by 5 points","Underused attention heads hold key to better pronoun translation","5-point pronoun gain from fine-tuning specific attention heads","Attention heads that track antecedents boost translation accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1336,"prompt_tokens":880,"completion_tokens":456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":391}},"tokens_in":496,"tokens_out":456,"duration_ms":4120,"temperature":1.0,"reasoning_tokens":391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:11:44.908352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same head-tuning pipeline on randomly selected heads or on heads that were classified as non-attending and non-responsive: if those heads produce the same or similar ContraPro gains, the improvement is not specific to the identified disambiguation heads. Alternatively, hold the attention mass on all other token relations fixed while strengthening only the target relation and check whether the accuracy gain survives; if it disappears, the reported effect is an artifact of global attention redistribution.","supporting_citations":[{"cited_title":"Amin Farajian, Rachel Bawden, Michael Zhang, and Andr \\'e F","cited_arxiv_id":null,"evidence_quote":"Supplies LCPT, the English-to-French contrastive pronoun test set used for the second language direction."}],"review_version":1}