{"id":"1869a7b7-769f-4160-8ec2-ac829b1a62bd","arxiv_id":"2505.08546","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The three tested English-to-Italian translation models correctly handle both sentences of a gendered minimal pair only 6 to 38 percent of the time, and they almost never override a masculine default in response to a feminine pronoun.","lead":"This paper introduces a new evaluation score, Minimal Pair Accuracy, to test whether English-to-Italian machine translation models use gendered pronouns in the source sentence to choose the correct grammatical gender of a profession noun. The score is low for all three tested models, and the results show a consistent male-as-default pattern, especially when feminine cues contradict stereotypes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MPA and attention results rest on an unvalidated WinoMT/fast_align pipeline for English-Italian; a manual error audit is needed before the headline numbers can be trusted.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the automatic alignment and morphological analysis pipeline is the foundation for both the MPA numbers and the attention aggregates, and it is not validated for this English-Italian dataset. I agree with that assessment. The central claim—that models ignore gender cues in most cases and show a masculine-default asymmetry—is quantified by MPA values that could be systematically biased if the pipeline mislocates the profession noun or misreads its gender. Because MPA is a conjunctive metric over pairs, even a small per-sentence error rate can produce a nontrivial compounded bias; and because the Pro-F/Pro-M comparison conditions on successful pairs, gendered patterns in pipeline errors (e.g., defaulting to masculine for ambiguous forms) could directly create or exaggerate the reported asymmetry. The attention analysis is even more sensitive to alignment since it needs the exact target index. This is not an observed error but a missing-validation gap. The concrete manual audit would settle whether the gap is real. If the audit shows high agreement, the MPA and asymmetry results would be substantially strengthened; if it shows meaningful disagreement, the numerical claims would need revision. Either way, the qualitative direction of the findings is likely to survive, but the specific magnitudes and the strength of the attention-based conclusions are contingent on this validation. Therefore the reader's CONDITIONAL verdict is appropriate, and my stress-test does not change it.","tokens_in":15822,"tokens_out":14478,"duration_ms":144767,"concrete_test":"Take a random sample of 100 minimal pairs per model (300 pairs total) from the accurately disambiguated subset (or from all pairs), manually annotate the Italian profession noun's gender and its matching source index, and compare against the fast_align/WinoMT pipeline output. Recompute MPA (Table 2) and the Pro-F/Pro-M breakdown (Table 3) with the manual labels, and also recompute the Figure 5 attention aggregates using the corrected target indices. If MPA changes by more than ~2 percentage points or the Pro-F/Pro-M ordering flips for any model, the unvalidated pipeline is load-bearing and the numerical claims need revision or a re-analysis with a validated aligner (e.g., awesome-align) and an Italian morphological analyzer with reported accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (Tables 2 and 3) depends on automatically locating the profession noun in the Italian translation and extracting its grammatical gender using WinoMT's pipeline plus fast_align (Sections 5.1 and 6.1). This pipeline is not validated for the English–Italian subset used here. Two concrete failure modes matter: (i) fast_align may misalign the source profession noun to the wrong target token, especially when Italian word order or article+noun structure differs; the extracted gender then comes from a different word (e.g., an adjective or an unrelated noun), directly corrupting MPA and the attention target index. (ii) Italian realizes gender on articles and adjectives as well as nouns; if the morphological analyzer reads only the noun form, epicene nouns (e.g., 'il/la cantante') or nouns whose ending is ambiguous ('il/la pianista') can be mislabeled, biasing both the gender accuracy and the Pro-F/Pro-M split. No error analysis, agreement rates, or alignment-quality statistics are reported. Because MPA is a joint per-pair metric (both pro- and anti-stereotypical sentences must be correct), even a modest per-sentence error rate compounds: a 5% per-sentence label error can shift a 30% MPA by several points and can distort the Pro-F vs Pro-M asymmetry if errors are gendered (e.g., if the analyzer defaults to masculine for ambiguous forms). The attention analysis inherits the same alignment dependency; a wrong target index yields attention weights from the wrong position, invalidating the 'diffused vs concentrated' comparison. This is a missing-validation issue, not an observed error, but it is the least-secure link in the argument chain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Minimal Pair Accuracy (MPA), a metric for measuring whether NMT models use gendered pronouns in English as contextual cues for disambiguating the grammatical gender of profession nouns in Italian translations. Applying MPA to the WinoMT challenge set for English-to-Italian translation, the authors report values of 6.12% for OPUS-MT, 30.24% for NLLB-200, and 38.45% for mBART (Table 2), and a breakdown showing that correctly disambiguated minimal pairs are predominantly associated with female-stereotyped professions (Table 3). They also analyze encoder self-attention weights between the gender cue and the profession noun on accurately gendered minimal pairs, concluding that masculine cues elicit more diffuse attention while feminine cues elicit more concentrated attention, with differences across models. The paper includes standard WinoMT accuracy results, an exploratory cross-attention analysis, a limitations section, and publicly released code.","tokens_in":16045,"tokens_out":5852,"duration_ms":57685,"significance":"If the MPA results are reliable, they provide a useful evaluation dimension beyond surface-level gender accuracy, directly targeting whether models adapt their translations to contextual gender cues rather than defaulting to stereotypes. The Pro-F/Pro-M asymmetry is an interesting empirical finding consistent with the male-as-norm bias and is worth reporting. The attention analysis is exploratory and the authors are appropriately cautious about causal claims; the release of the evaluation code is a concrete strength. However, the central quantitative claims depend on an unvalidated alignment and morphological-analysis pipeline for English-Italian, and the attention conclusions rest on a vaguely defined relevance threshold and visual inspection of averaged heatmaps. Both need to be addressed before the results can be fully credited.","major_comments":[{"comment":"The MPA results are computed using WinoMT's automatic pipeline to locate the profession noun in the Italian translation and extract its grammatical gender, together with fast_align to map source and target token indices. No validation or error analysis is reported for this English–Italian subset. Italian realizes gender on articles and adjectives as well as noun endings, and there are numerous epicene nouns (e.g., 'cantante', 'pianista') whose form alone is ambiguous; if the alignment or morphological tagging is wrong for even a modest fraction of sentences, the MPA percentages and the Pro-F/Pro-M split are systematically biased. Please report alignment quality and morphological-analyzer accuracy on a manually inspected sample, or otherwise demonstrate that the pipeline is accurate for this language pair.","section":"§5.1–5.2, Tables 2–3"},{"comment":"The headline differences and asymmetries are reported as point estimates without confidence intervals or significance tests. The differences between models (6.12% vs. 30.24% vs. 38.45%) and the Pro-F vs. Pro-M asymmetry (82.29% vs. 17.71% for OPUS-MT) should be accompanied by a bootstrap or McNemar test to establish that they are not due to sampling variability, especially because the number of correctly disambiguated minimal pairs is much smaller than the total number of pairs.","section":"§5.2, Tables 2–3"},{"comment":"The attention analysis identifies 'relevant' heads by comparing average attention weights to a uniform baseline of approximately 1/13, but the 'notable margin' above this baseline is never defined. The conclusions that masculine cues elicit 'weaker, more dispersed' attention and feminine cues elicit 'more localized, concentrated' attention are based on visual inspection of averaged heatmaps. Please define a quantitative concentration measure (e.g., entropy, variance, or maximum minus baseline), specify the threshold in advance, and report per-condition statistics rather than only aggregate heatmaps.","section":"§6.2, Figures 4–5"},{"comment":"The abstract claims that the models 'ignore available gender cues in most cases in favour of (statistical) stereotypical gender interpretation,' but this is not directly supported by MPA as defined. MPA is the proportion of minimal pairs in which both the pro-stereotypical and anti-stereotypical sentences receive the grammatically correct target gender; a low MPA could result from inconsistent cue use, morphological errors, or systematic defaulting. Without a baseline (e.g., a no-cue model or a chance-level expectation), the phrase 'ignore available gender cues' overinterprets the metric. Please either qualify the claim or provide a comparison baseline that would support the causal reading.","section":"Abstract and §5.2"}],"minor_comments":[{"comment":"The sentence 'We evaluate a number of NMT models using this metric, we show that they ignore available gender cues in most cases in favour of (statistical) stereotypical gender interpretation' contains a comma splice and should be rewritten for grammatical completeness.","section":"Abstract"},{"comment":"In the mBART ANTI-S example, 'Il analista' is ungrammatical Italian; the correct form is 'L'analista'. If this is verbatim model output, please indicate so; otherwise correct the translation in the figure.","section":"Figure 3"},{"comment":"The caption defines Pro-F and Pro-M as percentages of correctly disambiguated minimal pairs where the profession is stereotypically associated with women or men, respectively, but since each minimal pair contains both a pro- and an anti-stereotypical sentence, the relation to the pronoun should be clarified to avoid confusion.","section":"Table 3"},{"comment":"The description of extracting attention weights should specify the direction of attention (from the profession noun to the pronoun, or vice versa) and whether averaging over subword tokens is performed over query or key positions; this matters for interpreting the heatmaps.","section":"§6.1"},{"comment":"The reference 'Savoldi et al. (2024) A decade of gender bias in machine translation' is marked '[under review]'; references should be complete or moved to footnote, and the entry for 'Bentivogli Luisa et al. (2020)' reverses given and family names.","section":"References"},{"comment":"The heatmaps use a 'standardized colormap' but do not include a colorbar, making it impossible for readers to map colors to numerical attention values; please add a color scale.","section":"Figures 4–7"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful empirical contribution and the code release is a strength, but the central MPA numbers depend on an unvalidated alignment/morphology pipeline for English-Italian. A manual audit of a sample of the pipeline's outputs and basic significance tests would substantially strengthen the paper. The attention analysis is exploratory; the threshold issue is fixable. I would be willing to revise my assessment if the authors provide the requested validation and statistical grounding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is MPA. Scoring a model on whether it correctly genders both members of a minimal pair is a genuinely clean idea, and it gives a sharper picture than overall gender accuracy: OPUS-MT at 6%, NLLB-200 at 30%, mBART at 38%, with the Pro-F/Pro-M breakdown showing that masculine cues override female-stereotyped professions far more easily than the reverse. That asymmetry is consistent with a lot of prior work and is the kind of concrete, quantifiable claim the field can build on. The paper is also honest: it flags the binary-gender limitation, acknowledges that the attention analysis is not causal, and releases code.\n\nThe soft spots are real but proportionate. First, there are no confidence intervals or significance tests anywhere. The MPA differences between models are large enough that I doubt they vanish, but a bootstrap would have been cheap and would make the headline numbers much easier to trust. Second, the attention analysis relies on a vague threshold (a \"notable margin\" over 1/13), averaged heatmaps, and visual inspection, and the authors themselves note the gender composition imbalance. I would treat that entire section as exploratory evidence, not a mechanism finding. Third, and most importantly for the numbers: the MPA values and attention indices both depend on the WinoMT/fast_align pipeline to locate the profession noun and extract its gender in Italian. That pipeline is standard, but it is not validated for this EN-IT subset, and Italian epicene nouns and ambiguous endings are a plausible failure mode. A manual error audit on a few hundred sentences would settle this. The concern is missing validation, not an observed error, and the released code means it is checkable.\n\nThe citation pattern looks fine; they engage with the attention-as-explanation debate and prior context-mixing work. The central MPA result and the asymmetry claim are likely robust. I would send this to peer review with a request for error analysis on the alignment and some significance testing. The paper is most useful to people doing gender-bias evaluation or MT fairness audits, and MPA is the kind of simple metric that deserves to be adopted or critiqued explicitly. A serious referee should engage with it rather than desk-reject.","headline":"A clean new minimal-pair metric for measuring gender-cue reliance, with a robust male-default asymmetry result; the attention analysis is exploratory and the alignment pipeline needs validation.","tokens_in":16669,"tokens_out":2322,"would_cite":true,"duration_ms":25165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"English-to-Italian translation models ignore explicit gender pronouns in most sentence pairs, defaulting to stereotypical gender instead.","keywords":["gender bias","machine translation","minimal pair accuracy","gender disambiguation","stereotypical bias","attention analysis","encoder self-attention","English-Italian translation"],"falsifier":"Run the MPA calculation on a human-verified subset of the same sentences, manually checking alignment and target gender, and see whether the 6.12%, 30.24%, and 38.45% numbers move; if they change materially, the automatic pipeline is responsible for the reported cue-ignoring behavior. Alternatively, mask or remove the pronoun from the source and measure whether MPA drops; if it does not, the cue is not doing the work attributed to it.","tokens_in":15588,"feed_emoji":"⚖️","tokens_out":7239,"duration_ms":67019,"temperature":0.7,"pith_summary":"This paper tries to establish that neural machine translation models do not actually use the gender cues available in a source sentence when they translate; instead, they fall back on statistical stereotypes. To show this, the authors introduce Minimal Pair Accuracy (MPA), a metric built from sentence pairs that differ only in the pronoun referring to a profession noun. Across three English-to-Italian models, MPA reaches only 6.12%, 30.24%, and 38.45%, meaning the models consistently follow the cue in at most about two-fifths of pairs. The paper also finds an asymmetry: a masculine pronoun can switch a stereotypically female profession to a masculine form, but a feminine pronoun rarely switches a stereotypically male profession. If right, the paper would change how gender accuracy scores in machine translation are read, since a model can look accurate while almost never integrating contextual gender information.","feed_headline":"Gender cues steer translation models less than 40% of the time","feed_subtitle":"A new minimal-pair metric shows English-to-Italian systems default to stereotypes in most cases.","key_machinery":"Minimal Pair Accuracy (MPA) is the central artifact: two English sentences identical except for the pronoun (he vs. she) referring to the same profession noun are translated, and a pair counts as correct only if the model produces the matching grammatical gender in the Italian target noun in both directions. The second piece is an attention-weight analysis on the encoder: for the correctly gendered pairs, the authors extract the average self-attention weight from the profession noun to the gender cue across layers and heads, treating weights above the uniform baseline as evidence of cue integration. These two instruments together let the paper separate 'happens to produce the right gender' from 'consistently uses the cue to decide the gender.'","core_discovery":"The paper's central claim is that gender disambiguation in neural machine translation is driven more by learned statistical associations than by the explicit gendered pronoun in the source. Concretely, MPA—the percentage of minimal pairs in which a model correctly adapts the grammatical gender of the profession noun in both the pro-stereotypical and anti-stereotypical versions of a sentence—is 6.12% for OPUS-MT, 30.24% for NLLB-200, and 38.45% for mBART on the English–Italian challenge set. Among the pairs that are correctly disambiguated, the majority involve professions stereotypically associated with women receiving a masculine cue: 82.29%, 69.10%, and 61.90% respectively, while the reverse—feminine cues applied to male-stereotyped professions—ranges from 17.71% to 38.10%. The paper further claims that encoder self-attention between the profession noun and the pronoun shows gender-specific patterns: feminine pronouns produce concentrated, specialized attention, masculine pronouns produce weaker, more diffuse attention, and the models with more distributed attention (NLLB-200, mBART) are also the ones with higher MPA.","pith_inferences":["If MPA becomes a standard companion to accuracy metrics, model rankings could shift: a system optimized for surface gender correctness would no longer be credited with context sensitivity, and smaller models that genuinely track cues might be recognized.","The masculine-default asymmetry may not be specific to Italian morphology; the same MPA design could be applied to languages with different agreement systems or to gender-neutral cue forms to test whether the default-to-masculine pattern is grammatical or social in origin.","The attention result suggests a testable mechanism: if distributed encoding of a cue is what enables disambiguation, then fine-tuning or constraining attention in early layers of a single-head model like OPUS-MT should improve MPA; that hypothesis goes beyond what the paper proves."],"forward_implications":["A model can show respectable gender accuracy while almost never using pronouns as cues; MPA exposes that separation.","Masculine defaults are asymmetric: overriding a female-stereotyped profession with a male pronoun is common, while overriding a male-stereotyped profession with a female pronoun is rare.","Attention patterns differ by cue gender: feminine cues are encoded by specialized, localized heads, while masculine cues are diffuse; more diffuse, multi-layer encoding is associated with higher MPA.","Because the framework needs only a gendered target language, MPA can be applied to other language pairs and to any encoder-decoder or decoder-only model.","The observed attention results are correlational; the paper itself cautions that they do not prove a causal link, which motivates intervention-based follow-ups."],"supporting_citations":[{"why":"Supplies the challenge set and the alignment-and-morphology evaluation pipeline from which MPA is derived.","marker":"(Stanovsky et al., 2019)"},{"why":"Provides the pro-stereotypical and anti-stereotypical profession-gender assignments used to construct the minimal pairs.","marker":"(Zhao et al., 2018)"},{"why":"Contributes the Winogender sentences that form part of the challenge set.","marker":"(Rudinger et al., 2018)"},{"why":"Defines OPUS-MT, one of the three evaluated models.","marker":"(Tiedemann et al., 2023)"},{"why":"Defines NLLB-200, one of the three evaluated models.","marker":"(Costa-jussà et al., 2022)"},{"why":"Defines mBART, one of the three evaluated models.","marker":"(Liu et al., 2020)"},{"why":"Supplies the context-mixing quantification approach that frames the encoder self-attention analysis.","marker":"(Mohebbi et al., 2023b)"},{"why":"Provides prior evidence of default-to-masculine reasoning in language models, used to interpret the MPA asymmetry.","marker":"(Jumelet et al., 2019)"},{"why":"Raises the attention-as-explanation critique the paper cites when cautioning about its own attention results.","marker":"(Jain and Wallace, 2019)"},{"why":"Defines the multi-head attention mechanism whose weights are analyzed.","marker":"(Vaswani et al., 2017)"}],"fun_headline_variants":["Translation models ignore gender cues, default to stereotypes","New metric: NMT systems use stereotypes over gender pronouns","Feminine cues ignored more than masculine in translation models","Translation systems rely on stereotypes not gender cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that the automatic word alignment and morphological analysis used to pinpoint the profession noun and its grammatical gender in each translated sentence are accurate on this dataset, and that encoder attention weights are a meaningful proxy for how much the model actually uses the gender cue.","fun_headline_variants_meta":{"raw":{"variants":["Translation models ignore gender cues, default to stereotypes","New metric: NMT systems use stereotypes over gender pronouns","Feminine cues ignored more than masculine in translation models","Translation systems rely on stereotypes not gender cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":3088,"prompt_tokens":1008,"completion_tokens":2080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":2018}},"tokens_in":624,"tokens_out":2080,"duration_ms":14872,"temperature":1.0,"reasoning_tokens":2018,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:51:45.630708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MPA calculation on a human-verified subset of the same sentences, manually checking alignment and target gender, and see whether the 6.12%, 30.24%, and 38.45% numbers move; if they change materially, the automatic pipeline is responsible for the reported cue-ignoring behavior. Alternatively, mask or remove the pronoun from the source and measure whether MPA drops; if it does not, the cue is not doing the work attributed to it.","supporting_citations":[],"review_version":1}