Pith. sign in

REVIEW 4 major objections 5 minor 61 references

A Representation Level Analysis of NMT Model Robustness to Grammatical Errors

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that NMT encoders first detect a grammatical error, then move that word's representation toward its correct form, and that fine-tuning for robustness mainly amplifies this existing behavior.

desk verdict Serious empirical breadth, but the detect-then-correct narrative and Robustness Heads claim rest on a confound that needs controls before the headline strength is justified. read the letter →

arxiv 2505.21224 v1 pith:XSXKPQ34 submitted 2025-05-27 cs.CL

Abderrahmane Issam , Yusuf Can Semerci , Jan Scholtes , Gerasimos Spanakis This is my paper
classification cs.CL
keywords neuralmachinetranslationrobustnessgrammaticalerrorcorrectionGEDprobingCKArepresentationsimilarityattentionheadssyntheticnoisefine-tuningmultilingualNMT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that neural machine translation (NMT) encoders already implement an implicit grammatical-error-correction loop: as layers deepen, the encoder first marks an ungrammatical word as detectable by probes, then adjusts that word's representation so that it moves toward the representation of the correct word. The authors test this on four multilingual NMT models across five language directions using synthetic article, preposition, and noun-number errors plus a black-box inflectional attack. They find that GED probing accuracy rises in the first half of the encoder and then plateaus or drops, while a CKA-based distance between the noisy word and its clean counterpart shrinks through the layers, especially after fine-tuning on noisy data. They also isolate attention heads, called Robustness Heads, whose masking moves the noisy representation toward its clean form, and show these heads attend to linguistically relevant part-of-speech categories. A sympathetic reader would care because it suggests that robustness to grammatical noise is not an extra skill but a re-weighting of a structure that translation models already possess.

What carries the argument

The load-bearing objects are: (i) GED probes, single-layer linear classifiers trained on each encoder layer's word representations to detect the ungrammatical word; (ii) CKA (Centered Kernel Alignment) representational similarity, used as $1-\text{CKA}(fW, W)$ between noisy-word representations and their clean-form counterparts to measure correction; and (iii) Influential Heads and Robustness Heads, constructed by masking one attention head at a time and measuring how much the word representation moves, where Robustness Heads are the heads whose masking pulls the noisy representation toward its grammatical form. The framework's work is to convert the unobservable 'correction' into two measurable quantities, detection accuracy per layer and representation displacement per layer, and to tie the displacement to specific attention heads.

What would settle it

A control experiment that would refute the targeted-correction claim: compute $1-\text{CKA}$ between the noisy word's representation and (a) its clean form and (b) a third word of the same POS category placed in the identical sentence context, across encoder layers. If both distances shrink at the same rate, the representation change is generic context pooling rather than error correction; the paper's claim predicts a significantly larger drop toward the clean form, especially in deeper layers of noise-finetuned models.

Watch

Extended reading notes

Core claim

The central claim is that an NMT encoder operates as a detect-then-correct pipeline for grammatical errors. For each ungrammatical input, the encoder learns a representation of the erroneous word that is progressively steered toward the representation the same word would have in a clean sentence; the paper supports this with two measurements. Grammatical error detection probing shows that the erroneous word's identity is readily linear-decoded in lower-to-middle layers, with F1 rising early and then falling in deeper layers, which the authors read as the detection stage being superseded by correction. The representational distance $1-\text{CKA}$ between the noisy word representation and the clean-form representation decreases layer by layer for base models and collapses toward zero for noise-finetuned models, which they interpret as active correction. Fine-tuning for robustness on noisy data does not invent new behavior; it amplifies existing behavior, since noise-finetuned models rely more heavily on Robustness Heads in deeper layers, where the correction is hypothesized to occur.

Load-bearing premise

The detect-then-correct reading assumes that the shrinking CKA distance between a noisy word and its clean-form twin measures a targeted repair, rather than just both representations converging because the surrounding sentence is nearly identical; the paper itself flags this shared-context alternative and runs no control against another word in the same context.

Editorial extensions

If this is right

  • Fine-tuning only the encoder on synthetic grammatical errors is nearly as effective as full-model fine-tuning for closing the clean-versus-noisy COMET gap, while preserving performance on clean data.
  • Base, clean-finetuned, and noise-finetuned models share the same Robustness Heads and similar attention patterns over POS tags; robustness fine-tuning mainly increases reliance on these heads in deeper layers.
  • The detect-then-correct pattern holds across four model families and across article, preposition, noun-number, and MORPHEUS inflectional errors, so it is not an artifact of a single noise type.
  • Error handling differs across source languages; for instance, noun-number detection peaks at a higher F1 for French, where articles and adjectives carry number, than for English, suggesting robustness depends on the source language's linguistic signaling.
  • Since fine-tuning on grammatical errors can even slightly improve clean-data translation, the authors conclude that such fine-tuning acts as a regularizer and can be a practical, interpretable robustness strategy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the detect-then-correct account is right, the same mechanism may appear in decoder-only language models, where early layers might localize the error and later layers blend it away; the paper itself flags this as future work.
  • The Robustness Head identification method could be turned into an audit tool: given any new model, mask heads to locate which linguistic regularities it silently repairs, without needing adversarial fine-tuning.
  • Because shared context would also shrink CKA distance, a stronger test would compare the noisy representation's distance to its clean form against its distance to a randomly chosen word in the same sentence; the correction claim predicts a specific advantage, not just general convergence.
  • The language asymmetry suggests that multi-language robustness fine-tuning may suffer interference, and language-specific adapters could isolate the correction heads per language rather than letting languages compete.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper analyzes how NMT encoder representations respond to synthetic grammatical errors. It introduces three error types (article, preposition, noun-number) plus MORPHEUS perturbations, and compares Base, Clean-Finetuned, and Noise-Finetuned variants of four translation models across five language directions. The authors use GED probing to track error detection through layers, 1-CKA distance between noisy and clean word representations to track correction, and a head-masking method to define Robustness Heads and Influential Heads. The central claims are that NMT encoders implement a detect-then-correct GEC process and that robustness fine-tuning increases reliance on Robustness Heads in deeper layers.

Significance. If the central interpretation holds, the paper provides a model-agnostic, layer-resolved account of how NMT encoders handle ungrammatical input, and it identifies an attention-level mechanism that fine-tuning amplifies. The study is broad in coverage: four model families, five language directions, three targeted error types plus one black-box attack, and it ships code. The descriptive findings—GED probing curves, CKA distance trends, and attention-to-POS patterns—are internally consistent and potentially useful for interpreting robustness in NMT. The main risk is that the evidence for the 'correction' component and for the Robustness-Head overlap is confounded by shared context between the noisy and clean sentence pairs, as detailed below.

major comments (4)
  1. [§5.3, Figure 2] The CKA distance between the noisy word and its clean form decreases with depth even for Base and Clean-Finetuned models, and the paper itself explains this by noting that the two words share the same context and that representations move closer as the encoder integrates context. Therefore the near-zero distances observed for Noise-Finetuned models do not by themselves establish targeted correction; a control such as the CKA distance between the noisy word and a different word in the same sentence, or paired per-token distances for unchanged words, is needed to rule out generic context integration. This is load-bearing for RQ1 and for the interpretation of RQ3, and the paper should add such a control or substantially weaken the correction claim.
  2. [§3.3.4, §5.4.2, Figure 4] The Robustness Head distance 1-CKA(gw_hi, w) and the Influential Head distance 1-CKA(w_hi, w) coincide whenever the un-masked noisy representation is close to the clean representation, which is exactly the regime observed for Noise-Finetuned models in Figure 2. Thus the high Robustness/Influential overlap reported in Figure 4 may be a mathematical consequence of the convergence measured in §5.3 rather than independent evidence that fine-tuning recruits special heads. The authors should compare the observed overlap against a null model that preserves the marginal head distances, for example random head subsets matched on the distribution of 1-CKA values.
  3. [§3.3.4, §5.4.2] The procedure for selecting the sets of Robustness Heads and Influential Heads is not specified. Figure 4 reports an 'accuracy' between the two sets, but the manuscript does not state whether heads are selected by a threshold, a top-k rule, a statistical test, or some other criterion, nor does it report sensitivity to that choice. Without this, the result is not reproducible and the magnitude of the overlap cannot be interpreted. Please specify the selection rule and report how Figure 4 changes under reasonable alternatives.
  4. [Abstract, §1 RQ1, §6 Discussion] The claim that NMT encoders 'inherently implement' a GEC setup is stronger than the evidence supports. In §5.2, GED probing accuracy decreases in deeper layers only for Noise-Finetuned models, while Base and Clean-Finetuned models plateau; in §5.3, the distance decrease for Base and Clean-Finetuned models is attributed to shared context rather than correction. The Discussion already hedges by saying the behavior is 'more distinguishable in Noise-Finetuned models.' I recommend moving the RQ1 claim to that more modest level, or providing direct evidence for inherent detection-then-correction in Base models.
minor comments (5)
  1. [Appendix C.3] The heading 'Respresentation Similarity' contains a typo and should read 'Representation Similarity'.
  2. [Figure 4, §5.4.2] The y-axis label and caption say 'Accuracy' but the figure reports agreement between two sets of heads; please define this quantity in the text and state whether it is head-level accuracy, overlap proportion, or another measure.
  3. [§3.3.2] The notation fW for the ungrammatical word representations is confusing because f usually denotes a function; consider using X_noisy or W_noisy for consistency with W.
  4. [References] Durrani et al. 2023a and 2023b appear to be the same paper with identical titles and bibliographic details; please verify and correct the duplicate reference.
  5. [Appendix C.4] The sentence beginning 'Figure shows the accuracy...' appears to be a leftover placeholder; it should refer to a specific figure number.

Circularity Check

1 steps flagged · score 6.0 of 10

Robustness-vs-Influential head accuracy (Fig. 4) collapses by construction once the Section 5.3 noisy/clean CKA convergence holds; the RQ3 head-level mechanism claim reduces to the convergence it purports to explain.

  1. self definitional [Section 3.3.4 (Robustness Heads definition) and Section 5.4.2; relied on in Section 6 (RQ3) and the Abstract]
    "This requires a simple redefinition of Influential Heads, where instead of computing the distance to the original word itself, we compute the distance from the noisy word representation to the representation of its clean form. Formally, instead of computing 1−CKA (whi, w) we compute 1 − CKA (gwhi, w) ... The figure shows that this accuracy is higher in Noise-Finetuned models especially in deeper layers, which means that models after fine-tuning on noise, tend to employ more Robustness Heads for updating the noisy word representation."

    The two scores differ only in the reference representation: Influential Heads measure 1−CKA(masked_noisy, noisy), while Robustness Heads measure 1−CKA(masked_noisy, clean). Section 5.3 reports that for Noise-Finetuned models the noisy and clean representations converge to "almost 0. CKA distance in most cases." When those reference points coincide, the two CKA scores are the same function of the masked representation for every head, so ranking heads by influence and by robustness yields the same order and the Section 5.4.2 "accuracy" is high by construction — precisely in the models and layers where the Section 5.3 convergence holds.

full rationale

One step is genuinely circular under the paper's own equations. Section 3.3.4 defines Influential Heads by the distance from a head-masked noisy representation to the original noisy representation (1−CKA(whi, w)) and Robustness Heads by the distance of the same masked representation to the clean form (1−CKA(gwhi, w)). Because Section 5.3 reports near-zero CKA distance between noisy and clean representations for Noise-Finetuned models, the two reference points coincide, making the two scores the same function of the masked representation; the high Robustness-vs-Influential agreement in Figure 4 then follows mathematically rather than evidencing a dedicated correction subpopulation of heads. The RQ3 and Abstract claims of increased reliance on Robustness Heads therefore reduce to the Section 5.3 convergence. The remainder of the chain is not circular: GED probing F1 comes from independently trained linear probes on held-out data, and the CKA distance curves are empirical measurements; the paper itself concedes the plausible null explanation for the base-model trend ("both words have the same linguistic function ... and because they share the same context which leads their representation to move closer as the encoder integrates context into it"), which is an interpretation/control gap for the correction narrative rather than a definitional reduction. There is no load-bearing self-citation: the masking-influence method is credited to Voita et al. (2019), an external source, and no reliance on the authors' own prior theorems appears. Because the headline mechanism claim (Robustness Heads) reduces by construction while the underlying probing and distance measurements retain independent empirical content, the score is 6 (partial circularity) rather than 8–10.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The analysis rests on interpretive assumptions about what probes, CKA distances, and masking interventions mean; none of these are directly validated against external ground truth, and the head-selection procedure includes an unreported parameter. The Robustness Heads construct carries the heaviest load and the least independent evidence.

free parameters (1)
  • Head selection rule (threshold/top-k) for Robustness vs Influential Heads = not reported
    Figure 4 reports 'accuracy' between two sets of heads, but Section 3.3.4 does not state how heads are selected into the sets (e.g., top-k, threshold, or argmax). This choice, made by the authors, directly determines the reported accuracy and is not disclosed.
assumptions (5)
  • domain assumption Synthetic errors (one replacement per sentence) capture how models handle natural grammatical errors.
    Invoked in Section 3.1 and acknowledged as a limitation in Section 8; the entire analysis is performed on synthetic noise.
  • domain assumption Linear probes trained on a layer's word representations measure whether the model 'detects' the error at that layer.
    GED probing (Section 3.3.1) assumes linear separability in the representation space reflects the model's own detection capability, following Davis et al. (2022).
  • domain assumption Decreasing CKA distance between noisy and clean word representations means the model moves the noisy representation toward the clean form, rather than both representations converging due to shared context.
    The correction interpretation in Section 5.3 depends on this; the paper itself notes the shared-context explanation but provides no control to rule it out.
  • domain assumption Masking one attention head at a time isolates that head's causal contribution to a word's representation.
    The Robustness/Influential Head definitions in Section 3.3.4 treat heads as independently intervenable, an approximation that ignores interactions between heads and the non-linearity of the transformer.
  • domain assumption Attention scores to POS tags are a valid lens into what the model uses to handle errors.
    The POS attention analysis (Section 3.3.5) relies on the attention-as-explanation assumption, known to be imperfect, and uses spaCy tags as ground truth.
invented entities (1)
  • Robustness Heads
    purpose: Attention heads whose masking moves the ungrammatical word's representation away from the clean form; used to explain how fine-tuning improves robustness.
    The construct is defined via the paper's own masked-CKA measure, and its overlap with Influential Heads is confounded by representation convergence; no external falsifiable prediction about these heads is offered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Representation Level Analysis of NMT Model Robustness to Grammatical Errors." pith.science (2026). https://pith.science/paper/XSXKPQ34

@misc{pith2026250521224,
  author       = {Pith},
  title        = {Pith review of: A Representation Level Analysis of NMT Model Robustness to Grammatical Errors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSXKPQ34}},
  note         = {Machine review of arXiv:2505.21224}
}
read the original abstract

Understanding robustness is essential for building reliable NLP systems. Unfortunately, in the context of machine translation, previous work mainly focused on documenting robustness failures or improving robustness. In contrast, we study robustness from a model representation perspective by looking at internal model representations of ungrammatical inputs and how they evolve through model layers. For this purpose, we perform Grammatical Error Detection (GED) probing and representational similarity analysis. Our findings indicate that the encoder first detects the grammatical error, then corrects it by moving its representation toward the correct form. To understand what contributes to this process, we turn to the attention mechanism where we identify what we term Robustness Heads. We find that Robustness Heads attend to interpretable linguistic units when responding to grammatical errors, and that when we fine-tune models for robustness, they tend to rely more on Robustness Heads for updating the ungrammatical word representation.

Figures

Figures reproduced from arXiv: 2505.21224 by the authors.

Figure 1
Figure 1. GED probing performance of Noise [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. CKA distance of clean and noise word repre [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Robustness Heads attention to the 10 most common POS tags in the test set on En-Es. The scale of attention is relative to each base model and error. We highlight POS tags that are attended to the most across models. 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy OPUS-MT | Article OPUS-MT | Nounnum OPUS-MT | Prep 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy M2M100 | Article M2M100 | Nounnum M2M100 | Prep 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy MBART… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Accuracy of Robustness and Influential heads [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: COMET difference between performance on clean and noisy test sets after fine-tuning different parts of [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: GED probing performance of Noise-Finetuned, Clean-Finetuned and Base models on Fr-Es, En-De, En-It [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: CKA distance of clean and noise word representations across models and errors on Fr-Es, En-De, En-It [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Robustness Heads attention to the 10 most common POS tags in the test set on Fr-Es. The scale of attention is relative to each base model and error. We highlight POS tags that are attended to the most across models. 1 2 3 4 5 6 Layer Base Clean Noise Base Clean Noise B…
Figure 9
Figure 9. Figure 9: Robustness Heads attention to the 10 most common POS tags in the test set on En-De. The scale of attention is relative to each base model and error. We highlight POS tags that are attended to the most across models [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Robustness Heads attention to the 10 most common POS tags in the test set on En-It. The scale of attention is relative to each base model and error. We highlight POS tags that are attended to the most across models. 1 2 3 4 5 6 Layer Base Clean Noise Base Clean Noise …
Figure 11
Figure 11. Figure 11: Robustness Heads attention to the 10 most common POS tags in the test set on En-Nl. The scale of attention is relative to each base model and error. We highlight POS tags that are attended to the most across models [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Accuracy of Robustness and Influential heads on Fr-Es, En-De, En-It and En-Nl. We find the accuracy is [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: GED probing performance of Noise-Finetuned, Clean-Finetuned and Base models on En-Es, Fr-Es, [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: CKA distance of clean and noise word representations across models and errors on En-Es, Fr-Es, En-De, [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Accuracy of Robustness and Influential heads on En-Es, Fr-Es, En-De, En-It and En-Nl on Morpheus [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 31 canonical work pages

  1. [1]

    Antonios Anastasopoulos. 2019. https://doi.org/10.18653/v1/W19-4822 An analysis of source-side grammatical errors in NMT . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 213--223, Florence, Italy. Association for Computational Linguistics

    work page doi:10.18653/v1/w19-4822 2019

  2. [2]

    Nguyen, and David Chiang

    Antonios Anastasopoulos, Alison Lui, Toan Q. Nguyen, and David Chiang. 2019. https://doi.org/10.18653/v1/N19-1311 Neural machine translation of text from non-native speakers . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) ...

    work page doi:10.18653/v1/n19-1311 2019

  3. [3]

    Ankur Bapna and Orhan Firat. 2019. https://doi.org/10.18653/v1/D19-1165 Simple, scalable adaptation for neural machine translation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538--1548, Hong Kong, China. Associat...

    doi:10.18653/v1/d19-1165 2019

  4. [4]

    Yonatan Belinkov and Yonatan Bisk. 2018. https://openreview.net/forum?id=BJ8vJebC- Synthetic and natural noise both break neural machine translation . In International Conference on Learning Representations

    2018

  5. [5]

    Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. https://doi.org/10.1162/coli_a_00367 On the linguistic representational power of neural machine translation models . Computational Linguistics, 46(1):1--52

    doi:10.1162/coli_a_00367 2020

  6. [6]

    Yonatan Belinkov, Llu \' s M \`a rquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017. https://aclanthology.org/I17-1001 Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Lo...

    work page 2017

  7. [7]

    Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. 2024. https://arxiv.org/abs/2406.19783 Nlperturbator: Studying the robustness of code llms to natural language variations . Preprint, arXiv:2406.19783

    arXiv 2024

  8. [8]

    Christian Cianfarani, Arjun Nitin Bhagoji, Vikash Sehwag, Ben Zhao, Haitao Zheng, and Prateek Mittal. 2022. https://openreview.net/forum?id=SbAaNa97bzp Understanding robust learning through the lens of representation similarities . In Advances in Neural Information Processing Systems

    work page 2022

Show all 61 references
  1. [9]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/W19-4828 What does BERT look at? an analysis of BERT ' s attention . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL...

    2019 doi

  2. [10]

    Christopher Davis, Christopher Bryant, Andrew Caines, Marek Rei, and Paula Buttery. 2022. https://doi.org/10.18653/v1/2022.conll-1.25 Probing for targeted syntactic knowledge through grammatical error detection . In Proceedings of the 26th Conference on Computational Natural L...

    2022 doi

  3. [11]

    Nadir Durrani, Fahim Dalvi, and Hassan Sajjad. 2023 a . http://jmlr.org/papers/v24/23-0074.html Discovering salient neurons in deep nlp models . Journal of Machine Learning Research, 24(362):1--40

    2023

  4. [12]

    Nadir Durrani, Fahim Dalvi, and Hassan Sajjad. 2023 b . http://jmlr.org/papers/v24/23-0074.html Discovering salient neurons in deep nlp models . Journal of Machine Learning Research, 24(362):1--40

    2023

  5. [13]

    Nadir Durrani, Hassan Sajjad, and Fahim Dalvi. 2021. https://doi.org/10.18653/v1/2021.findings-acl.438 How transfer learning impacts linguistic knowledge in deep NLP models? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4947--4957, Online...

    2021 doi

  6. [14]

    Allyson Ettinger. 2020. https://doi.org/10.1162/tacl_a_00298 What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models . Transactions of the Association for Computational Linguistics, 8:34--48

    2020 doi

  7. [15]

    Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond en...

    2021

  8. [16]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.2 Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics ...

    2022

  9. [17]

    Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020. https://doi.org/10.1109/ICASSP40776.2020.9054626 Europarl-st: A multilingual corpus for speech translation of parliamentary deb...

    2020

  10. [18]

    Emi Izumi, Kiyotaka Uchimoto, and Hitoshi Isahara. 2004. http://www.lrec-conf.org/proceedings/lrec2004/pdf/470.pdf The overview of the SST speech corpus of J apanese learner E nglish and evaluation through the experiment on automatic detection of learners ' errors . In Proceed...

    2004

  11. [19]

    Sai Muralidhar Jayanthi and Adithya Pratapa. 2021. https://doi.org/10.18653/v1/2021.sigmorphon-1.6 A study of morphological robustness of neural machine translation . In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morpholo...

    2021 doi

  12. [20]

    Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. https://doi.org/10.18653/v1/D19-5506 Training on synthetic noise improves robustness to natural noise in machine translation . In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NU...

    2019 doi

  13. [21]

    Huda Khayrallah and Philipp Koehn. 2018. https://doi.org/10.18653/v1/W18-2709 On the impact of various types of noise on neural machine translation . In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74--83, Melbourne, Australia. Associatio...

    2018 doi

  14. [22]

    Josef Klafka and Allyson Ettinger. 2020. https://doi.org/10.18653/v1/2020.acl-main.434 Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words . In Proceedings of the 58th Annual Meeting of the Association for Computation...

    2020 doi

  15. [23]

    Tom Kocmi, Vil \'e m Zouhar, Christian Federmann, and Matt Post. 2024. https://doi.org/10.18653/v1/2024.acl-long.110 Navigating the metrics maze: Reconciling score magnitudes and accuracies . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui...

    2024 doi

  16. [24]

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 2019. https://arxiv.org/abs/1905.00414 Similarity of neural network representations revisited . CoRR, abs/1905.00414

    2019 arXiv

  17. [25]

    Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. https://doi.org/10.18653/v1/D19-1167 Investigating multilingual NMT representations at scale . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationa...

    2019 doi

  18. [26]

    Liu, Matt Gardner, Yonatan Belinkov, Matthew E

    Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. https://doi.org/10.18653/v1/N19-1112 Linguistic knowledge and transferability of contextual representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the ...

    2019 doi

  19. [27]

    Wong, Liang Ding, Lidia S

    Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, Shuming Shi, and Zhaopeng Tu. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.247 On the complementarity between pre-training and back-translation for neural machine translation . In Findings of the Assoc...

    2021 doi

  20. [28]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  21. [29]

    Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. https://doi.org/10.18653/v1/2020.blackboxnlp-1.4 What happens to BERT embeddings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, ...

    2020 doi

  22. [30]

    Hedderich, and Dietrich Klakow

    Marius Mosbach, Anna Khokhlova, Michael A. Hedderich, and Dietrich Klakow. 2020. https://doi.org/10.18653/v1/2020.blackboxnlp-1.7 On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained transformers . In Proceedings of the Third ...

    2020 doi

  23. [31]

    Courtney Napoles, Aoife Cahill, and Nitin Madnani. 2016. https://doi.org/10.18653/v1/W16-0501 The effect of multiple grammatical errors on processing non-native writing . In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications , pages...

    2016 doi

  24. [32]

    Pavan Kalyan Reddy Neerudu, Subba Oota, Mounika Marreddy, Venkateswara Kagita, and Manish Gupta. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.477 On robustness of finetuned transformer-based NLP models . In Findings of the Association for Computational Linguistics: EM...

    2023 doi

  25. [33]

    Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. https://doi.org/10.3115/v1/W14-1701 The C o NLL -2014 shared task on grammatical error correction . In Proceedings of the Eighteenth Conference on Computational Na...

    2014 doi

  26. [34]

    NLLB Team , Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Pran...

    2022 arXiv

  27. [35]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...

    2002

  28. [36]

    Peyman Passban, Puneeth Saladi, and Qun Liu. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.323 Revisiting robust neural machine translation: A transformer case study . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3831--3840, Punta Can...

    2021 doi

  29. [37]

    Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. https://doi.org/10.3115/1220175.1220230 Learning accurate, compact, and interpretable tree annotation . In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of t...

    2006

  30. [38]

    Jason Phang, Haokun Liu, and Samuel R. Bowman. 2021. https://doi.org/10.18653/v1/2021.blackboxnlp-1.42 Fine-tuned transformers show clusters of similar representations across layers . In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networ...

    2021 doi

  31. [39]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  32. [40]

    Alessandro Raganato and J \"o rg Tiedemann. 2018. https://doi.org/10.18653/v1/W18-5431 An analysis of encoder representations in transformer-based machine translation . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing and Interpreting Neural Networks for NLP...

    2018 doi

  33. [41]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

    2020 doi

  34. [42]

    Elizabeth Salesky, Matthias Sperber, and Alexander Waibel. 2019. https://doi.org/10.18653/v1/N19-1285 Fluent translations from disfluent speech in end-to-end speech translation . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Comput...

    2019 doi

  35. [43]

    Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020. https://doi.org/10.18653/v1/2020.acl-main.263 It`s morphin' time! C ombating linguistic discrimination with inflectional perturbations . In Proceedings of the 58th Annual Meeting of the Association for Computation...

    2020 doi

  36. [44]

    Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. https://arxiv.org/abs/2008.00401 Multilingual translation with extensible multilingual pretraining and finetuning . Preprint, arXiv:2008.00401

    2020 arXiv

  37. [45]

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. https://openreview.net/forum?id=SJzSgnRcKX What do you learn from context? probing for sentence structure in contex...

    2019

  38. [46]

    J \"o rg Tiedemann and Santhosh Thottingal. 2020. https://aclanthology.org/2020.eamt-1.61 OPUS - MT -- building open translation services for the world . In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479--480, Lisboa, P...

    2020

  39. [47]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  40. [48]

    Ra \'u l V \'a zquez, Hande Celikkanat, Mathias Creutz, and J \"o rg Tiedemann. 2021. https://doi.org/10.18653/v1/2021.acl-srw.35 On the differences between BERT and MT encoder spaces and how to address them in translation tasks . In Proceedings of the 59th Annual Meeting of t...

    2021 doi

  41. [49]

    Elena Voita, Rico Sennrich, and Ivan Titov. 2019. https://doi.org/10.18653/v1/D19-1448 The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives . In Proceedings of the 2019 Conference on Empirical Methods ...

    2019 doi

  42. [50]

    Tao Wang, Chengqi Zhao, Mingxuan Wang, Lei Li, Hang Li, and Deyi Xiong. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.396 Secoco: Self-correcting encoding for neural machine translation . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4...

    2021 doi

  43. [51]

    Yibin Wang, Yichen Yang, Di He, and Kun He. 2023. https://doi.org/10.18653/v1/2023.findings-acl.42 Robustness-aware word embedding improves certified robustness to adversarial word substitutions . In Findings of the Association for Computational Linguistics: ACL 2023, pages 67...

    2023 doi

  44. [52]

    John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2020. https://doi.org/10.18653/v1/2020.acl-main.422 Similarity analysis of contextual word representation models . In Proceedings of the 58th Annual Meeting of the Association for Computatio...

    2020 doi

  45. [53]

    Weiwen Xu, Ai Ti Aw, Yang Ding, Kui Wu, and Shafiq Joty. 2021. https://doi.org/10.18653/v1/2021.naacl-industry.11 Addressing the vulnerability of NMT in input perturbations . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computation...

    2021 doi

  46. [54]

    Yichen Yang, Xiaosen Wang, and Kun He. 2022. https://openreview.net/forum?id=HOOMjIUscgq Robust textual embedding against word-level adversarial attacks . In The 38th Conference on Uncertainty in Artificial Intelligence

    2022

  47. [55]

    Jingyi Zhang, Gerard de Melo, Hongfei Xu, and Kehai Chen. 2023. https://doi.org/10.18653/v1/2023.wmt-1.45 A closer look at transformer attention for multilingual translation . In Proceedings of the Eighth Conference on Machine Translation, pages 496--506, Singapore. Associatio...

    2023 doi

  48. [56]

    Yuting Zhao and Ioan Calapodescu. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.582 Multimodal robustness for neural machine translation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8505--8516, Abu Dhabi, United Arab Em...

    2022 doi

  49. [57]

    Shuyan Zhou, Xiangkai Zeng, Yingqi Zhou, Antonios Anastasopoulos, and Graham Neubig. 2019. https://doi.org/10.18653/v1/W19-5368 Improving robustness of neural machine translation with multi-task learning . In Proceedings of the Fourth Conference on Machine Translation (Volume ...

    2019 doi

  50. [58]

    Yichu Zhou and Vivek Srikumar. 2022. https://doi.org/10.18653/v1/2022.acl-long.75 A closer look at how fine-tuning changes BERT . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1046--1061, Dublin, Irela...

    2022 doi

  51. [59]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. 2024. https://arxiv.org/abs/2306.04528 Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts ...

    2024 arXiv

  52. [60]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.