{"id":"c0a36927-5bbd-4cc5-9c66-c59cb490c36f","arxiv_id":"2505.07856","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Circuits extracted from inflectionally varied Polish sentences are more robust to adversarial word attacks than circuits from syncretic or English variants, identifying layer-0 attention heads as inflection-specific.","lead":"Researchers tested four adversarial word-attack methods on English, Polish, and Czech text classifiers and measured how much accuracy dropped. Using circuit discovery, they identified internal model components that process Polish word endings and found those components help models resist attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Circuit-robustness attribution is confounded: 'inflectional' and 'syncretic' circuits differ in entire EAP-IG edge sets, so the robustness gain in §5.4 cannot be assigned to the identified a0 heads without causal ablation.","rationale":"The reader's weakest assumption was the cross-language comparison in Section 4.2, where different pretrained models and datasets confound language with architecture and task difficulty. That is a valid concern, but the paper's more controlled mechanistic experiment (same XLM-RoBERTa model, same MultiEmo task, Polish vs English data) is where the strongest causal claim about inflection actually lives. In that experiment, the load-bearing flaw is the comparison of independently discovered EAP-IG circuits: the 'inflectional' circuit differs from the 'syncretic' circuit in the entire set of selected edges, not just in the five attention heads highlighted in Figure 5. Any robustness difference in Table 5 could therefore be caused by any of the many changed edges, or by the different synonym sets used to construct the parallel corpus, rather than by the inflection-specific heads. This is a correctable but central issue: a matched ablation or activation-patching study of the five heads would settle it. Because the concern is addressable with additional analysis rather than being a fundamental flaw in the research question, the conditional verdict stands; no change to the reader's verdict is needed, but the mechanistic conclusion should be explicitly downgraded until such a test is run.","tokens_in":14222,"tokens_out":5241,"duration_ms":54540,"concrete_test":"Re-run the Section 5.4 robustness comparison with matched edge sets: for each graph size, start from the Polish Syncretic circuit's edge set and add only the outgoing edges of heads a0.h3, a0.h5, a0.h6, a0.h7, a0.h11 (equivalently, remove exactly those heads from the Inflectional circuit), keeping all other edges identical. If the robustness gap between Inflectional and Syncretic circuits disappears or reverses in this matched comparison, the improvement attributed to inflection-specific heads is an artifact of EAP-IG's different edge selections rather than a causal effect of those heads. Report per-size and per-attack accuracy with bootstrap confidence intervals over the 244 examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanistic claim (Sections 5.4 and 6.1) is that adding inflection-specific components — heads a0.h3, a0.h5, a0.h6, a0.h7, a0.h11 — improves adversarial robustness. This attribution is not established by the experiment as run. In Section 5.3, EAP-IG selects the top-N edges separately for each dataset variant (Polish Syncretic, Polish Inflectional, English). The three 'circuits' are therefore different subgraphs of the model, not a common circuit with a controlled addition or removal of the Layer-0 heads. When the corrupted dataset changes from syncretic to inflectional, the importance scores of all edges change, so the Inflectional circuit can differ from the Syncretic circuit in many components beyond the five heads; Figure 5 shows added triangular nodes and removed dotted nodes as well. Comparing the robustness of these independently selected circuits in Table 5 cannot establish that the five heads cause the improvement. Moreover, Table 5 shows the Inflectional circuit is not consistently best cell-by-cell (e.g., at sizes 75, 100, 150 the Syncretic or EN circuits often score higher); only the means are highest, and no variance or significance testing is reported for the 244-example dataset. The dataset construction also filters out pairs where predictions do not change and uses different synonym sets across variants, so the heads may reflect lexical or tokenization differences rather than inflectional morphology. A causal test on the heads themselves is needed before the Section 6.1 conclusion can stand.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates adversarial robustness in inflectional languages (Polish, Czech) versus English across two classification tasks, comparing four attack methods (TextBugger, TextFooler, WordNetTextFooler, BERT-Attack) and three similarity metrics (ROUGE, NLI, Semantic Similarity). It then introduces a parallel 'Inflectional Circuit Detection Dataset' built from MultiEmo and uses EAP-IG to extract circuits for Polish syncretic, Polish inflectional, and English variants of the same sentiment task. The authors report that the Polish inflectional circuit contains additional Layer-0 attention heads (a0.h3, a0.h5, a0.h6, a0.h7, a0.h11) and that this circuit shows the highest average robustness under the four attacks, concluding that incorporating inflection in training data can improve adversarial robustness. The paper also reports a comparison of similarity metrics, finding TextBugger attacks less semantically faithful and NLI similarity unstable.","tokens_in":14536,"tokens_out":3806,"duration_ms":36758,"significance":"If the central claims were established, the paper would make a useful contribution: it targets an understudied question (adversarial robustness in inflectional languages), proposes a new parallel dataset for circuit discovery, and applies mechanistic interpretability methods to a non-English language. The dataset and the EAP-IG protocol are potentially reusable assets. However, the load-bearing conclusions are not currently supported: the cross-language comparison is confounded by different models and datasets, and the circuit-robustness comparison compares entire independently selected subgraphs rather than isolating the identified heads, so the causal attribution to inflection-specific heads is not established. The manuscript would need substantial additional analysis, including causal ablations and statistical testing, before the main claims can be accepted.","major_comments":[{"comment":"The cross-language robustness comparison treats language as the independent variable, but each language uses a different pretrained model (BERT, HerBERT, RobeCzech) and different datasets, with Czech baselines at 83-86% accuracy versus 94% and above for Polish and English. Any difference in robustness under attack could be due to model architecture, pretraining corpus, task difficulty, or dataset size rather than inflectional morphology. The conclusion in Section 6.1 that 'incorporating inflexion in language model training corpus can increase its robustness' is therefore not supported by this comparison alone. The authors should either control for these variables, provide a causal identification strategy, or substantially soften the claim.","section":"Section 4.2, Tables 1-2"},{"comment":"The claim that the five Layer-0 heads (a0.h3, a0.h5, a0.h6, a0.h7, a0.h11) are causally protective is not established. The Polish Syncretic and Polish Inflectional circuits are separately extracted subgraphs selected by EAP-IG on different corrupted datasets; Figure 5 shows that the circuits differ in many added and removed components, not just the Layer-0 heads. Comparing the adversarial robustness of these whole circuits cannot isolate the effect of the identified heads. A causal test, such as ablating or patching only the candidate heads within a fixed circuit, is needed to support the attribution. Additionally, Table 5 shows the inflectional circuit is not consistently better cell-by-cell (e.g., at sizes 75, 150, 200 for TextFooler, or size 200 for TextBugger), so 'consistently achieved the highest average robustness' overstates the evidence.","section":"Sections 5.3-5.4, Figure 5, Table 5"},{"comment":"The Inflectional Circuit Detection Dataset is constructed with a selection step that removes clean-corrupted pairs for which the model's prediction does not change. This filters on the outcome variable and can bias circuit discovery toward pairs that already produce model disagreement, potentially inflating or distorting EAP-IG importance scores. The dataset is also expanded with Llama-3.3-70B-Instruct without any validation of the generated examples, and the three variants use different synonym sets; thus the observed circuit differences may reflect lexical or tokenization properties rather than inflectional morphology per se. The authors should report the number of excluded pairs, validate the LLM-generated examples, and assess sensitivity to the synonym selection.","section":"Section 5.1, dataset construction"},{"comment":"The manuscript states that the MultiEmo-based mechanistic analysis used a training part of 334,960 examples and a test part of 41,710 examples, but Table 1 reports the MultiEmo PL dataset as having 4,319 training and 680 test examples. This is a large discrepancy that affects reproducibility and the interpretation of the circuit experiments. The authors need to clarify exactly which data split was used for the XLM-RoBERTa model in Section 5 and reconcile the numbers.","section":"Section 5, vs. Table 1"},{"comment":"No statistical significance testing or variance reporting is provided for the circuit robustness comparisons. The differences between the inflectional and syncretic circuits at many individual graph sizes appear small (e.g., 48.05 vs. 48.05 for TextBugger at size 150), and the means are driven substantially by the size-50 row. A paired bootstrap or other appropriate test across the 244 examples should be reported before claiming that the inflectional circuit is more robust.","section":"Table 5"}],"minor_comments":[{"comment":"There is inconsistent spelling of 'inflection/inflexion'; the abstract and Section 6.1 use 'inflexion' while the rest of the paper uses 'inflection'. Please standardize.","section":"Throughout"},{"comment":"Several numeric entries use a comma as a decimal separator (e.g., '95,17') while others use a period; please use a consistent format.","section":"Table 1"},{"comment":"The TextFooler and WordNetTextFooler rows for Wiki_PL report identical ROUGE and Semantic similarity values (96.36 ± 3.56 and 97.52 ± 1.43). This looks like a data-entry or copying error and should be checked.","section":"Table 3, Wiki_PL rows"},{"comment":"The text says 'Mechanical Interpretability' where 'Mechanistic Interpretability' is meant; also the EAP reference [21] is to a blog post, and the formal EAP citation (Nanda, 2023) may be preferable.","section":"Section 5.2"},{"comment":"The axis label 'Semanctic' is misspelled; it should be 'Semantic'.","section":"Figures 2 and 3"},{"comment":"The sentence 'TextBugger consistently produced statistically lower-quality attacks' is not supported by any reported statistical test; please either add the test or rephrase to 'lower-quality on the reported metrics'.","section":"Section 6.1"},{"comment":"The paper does not state where the dataset, code, or trained circuits will be released; given the new benchmark is a claimed contribution, a reproducibility statement would be helpful.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core mechanistic claim (Section 6.1) rests on comparing independently selected EAP-IG circuits, and the cross-language comparison in Section 4.2 is not a controlled experiment. I would advise the editor that the paper needs a causal ablation of the identified heads and a statistically grounded analysis of Table 5 before it can be considered for publication. The dataset and protocol are promising, but the current evidence does not support the strong conclusions drawn in the abstract and Section 6.1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jan, quick take on arXiv:2505.07856. The paper asks a good question—does inflectional morphology change adversarial robustness, and can we identify the circuits that process it?—but the headline mechanistic conclusion doesn't survive contact with the experiment as run. Still, there's usable material here: the dataset design and the negative lesson are worth your time.\n\nWhat's actually new: the Inflectional Circuit Detection Dataset (three parallel 244-example sets: Polish syncretic, Polish inflectional, English), the first EAP-IG comparison across these variants, and the four-attack evaluation in Polish, Czech, and English. The similarity-metric analysis (ROUGE vs NLI vs semantic similarity) is solid descriptive work; they show the NLI metric is unstable and identify which attack methods produce low-quality examples. Citation pattern is fine—WordNetTextFooler is their own earlier method and they cite it as such.\n\nThe soft spots are load-bearing. Section 4.2 compares robustness across languages but each language uses a different pretrained model (BERT, HerBERT, RobeCzech) and different datasets, with Czech baselines at 83–86% accuracy versus 94%+ for Polish and English. You cannot attribute observed accuracy drops to inflection without addressing those confounds. That's fixable, at least partially, by narrowing the claims or adding matched control models.\n\nThe deeper problem is in Section 5. EAP-IG independently selects top-N edges per dataset variant. The Polish Inflectional and Polish Syncretic circuits are different subgraphs—Figure 5 shows added and removed nodes beyond the five Layer-0 heads. Comparing robustness of these separately selected circuits (Table 5) does not isolate the heads a0.h3, a0.h5, a0.h6, a0.h7, a0.h11. The stress-test note is right: you need a causal ablation on the heads themselves. Also, Table 5 shows the inflectional circuit is not consistently best cell-by-cell; only the means are higher, with no variance or significance tests on 244 examples. The dataset also excludes unchanged-prediction pairs and adds LLM-expanded examples without validation, which can bias the EAP scores.\n\nVerdict: the paper deserves serious peer review, but with a major revision. Release the dataset, run a proper causal test (patch the five heads into the syncretic circuit, or ablate them from the inflectional one), and either fix the cross-language confounds or scale back the generalization claim. If the causal test fails, the paper still works as a descriptive study of adversarial attacks in inflected languages—minus the 'inflection-specific components' claim. For a reading group, it's a good case study in why circuit comparisons need matched subgraphs. I wouldn't cite it until the data and the causal test are out.","headline":"Good question and a useful dataset, but the causal claim about inflection circuits is not established by the experiment as run.","tokens_in":15051,"tokens_out":2912,"would_cite":false,"duration_ms":26630,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that inflectional morphology in a language model's training data increases its resilience to word-form adversarial attacks, and that this resilience is carried by five specific attention heads in the model's first layer.","keywords":["adversarial examples","inflectional morphology","mechanistic interpretability","circuit discovery","text classification","sentiment analysis","Polish","robustness"],"falsifier":"Fine-tune one pretrained model on matched English and Polish sentiment data of similar length and difficulty, then run TextBugger and the other three attacks; if English attack accuracy is not below Polish, the inflection-robustness claim is unsupported. Alternatively, patch out the five layer-0 heads a0.h3, a0.h5, a0.h6, a0.h7, a0.h11 from the Polish circuit and check whether adversarial accuracy drops to the syncretic circuit's level; if it does not, those heads are not the mechanism.","tokens_in":14054,"feed_emoji":"🛡️","tokens_out":8222,"duration_ms":75677,"temperature":0.7,"pith_summary":"This paper asks whether the inflectional complexity of a language changes how vulnerable text classifiers are to adversarial examples. It compares Polish, Czech, and English under four attack methods and reports that character-level word-bugging attacks (TextBugger) are the most effective, while synonym-substitution attacks are gentler but less damaging. Its central claim is that models trained on inflectionally rich language become more resilient to word-form attacks, and that this resilience is concentrated in identifiable attention heads in the transformer's first layer. To support this, the authors build a parallel Polish/English dataset of sentiment sentences that differ only in whether the key noun is syncretic or regularly inflected, and use circuit-discovery techniques to isolate the components that react to inflection. A reader should care because adversarial robustness is usually studied on English, and this work suggests morphological variation is a defensive resource, not just a complication.","feed_headline":"Inflection in training data hardens models against word-bugging attacks","feed_subtitle":"Mechanistic analysis finds Polish attention heads that keep sentiment predictions stable when words are corrupted.","key_machinery":"The central object is the circuit, a minimal subgraph of a transformer made of attention heads and MLP layers connected by high-importance edges. The method for finding circuits is Edge Attribution Patching with Integrated Gradients (EAP-IG), which scores each directed edge by its contribution to the model's confidence in the correct class when inputs are switched from clean to corrupted. The paper's own instrument is the Inflectional Circuit Detection Dataset, a parallel Polish/English sentiment corpus whose clean and corrupted sentences differ by a single noun: one variant uses a syncretic noun (a single form covering several grammatical cases) and another uses a regularly inflected synonym. Because clean and corrupted inputs keep identical token counts, the circuit-discovery pipeline can attribute the prediction difference to specific edges, exposing the five layer-0 heads that respond to inflection.","core_discovery":"The central claim is that inflectional morphology can act as a defensive resource in text classification. In the authors' experiments the Polish and Czech sentiment models generally lost less accuracy under attacks than the English model, and the Polish inflectional circuit outperformed both the syncretic Polish circuit and the English circuit in mean accuracy under all four attack methods. Mechanistically, the paper identifies five attention heads in layer 0 of the multilingual model — a0.h3, a0.h5, a0.h6, a0.h7, and a0.h11 — that appear in circuits built from inflected Polish inputs across all circuit sizes and are absent from the English reference circuit. The authors interpret these heads as the components that track inflectional endings, and argue that when an attack modifies part of a word, circuits containing these heads preserve classification confidence better than circuits without them.","pith_inferences":["If the protective effect is real, then multilingual models may become adversarially more robust as the morphological richness of their pretraining corpus rises; a direct test would fine-tune one architecture on artificially inflected English and measure whether TextBugger attack success falls.","Because the inflection heads are found in a multilingual model, the same heads may track case or gender contrasts in other inflection-rich languages such as Czech, German, or Russian; this could be checked by running the same circuit-discovery protocol on Czech parallel data and comparing circuit overlap.","The paper's cross-language comparison varies language together with model architecture and dataset, so its strongest claim would need a matched-design study holding architecture and data source fixed while varying only morphological complexity.","If the heads are causal rather than merely correlational, then ablating or patching just these five heads from the Polish circuit should remove most of its adversarial advantage; that experiment is a natural next step the paper does not run."],"forward_implications":["In sentiment analysis, the Polish and Czech models retained more accuracy under all four attacks than the English model, so the robustness advantage of inflectional languages appears specifically in sentiment tasks, not in topic classification.","TextBugger is the most effective attack across languages and tasks, yet its outputs score lowest on statistical similarity, meaning the attack that works best is also the easiest to detect by ROUGE.","ROUGE and semantic similarity are stable measures of adversarial example quality, while NLI-based similarity is unreliable because a tail of examples receives near-zero entailment scores.","The five layer-0 heads a0.h3, a0.h5, a0.h6, a0.h7, and a0.h11 recur in the Polish inflectional circuit at every circuit size and are absent from the English circuit, localizing inflection processing to a compact set of components."],"supporting_citations":[{"why":"Supplies the EAP-IG circuit-discovery method that identifies the inflection-related attention heads.","marker":"[9]"},{"why":"The XLM-RoBERTa model whose Polish and English MultiEmo classifiers are analyzed.","marker":"[5]"},{"why":"MultiEmo is the sentiment corpus from which the Inflectional Circuit Detection Dataset is built.","marker":"[12]"},{"why":"Introduces Edge Attribution Patching, the base method EAP-IG improves on.","marker":"[21]"},{"why":"Defines the notion of a circuit that the paper uses to localize inflection processing.","marker":"[22]"},{"why":"Source of syncretic noun forms used to construct the clean-corrupted parallel dataset.","marker":"[26]"},{"why":"Polimorf supplies the inflectional word forms needed to build regular-inflection variants.","marker":"[35]"},{"why":"Defines TextBugger, the character-level attack that drives the main robustness findings.","marker":"[14]"}],"fun_headline_variants":["Inflectional morphology boosts adversarial robustness in NLP models","Polish attention heads defend against word attacks","Inflection helps models resist word-bugging attacks","Mechanistic insight: inflectional circuits preserve sentiment","Inflected training data strengthens text classifiers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central conclusion rests on treating language as the cause of the robustness gap, even though each language uses a different pretrained model and different datasets; if the gap comes from architecture or data difficulty rather than inflection, the conclusion does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Inflectional morphology boosts adversarial robustness in NLP models","Polish attention heads defend against word attacks","Inflection helps models resist word-bugging attacks","Mechanistic insight: inflectional circuits preserve sentiment","Inflected training data strengthens text classifiers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001284,"raw_usage":{"total_tokens":5231,"prompt_tokens":915,"completion_tokens":4316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":4246}},"tokens_in":531,"tokens_out":4316,"duration_ms":29348,"temperature":1.0,"reasoning_tokens":4246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:14:45.183536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune one pretrained model on matched English and Polish sentiment data of similar length and difficulty, then run TextBugger and the other three attacks; if English attack accuracy is not below Polish, the inflection-robustness claim is unsupported. Alternatively, patch out the five layer-0 heads a0.h3, a0.h5, a0.h6, a0.h7, a0.h11 from the Polish circuit and check whether adversarial accuracy drops to the syncretic circuit's level; if it does not, those heads are not the mechanism.","supporting_citations":[{"cited_title":"Hanna, S","cited_arxiv_id":null,"evidence_quote":"Supplies the EAP-IG circuit-discovery method that identifies the inflection-related attention heads."},{"cited_title":"Koco ´n, P","cited_arxiv_id":null,"evidence_quote":"MultiEmo is the sentiment corpus from which the Inflectional Circuit Detection Dataset is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Edge Attribution Patching, the base method EAP-IG improves on."},{"cited_title":"Saloni, M","cited_arxiv_id":null,"evidence_quote":"Source of syncretic noun forms used to construct the clean-corrupted parallel dataset."},{"cited_title":"Woli´nski","cited_arxiv_id":null,"evidence_quote":"Polimorf supplies the inflectional word forms needed to build regular-inflection variants."}],"review_version":1}