{"id":"56ed476d-afe8-40f8-ba1a-8b852532b27a","arxiv_id":"2412.10805","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Pre-trained language models are vulnerable to phonologically and orthographically motivated character substitutions in Indic languages, but less so than to unconstrained random character substitution.","lead":"This paper tests whether language models are more robust to linguistically grounded text changes, such as swapping similar-sounding letters, than to random character changes. It runs these attacks on 12 Indic languages and several multilingual models, and finds that linguistically grounded changes still break model predictions, though less than random edits.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Candidate-pool confound: the claim that PLMs are less susceptible to linguistic attacks is not identified without controlling for the larger random-candidate search space.","rationale":"The paper's strongest claim has two components: (1) linguistic attacks are effective, and (2) they are less harmful than non-linguistic attacks. Component (1) is supported by large accuracy drops (e.g., IndicBERTv2 sentiment: 0.937 → 0.276 for phonology). Component (2) is not supported as a statement about models because the attack families differ in candidate-pool size. The paper acknowledges this confound in §6.1.3 but does not control it. A model with zero knowledge of linguistic similarity would still exhibit lower success for the smaller candidate set. The detailed tables show that this confound is systematic across languages and models. Therefore the central relative claim is the least secure part of the paper. The proposed matched-pool random baseline would settle it. If the matched-pool baseline reproduces the linguistic-attack drops, then the title question should be answered 'yes, models are agnostic, conditional on search-space size'—or at least the paper should not claim lower susceptibility. If it does not reproduce, the claim is strengthened. This is a resolvable empirical question, so the conditional verdict remains appropriate; no change to the reader's verdict is needed, but the revision should be mandatory.","tokens_in":44235,"tokens_out":9230,"duration_ms":87881,"concrete_test":"Re-run the §3 attack pipeline with a matched-pool random baseline: for each target word, draw uniformly (without replacement) a random subset of same-type character substitutions whose size equals the mean candidate-pool size of the phonetic (or orthographic) attack, and use the identical word-ordering, similarity threshold, and query budget. Compare after-attack accuracy and label-flip rate for each language/task. If the matched-size random attack still drops accuracy by more than 5 points relative to the linguistic attack, the 'lower susceptibility' claim survives; if the drops converge, the relative claim is an artifact of candidate-pool size and should be reframed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The relative part of the central claim—that PLMs show 'slightly lower susceptibility to linguistic attacks' (Abstract, §6.1.3)—rests on a comparison between random and linguistically constrained character substitutions in which the candidate pool sizes are not matched. The paper itself acknowledges that 'higher number of perturbed candidates are generated in case of random attacks compared to linguistic attacks' (§6.1.3), but it never controls for this. If the model were fully agnostic to linguistic structure, a restricted candidate set should produce exactly the observed pattern: fewer opportunities to find a label-flipping substitution, hence lower attack success. The detailed tables confirm the confound: e.g., Table 9 (IndicSentiment, IndicBERTv2) reports avg candidates per word for Hindi of 3.75 (Rand), 2.725 (Phono), 1.075 (Ortho), and random attacks also consume more queries. The same pattern appears across languages and models. Consequently the evidence does not identify a model-level 'lower susceptibility' to linguistic perturbations; it identifies a property of the attack families' search spaces. The absolute vulnerability claim (linguistic attacks do flip predictions) remains well supported, but the title's 'agnostic' question is only answered if the search-space factor is removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether multilingual pre-trained language models are agnostic to linguistically grounded character-level perturbations, focusing on phonological and orthographic attacks across 12 Indic languages, three NLU tasks (IndicSentiment, IndicXParaphrase, IndicXNLI), and four models (IndicBERTv2, MuRIL, XLM-R, mBERT). The authors curate cross-script character-confusion resources, build a black-box attack pipeline based on a TextFooler-style word-importance score, and compare linguistic attacks against a random character-substitution baseline with LaBSE, chrF, and BERTScore filters. The absolute finding is that linguistic perturbations cause large accuracy drops: for example, IndicBERTv2 on IndicSentiment falls from 0.937 to about 0.28 after phonological attacks, and 0.462 after orthographic attacks in Table 1. A human evaluation on Hindi and Bengali indicates that linguistic perturbations are judged closer to the original in grammaticality and similarity than random perturbations. The abstract and Section 6.1.3 further claim that PLMs exhibit a slightly lower susceptibility to linguistic attacks than to random attacks. That relative claim is the main point of contention in this report, because the two attack families are not compared under matched search-space conditions.","tokens_in":44450,"tokens_out":7700,"duration_ms":72161,"significance":"If the absolute vulnerability finding stands, the paper provides a useful first systematic map of multilingual PLM robustness to linguistically natural perturbations in Indic languages, and the released character-similarity tables across nine scripts are a practical resource for future robustness work. The human evaluation is a genuine strength and supports the subtlety of the crafted perturbations. The paper does not rely on machine-checked proofs or fitted derivations; its value is empirical breadth and hand-curated linguistic resources rather than formal guarantees. The relative claim about agnosticism, however, is currently under-identified because random and linguistic attacks differ in candidate-pool size and possibly in filtering constraints. This weakens the headline interpretation and requires either additional controlled experiments or a carefully qualified reformulation.","major_comments":[{"comment":"The relative claim that PLMs are 'slightly lower susceptibility to linguistic attacks' is not identified because the random and linguistic attack families are compared with unconstrained candidate-pool sizes. The paper itself states in §6.1.3 that 'Due to unconstrained search space, higher number of perturbed candidates are generated in case of random attacks compared to linguistic attacks, contributing to their higher attack success.' Table 9 (IndicSentiment, IndicBERTv2, Hindi) reports average candidates per word of 3.75 for Random versus 2.725 for Phono and 1.075 for Ortho, with query counts 50.787 versus 47.856 and 36.864 respectively; the same ordering appears across languages, models, and tasks in Tables 9–20. If a model were fully agnostic to linguistic structure, a restricted candidate set would produce exactly the observed smaller accuracy drop. To answer the title's 'agnostic' question, the comparison should match candidate-pool sizes (e.g., subsample random candidates to the size of the linguistic pool for each word) or report attack success per candidate or per query. Without this control, the evidence supports the absolute claim that linguistic attacks flip predictions, but not the relative claim that models are less susceptible to them.","section":"§6.1.3, Table 9"},{"comment":"All results are averages over three random trials (§5.2), yet no error bars, standard deviations, or significance tests are reported for any after-attack accuracy. For example, §6.1.2 asserts that 'phonological perturbations cause more damage than orthographic perturbations' for IndicSentiment, and §6.1.4 ranks language families by robustness; these comparisons rely on differences that may be within seed noise, and the paper provides no evidence that the differences are stable across trials. Please report per-trial values or confidence intervals for at least the headline results in Table 1, and for the language-family and cross-model claims, or explicitly state that the observed differences were not tested for significance.","section":"§5.2, Tables 1, 9–20"},{"comment":"The word-importance saliency in Eq. (1), which masks a word and compares output probabilities, is inherited from TextFooler and is assumed to transfer to 12 Indic languages across three tasks without per-language validation. The paper does not check whether the [MASK]-based probability difference reliably identifies words whose substitution changes predictions in low-resource languages such as Bodo, or in Dravidian languages with complex morphology and longer words. If the heuristic is unreliable, the attack may select suboptimal target words, so the reported accuracy drops could misrepresent the models' true vulnerability to linguistic perturbations. The authors should validate the importance ranking against an oracle or an alternative saliency method on a subset of languages spanning at least one Indo-Aryan, one Dravidian, and one Sino-Tibetan language.","section":"§3, Eq. (1)"},{"comment":"The LaBSE cosine threshold of 0.6 is a free parameter chosen 'based on experiments with various threshold values,' but no sensitivity analysis is reported. Because this threshold filters candidate words and thus directly controls candidate-pool sizes and attack success rates, the main quantitative results, including the after-attack accuracy in Table 1, may depend on this arbitrary choice. Please provide a sensitivity analysis over thresholds (e.g., 0.5, 0.6, 0.7) for at least one model/task, and clarify whether the same threshold is applied to random attacks; if it is not, the random-versus-linguistic comparison compounds another uncontrolled difference.","section":"§5.4"}],"minor_comments":[{"comment":"Several column headers are run together without separators (e.g., 'OriginalAccuracyAfter-AttackAccuracy% PerturbedWordsQueryNumber'), and the appendix tables use 'A vg.' with a stray space in 'A vg. No. of Candidates per word'; this makes the tables unnecessarily hard to read.","section":"Table 1 and Appendix D"},{"comment":"The human evaluation reports average grammaticality and similarity scores from three annotators per language but provides no inter-annotator agreement measure (e.g., Cohen's kappa or per-annotator raw scores); please add this information or state that agreement was not computed.","section":"§6.2"},{"comment":"There is a typo in the sentence 'Dravidian languages are most robust to even synonym-baed substitution at the word level' — 'synonym-baed' should be 'synonym-based'.","section":"Appendix E"},{"comment":"In the mBERT or XLM-R appendix tables (e.g., Table 15, bd/Ortho row), the Phonetic Similarity column is listed as 0 even though phonetic similarity is presumably not applicable to orthographic attacks; use a dash to match the other orthographic rows.","section":"Table 15"},{"comment":"The paper says 'We release linguistic resources,' but I did not find a URL or repository link in the manuscript; please include the link in the camera-ready version so the resources are actually accessible.","section":"Contributions, Section 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's most valuable contribution is the curated cross-script character-similarity resources and the large-scale empirical vulnerability map across Indic languages, tasks, and models. The absolute finding that linguistic attacks cause large accuracy drops is solid and worth publishing. The relative 'agnostic' claim is currently confounded by candidate-pool sizes, but this is fixable either by adding matched-pool controls or by reformulating the conclusion to focus on the well-supported absolute result. I would also ask for error bars and a sensitivity analysis of the LaBSE threshold before acceptance. The paper fits an NLP robustness venue and I do not see a scope mismatch."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll cut to the chase. This paper is a solid empirical contribution for the resources alone: the first systematic, multi-task, multi-model evaluation of linguistically grounded character-level attacks across 12 Indic languages, plus hand-curated phonological and orthographic similarity mappings across nine scripts. The absolute vulnerability result is convincing: accuracy drops are large across models and tasks, and the human evaluation, though small, supports the claim that linguistic perturbations are subtle.\n\nThe problem is the relative claim in the title and abstract. The paper concludes that PLMs are \"slightly less susceptible\" to linguistic attacks than to random character attacks, but the comparison is confounded. Random attacks draw from a much larger candidate pool per word; the paper's own tables show this (e.g., Hindi IndicSentiment: 3.75 random candidates per word vs. 2.7 phonological and 1.1 orthographic). The authors acknowledge this in §6.1.3 but do not control for it. If the models were fully agnostic to linguistic structure, the restricted candidate sets would produce exactly the observed pattern—fewer flips simply because there are fewer options. The \"agnostic\" question is not actually answered by this comparison.\n\nOther soft spots are more minor: no error bars or significance tests despite averaging three random trials; the 0.6 similarity threshold is chosen post-hoc and affects all results; and there is no code release for the full pipeline, only the curated mappings in the appendix. The TextFooler importance-score heuristic is borrowed without per-language validation, but given the size of the accuracy drops, I'd treat that as secondary.\n\nBottom line: this deserves a serious referee. The benchmark is valuable, the resources are reusable, and the finding that linguistic perturbations are effective is solid. But the paper should either equalize candidate-pool sizes across attack types or explicitly restrict the conclusion to \"linguistic attacks are effective\" and drop the \"less susceptible\" language. I'd cite it for the resources and bring it to reading group to discuss the confound.","headline":"A useful first benchmark of linguistically grounded attacks for Indic languages, but the headline 'less susceptible' claim is confounded by candidate-pool size.","tokens_in":44961,"tokens_out":3377,"would_cite":true,"duration_ms":31672,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-trained multilingual models are not agnostic to linguistically grounded perturbations: replacing a single character with a phonetically or orthographically similar one flips predictions across 12 Indic languages and three tasks…","keywords":["adversarial attacks","linguistic perturbations","phonological similarity","orthographic similarity","Indic languages","multilingual language models","text classification robustness","character-level attack"],"falsifier":"Run the attack pipeline with random and linguistic candidate pools matched to the same size per word; if the accuracy drops become equal, the paper's conclusion that linguistic attacks cause less harm than random attacks is an artifact of search-space size rather than a property of linguistic structure.","tokens_in":44008,"feed_emoji":"🔤","tokens_out":12234,"duration_ms":107351,"temperature":0.7,"pith_summary":"Pre-trained multilingual models are not agnostic to linguistically grounded perturbations. Swapping a single character for a phonologically similar sound or a visually similar glyph flips predictions in all 12 Indic languages, across three downstream tasks and four models. Random character substitutions of the same character type cause even larger accuracy drops, so linguistic attacks appear slightly less harmful than unconstrained ones—but they remain effective, and they are far harder for a human reader to detect. Human evaluators rate the linguistically perturbed sentences as close to the original in grammar and meaning, which is what makes these edits a realistic real-world risk.","feed_headline":"No: single-letter linguistic swaps fool models in 12 Indic languages","feed_subtitle":"Phonetic and visual look-alike substitutions break sentiment, NLI, and paraphrase models—yet read as natural to humans.","key_machinery":"The carrying mechanism is a black-box attack pipeline. It first ranks words by a masking-based importance score, $I_{w_i}$, defined in Eq. (1) as the change in predicted-label probability when the word is replaced by the [MASK] token, following the method the paper adapts from its cited baseline. It then builds, for each important word, a candidate pool of single-character edits that are linguistically similar: phonologically similar characters (homorganic consonants, which share a place of articulation and differ only in voicing or aspiration; long/short vowel pairs; sibilants) or orthographically similar characters from a curated confusion set per script. Candidate edits are filtered to stay close to the original using LaBSE cosine similarity, chrF, BERTScore, and a phonetic similarity measure, with a threshold of 0.6. The random baseline substitutes a same-type character without any linguistic constraint, which gives it a larger candidate pool; the paper explicitly attributes the random attacks' higher success rate to this unconstrained search space.","core_discovery":"The paper's central discovery is that the answer to its title question is no: language models are not agnostic to linguistically grounded perturbations. When an important word in a sentence gets one character replaced by a phonetically similar character (a homorganic consonant, a short/long vowel counterpart, or another sibilant) or by a visually similar character from the same script, the model's prediction changes with high confidence. On IndicSentiment with IndicBERTv2, phonological attacks cut accuracy from 0.937 to 0.276 and orthographic attacks to 0.462, while random substitution of the same character type cuts it to 0.194, and the paper summarizes the pattern as linguistic attacks successfully deceiving the model while causing less harm than non-linguistic random attacks. Because the human evaluation shows linguistic edits preserve grammaticality and semantic similarity far better than random edits, the paper concludes that these constrained, linguistically plausible perturbations are an effective and more realistic threat than unconstrained random noise.","pith_inferences":["Editorial inference: the random-versus-linguistic comparison is not pool-matched; the paper's tables show random attacks drawing on more candidates per word than phonological attacks, so the lower harm of linguistic attacks could be a search-space effect rather than linguistic structure. An equal-candidate experiment would separate the two.","Editorial inference: the curated visual-similarity and phonetic-similarity confusion sets could be reused as a diagnostic suite or augmentation source for OCR/ASR robustness and spelling correction, not only as an attack generator.","Editorial inference: if the masking-based saliency ranking is unreliable in low-resource languages, the measured accuracy drops would misstate true vulnerability; a per-language check against human-identified keywords would settle whether the pipeline transfers."],"forward_implications":["Human judges cannot easily spot linguistically grounded edits, so these attacks are realistic routes into deployed systems rather than artificial worst cases.","OCR and ASR pipelines produce exactly these phonological and orthographic confusions, meaning everyday noisy input can flip a model's prediction without any adversarial intent.","Robustness differs sharply by language family: Dravidian languages survive better because their words are longer and morphologically richer, while low-resource Sino-Tibetan Bodo is the most fragile, so robustness claims need to be made per language.","Phonological attacks are more damaging than orthographic attacks in most settings, so defenses need to address specific confusion classes (homorganic consonants and vowel-length alternations) rather than treating all single-character edits alike."],"supporting_citations":[{"why":"Supplies the masking-based word importance score (Eq. 1) and the attack procedure the paper adapts to Indic languages.","marker":"Jin et al. (2020a)"},{"why":"Word-substitution attack approach the paper also builds on in its candidate-selection methodology.","marker":"Li et al. (2020b)"},{"why":"Provides the script conversion used to extend phonetic and orthographic confusion sets across scripts and the phonetic similarity measure used for filtering.","marker":"Kunchukuttan (2020)"},{"why":"IndicX-TREME benchmark supplies the three tasks—IndicSentiment, IndicXParaphrase, IndicXNLI—used in the evaluation.","marker":"Doddapaneni et al. (2023a)"},{"why":"Provides the IndicBERTv2 model and the fine-tuning hyperparameters used for all models.","marker":"Doddapaneni et al. (2023b)"},{"why":"Provides the MuRIL multilingual model evaluated in the study.","marker":"Khanuja et al. (2021)"},{"why":"Provides the XLM-R model evaluated in the study.","marker":"Conneau et al. (2020)"},{"why":"Provides the mBERT model evaluated as one of the four multilingual models.","marker":"Devlin et al. (2019)"},{"why":"LaBSE sentence embeddings are the semantic-similarity filter that keeps perturbed sentences close to the originals.","marker":"Feng et al. (2020)"},{"why":"IndoWordNet supplies the synonym candidates for the word-level substitution experiments reported in the appendix.","marker":"Bhattacharyya (2010)"}],"fun_headline_variants":["Linguistic look-alikes fool Indic language models","Phonetic and visual char swaps trick 12 Indic languages","Indic models not agnostic to linguistically grounded edits","Single-char swaps undermine Indic NLP tasks","Even subtle, natural edits deceive Indic PLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measurement depends on the masking-based word importance score inherited from an English-focused attack method ranking the truly decision-relevant words correctly in all 12 languages, although it was never validated per language.","fun_headline_variants_meta":{"raw":{"variants":["Linguistic look-alikes fool Indic language models","Phonetic and visual char swaps trick 12 Indic languages","Indic models not agnostic to linguistically grounded edits","Single-char swaps undermine Indic NLP tasks","Even subtle, natural edits deceive Indic PLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1607,"prompt_tokens":876,"completion_tokens":731,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":654}},"tokens_in":492,"tokens_out":731,"duration_ms":7125,"temperature":1.0,"reasoning_tokens":654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:35:29.004694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the attack pipeline with random and linguistic candidate pools matched to the same size per word; if the accuracy drops become equal, the paper's conclusion that linguistic attacks cause less harm than random attacks is an artifact of search-space size rather than a property of linguistic structure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the script conversion used to extend phonetic and orthographic confusion sets across scripts and the phonetic similarity measure used for filtering."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IndoWordNet supplies the synonym candidates for the word-level substitution experiments reported in the appendix."}],"review_version":1}