{"id":"af6618eb-f864-42bd-9514-271a1cf6d71f","arxiv_id":"2602.04613","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across 20 language directions and three model families, roughly 1% of attention heads — split into separable language-identification and meaning-equivalence groups — causally drive LLM translation, and steering them reproduces instruction-based translation quality.","lead":"Large language models carry out translation by delegating the work to a small, dedicated set of internal processing units: one group picks the output language, another preserves the meaning. Injecting signals from just those units makes a model translate from a bare prompt nearly as well as with explicit instructions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"§3.2's sentence-equivalence corruption alters lexical/syntactic statistics along with meaning, so the identified 'translation heads' may encode task coherence rather than semantic equivalence; this decomposition confound is the core load-bearing concern.","rationale":"The reader's weakest_assumption — that the two corrupted prompt families isolate exactly the intended subtasks — is the same concern I consider most load-bearing. The c_MT corruption is a broad intervention: replacing target translations with random same-language sentences changes not only semantic equivalence but also target-side lexical statistics, syntactic template distribution, and the coherence of the few-shot mapping. Activation patching (Section 3.1) then measures which heads restore log-probability of the correct next token; these heads could be detecting 'the prompt defines a translation task' rather than 'the target should be equivalent in meaning to the source.' The steering and ablation experiments are strong and partially mitigate this: steering the identified translation heads induces actual translation (not random text), and ablating them degrades quality with distinct failure modes. This suggests the heads are not merely detecting language. However, no experiment compares against an alternative corruption that breaks meaning while preserving translation-like surface statistics, so the semantic interpretation of the 'translation heads' remains underdetermined. I propose a mismatched-pair corruption as a concrete discriminator. I agree with the reader's identification of this as the weakest assumption. I do not think the paper's core empirical finding (sparse heads are causally involved in MT and can be steered) is overturned, but the decomposition claim should be conditional on this control passing. Hence CONDITIONAL rather than REJECT. The paper otherwise shows good practice: multiple model families, random-ablation control, transferability checks, and qualitative failure modes. Code is linked but not verifiable from the text, another reason for a conditional stance.","tokens_in":102689,"tokens_out":10277,"duration_ms":118294,"concrete_test":"Construct a third corruption c_mismatch: for each demonstration (x_i, y_i), keep y_i but pair it with a different source sentence x_j (j≠i) from the same dev split, so y_i is a correct Ltgt translation of some other source, preserving translation-like surface statistics while breaking source–target equivalence. Run head identification (§3.1) on Gemma-3-12B for English→French with c_mismatch and compare the top-1% translation heads with those from the original random-target c_MT (Jaccard overlap) and steer with the c_mismatch heads. If Jaccard > 0.7 and steering BLEU is within ~2 of the original, the random-statistics confound is not the driver; if not, the 'sentence equivalence' heads are confounded by non-semantic prompt statistics.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper decomposes MT into target language identification and sentence equivalence using two corrupted prompt families (§3.2). The sentence-equivalence corruption c_MT replaces each demonstration target with a random sentence in the correct target language. This removes source–target meaning correspondence, but it also changes target-side lexical/syntactic statistics, makes the prompt a set of unrelated Q/A pairs, and alters the difficulty of inferring a coherent input–output mapping. Activation patching (§3.1) then identifies heads that restore the correct next-token probability when patching from corrupted to clean; these heads may be detecting 'this prompt is/is not a coherent translation task' rather than specifically 'the target should preserve the source's meaning.' Similarly, c_lang replaces each target with a correct translation in a language selected from a set, which may mix languages across demonstrations and conflate target-language identity with mapping consistency. The steering and ablation experiments show the identified heads are causally involved in MT and target-language choice, but they do not rule out that 'translation heads' encode task coherence rather than sentence equivalence. The paper does not include a limitations discussion acknowledging this confound, which directly bears on the central decomposition claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a mechanistic decomposition of sentence-level machine translation in LLMs into two subtasks: target language identification and sentence equivalence. Using activation patching on two corrupted few-shot prompt families — wrong/mixed target languages, and random same-language target sentences — the authors identify roughly 1% of attention heads as specialized per subtask across Gemma-3, Qwen-3, and Llama-3 models and 20 translation directions. Steering these heads with mean-activation vectors under an instruction-free zero-shot prompt yields translation quality close to instructed zero-shot prompting (e.g., 33.3 vs 32.6 BLEU for Gemma-3-12B English→X), while ablating them produces function-specific failures (language switching vs. loss of meaning). Equivalence vectors transfer across directions. The paper also analyzes the effect of shot count, amplification factor, token position, and compares the identified heads with those from prior word-level MT studies.","tokens_in":102944,"tokens_out":5975,"duration_ms":66452,"significance":"If the causal claims hold, the findings are significant for mechanistic interpretability and MT: they provide evidence that a sparse set of attention heads can control an entire sentence-level NLP task, that the heads split into largely disjoint functional groups, and that steering only these heads suffices to elicit translation without instructions. The breadth — three model families, multiple scales, 20 directions, several automatic metrics — and released code are strengths. The causal validation via steering and ablation is the most compelling part and gives the paper real value. However, the semantic interpretation of the two head classes rests on an unexamined assumption that the two corruptions isolate exactly the intended subtasks; the paper would be materially strengthened by control experiments and an explicit limitations discussion.","major_comments":[{"comment":"Replacing each demonstration target with a random same-language sentence removes source–target meaning correspondence, but it also changes target-side lexical/syntactic statistics, discourse coherence, and the ease of inferring a coherent input–output mapping. Activation patching may therefore identify heads that encode 'this prompt is/is not a coherent translation task' rather than 'preserve the source's meaning.' This confound directly underdetermines the central claim that translation heads encode sentence equivalence. The paper provides no control (e.g., permuting Q/A pairs while using genuine translations, or using translations of different source sentences) and no limitations discussion. Please add such controls or explicitly temper the semantic interpretation.","section":"§3.2, sentence-equivalence corruption c_MT"},{"comment":"The text says each demonstration target is replaced by a correct translation in a language selected from the set L, while the Figure 1 caption describes the corruption as 'mixing multiple target languages.' Mixing languages removes cross-demonstration target-language consistency in addition to removing the intended target language, conflating target-language identity with mapping consistency. If the goal is to isolate target-language identification, a cleaner corruption would use a single wrong target language for all demonstrations, or at least report both variants. The causal steering results soften this concern but do not fully resolve the identification confound.","section":"§3.2, language-identification corruption c_lang"},{"comment":"The headline 'comparable' claims rest on small BLEU differences (e.g., 33.3 vs 32.6 for Gemma-3-12B English→X) on a single devtest split with greedy decoding. No confidence intervals, bootstrap resampling, or significance tests are reported. With roughly 1,012 sentences, a 0.7 BLEU difference may be within noise. Please provide uncertainty estimates and, if possible, per-direction paired comparisons. This matters because the steering-sufficiency claim is one of the paper's main conclusions.","section":"§5, Figs. 3–4"}],"minor_comments":[{"comment":"Typos and small phrasing issues: 'a instruction-free' in the Figure 5 caption; 'ANALISYS' in Appendix B.4.1; 'minimal performance loss performance' in Section 7. The abstract says 'three families' while the setup lists four model families (Gemma-3, Llama-3.2, Llama-2-7B, Qwen-3); clarify the grouping.","section":"General"},{"comment":"The text says '5 to 10 heads' and gives '5 out of 1,024' as an example; specify the model and total head count corresponding to each number. The heatmaps in Figures 3–4 are dense; increasing font size and labeling the axes more explicitly would help.","section":"§5"},{"comment":"The citation 'Anonymous (2025)' appears in the related work and in the head-classification comparison. In a final version this should be de-anonymized or clearly marked as a self-citation, depending on the submission venue's anonymity policy.","section":"§6 / References"},{"comment":"The notation DKL is used without specifying the base (natural log vs log2); the KL values are only used comparatively, but stating the base would be cleaner.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a strong causal component, but the identification confound in §3.2 is load-bearing for the 'sentence equivalence' label and should be addressed with controls or a clear limitation statement before publication. The lack of significance testing on the small BLEU gaps is also a correctness-risk issue. I would not reject the paper, but I would not accept it in its current form. The reference to 'Anonymous (2025)' appears to be the authors' own anonymized work; if this is not a double-blind venue, it should be de-anonymized."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is genuinely new: sentence-level activation patching, with a KL-divergence rule for choosing the target position, identifies sparse attention head sets for target-language identity and sentence equivalence, and steering roughly 1% of those heads reproduces instructed zero-shot MT across several model families. The causal validation is the strong part. Steering and ablation experiments have the right controls, including random head ablation, and the failure modes are distinct: ablating language heads causes language switching, ablating translation heads degrades quality while keeping the language. The equivalence-vector transfer across directions is a nice addition, and the stability of identified heads across number of shots and token positions is reassuring. Code is promised, and the main hyperparameter (alpha) is fixed at 1 with a sensitivity study, which is honest. This deserves a serious referee.\n\nThe soft spot is the load-bearing premise in §3.2. The sentence-equivalence corruption replaces each target with a random same-language sentence. That removes source-target meaning, but it also changes lexical/syntactic statistics, makes the prompt a set of unrelated Q/A pairs, and alters cue difficulty. So the \"translation heads\" could partly encode task coherence rather than semantic equivalence specifically. Steering and ablation show these heads are causally involved in MT, but they do not pin down the semantic-equivalence interpretation. The language corruption has a milder version of the same issue, since mixing languages across demonstrations also changes mapping consistency. The paper does not acknowledge this confound explicitly. It is a real limitation, but not a fatal one, because the qualitative dissociations and cross-direction transfer are hard to explain purely by task coherence.\n\nMinor issues: all 20 directions are English-centric, so claims of generality rest on a symmetric but still English-centered sample; the code repository cannot be verified from the text. These are minor.\n\nBottom line: a well-executed, careful mechanistic interpretability study that advances the sentence-level MT front. The decomposition claim should be read with the confound in mind, but the causal core is solid. I would take this seriously as a referee and would likely cite it in MT or interpretability work.","headline":"Solid empirical MI paper; the sparse-head decomposition mostly holds up, but the corruption-based subtask isolation is cleaner in presentation than in practice.","tokens_in":103433,"tokens_out":1277,"would_cite":true,"duration_ms":22110,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that sentence-level machine translation in large language models is carried by a sparse set of attention heads — roughly one percent — split into mostly disjoint 'language heads' that pick the output language and 'translati","keywords":["machine translation","mechanistic interpretability","attention heads","activation patching","steering vectors","target language identification","sentence equivalence","in-context learning"],"falsifier":"A decisive check: construct a corruption that swaps every content word of the demonstration targets for same-language synonyms (preserving syntax, length, and target language) and see whether activation patching still isolates the same 'translation heads'; if the head set changes drastically, the claimed equivalence subtask is an artifact of the particular corruption. Alternatively, ablate the top 'language heads' on a direction into English, where the paper reports language heads matter less — if target-language accuracy stays at random-ablation levels but BLEU still collapses, the functional","tokens_in":102564,"feed_emoji":"🧠","tokens_out":4394,"duration_ms":48353,"temperature":0.7,"pith_summary":"The paper tries to establish that LLM sentence-level translation is mechanistically localized: a tiny fraction of attention heads (~1%) causally drives the task, and this set divides into two functional groups — language heads that determine the target language and translation heads that maintain meaning equivalence with the source. Using activation patching on contrastive few-shot prompts, the authors identify these heads across three model families and twenty language directions, finding the two groups mostly disjoint (Jaccard index around 0.13). They then build steering vectors from the heads' averaged activations and show that steering the top 1% of each group under a prompt-free zero-shot setting matches or approaches instructed zero-shot translation quality, while ablating them selectively destroys the corresponding function. If true, this means translation capability can be elicited and disabled through a very small, targeted set of components, and that 'which language' and 'what it means' are represented separately inside the model.","feed_headline":"Sparse 1% of heads drives LLM translation","feed_subtitle":"Steering those heads alone matches instruction prompting; language and meaning live in disjoint head sets.","key_machinery":"The central object is the attention head set identified by activation patching with contrastive few-shot prompts. For sentence-level outputs, the paper introduces a KL-divergence criterion to pick the target token position where clean and corrupted prompts diverge most, then patches activations at that position. The identified heads are used to build mean-activation steering vectors (scaled by an amplification factor) that are added to the residual stream, turning an instruction-free prompt into a translation prompt.","core_discovery":"The central claim is that LLM sentence-level MT decomposes into two causally separable subtasks, each mediated by its own sparse set of attention heads. Language heads — identified by corrupting demonstration targets into a different language while preserving meaning — control the output language; translation heads — identified by replacing targets with random same-language sentences — control sentence equivalence. Repeatedly, only 5–10 heads out of thousands stand out, the same heads recur across translation directions, and the two groups barely overlap. Steering the average activations of just 1% of each group reproduces translation quality comparable to an explicit instruction (e.g., 33.3","pith_inferences":["A testable prediction the paper does not make: if the same decomposition holds in other cross-lingual tasks (paraphrase, summarization, code-switching), 'language' and 'content' may be generally separable at the head level.","The paper's own evidence shows first-token selection identifies similar heads to their KL-based method for larger models; a cheaper identification protocol might be possible, but would need validation on smaller models.","The contrastive corruptions likely change more than the intended subtask (e.g., lexical statistics, syntactic templates), so the cleanest test of the decomposition would use synthetic corruptions that vary only the target language at constant token distribution; the paper's causal steering partially addresses but does not fully rule out this confound.","Because language vectors decode to language-associated tokens while equivalence vectors decode to overlapping high-vocabulary tokens, one could attempt to 'retarget' a model into a language it generates poorly by swapping language vectors alone — though the paper's low-resource results suggest this may hit a generation-quality ceiling."],"forward_implications":["Translation is mechanistically localized: interventions on ~1% of attention heads are sufficient to elicit or disrupt sentence-level MT, so model editing and steering are viable at this scale.","Language identity and meaning equivalence are separately encoded, with mostly disjoint head sets (Jaccard index ≈0.13), refining the picture of how multilingual LLMs organize translation.","Instruction-free steering with language and equivalence vectors matches instructed zero-shot prompting (e.g., 33.3 vs 32.6 BLEU for Gemma-3-12B English→X), so prompts may be replaceable by targeted activation control.","Ablating language heads collapses target-language accuracy when translating out of English, while ablating translation heads degrades meaning; the asymmetry reflects English-centric pretraining.","Equivalence vectors transfer across translation directions with minimal quality loss, suggesting the meaning-preservation component is largely direction-agnostic."],"fun_headline_variants":["Steering 1% of heads matches instruction prompting","Language and meaning live in disjoint head sets","Sparse heads enable instruction-free translation","One percent of attention heads steers translation","LLM translation decoupled: meaning vs language heads"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the two corrupted prompt families isolate exactly the two intended subtasks — replacing demonstration targets with another language isolates 'which language to output', and replacing them with random same-language sentences isolates 'preserve meaning'; if those corruptions also shift lexical, syntactic, or difficulty properties, the identified head sets may be conflated.","fun_headline_variants_meta":{"raw":{"variants":["Steering 1% of heads matches instruction prompting","Language and meaning live in disjoint head sets","Sparse heads enable instruction-free translation","One percent of attention heads steers translation","LLM translation decoupled: meaning vs language heads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1109,"prompt_tokens":668,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":412,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":412,"tokens_out":441,"duration_ms":5298,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:30:54.128973+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check: construct a corruption that swaps every content word of the demonstration targets for same-language synonyms (preserving syntax, length, and target language) and see whether activation patching still isolates the same 'translation heads'; if the head set changes drastically, the claimed equivalence subtask is an artifact of the particular corruption. Alternatively, ablate the top 'language heads' on a direction into English, where the paper reports language heads matter less — if target-language accuracy stays at random-ablation levels but BLEU still collapses, the functional","supporting_citations":[],"review_version":1}