{"id":"f4d9c09f-b392-4b11-89f1-b2c1f924728e","arxiv_id":"2601.08146","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A mechanism-guided fine-tuning method that updates only task-relevant attention heads (found via adapted CD-T) can reduce catastrophic forgetting in low-resource cross-lingual transfer, though the headline results rely on choosing the best of two update schemes on test data.","lead":"This paper proposes CT-SFT, a fine-tuning method that updates only a small set of attention heads identified as task-relevant in a source language, then transfers to low-resource languages. It reports improved target accuracy and much less forgetting than full fine-tuning, but the headline comparisons select the better of two variants using test data and omit error bars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported CT-SFT gains are not attributable to mechanism targeting because the ablations that would establish this (random-sparse and least-relevant head baselines) are omitted in the cross-lingual setting, and an oracle scope choice is used for the headline comparisons.","rationale":"The reader's weakest_assumption focuses on the circuit's validity/transferability (mean-baseline neutrality, head relevance transferring across languages). I partially agree: the shared-pool protocol in Appendix D.3 is indeed not validated for cross-lingual transfer and the 50-example mean baseline could be noisy; those are plausible issues. But I identify a more direct, observable flaw: even if the discovered heads are a faithful mechanism, the paper's headline cross-lingual results do not rule out that the gains come from sparse updating alone, because the random-head and least-relevant baselines are absent from the cross-lingual tables despite being present in the within-language sanity check. Moreover, the headline retention and accuracy numbers depend on 'taking the better of mechanism-preserving/editing scopes' (Table 3 caption), which is a test-data-driven oracle choice unless a protocol specifies validation-based scope selection. This is a load-bearing concern because if a random sparse baseline matches CT-SFT, the causal/mechanistic interpretation collapses; if the scope is oracle-selected, the method as specified could underperform full fine-tuning. The paper includes helpful honest limitations (single family, weak-competence instability), but the specified evaluation protocol is not fixed. I therefore recommend CONDITIONAL rather than full REJECT: the concern is concrete and testable, and the paper's own within-language data suggest sparse baselines are strong, but the issue is empirical and fixable by rerunning the missing controls rather than by a fundamental logical impossibility.","tokens_in":17118,"tokens_out":1971,"duration_ms":17011,"concrete_test":"Rerun the NusaX cross-lingual transfer protocol (Table 2) and catastrophic-forgetting protocol (Table 3) with the specification fixed to: (i) select Circuit versus NearZero scope solely on a held-out validation split before test evaluation, or at minimum report both variants un-bolded without claiming the better one; and (ii) include random-head and least-relevant-head sparse baselines matched for parameter budget (6/12/18 heads + LayerNorm) in the cross-lingual setting, as already done in the within-language sanity check. If random-sparse or least-relevant baselines match or exceed the circuit-selected results, or if the validation-selected scope fails to reproduce the gains, then the attributed 'mechanism targeting' effect is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that updating a task-discovered circuit of attention heads outperforms continued full fine-tuning and uniquely reduces catastrophic forgetting (Abstract; §6.2, Tables 2–3). The load-bearing causal premise is that selection by circuit relevance, not sparsity or parameter-count alone, drives the reported benefits. The paper's own within-language sanity check (§6.1, Fig. 2) shows random sparse head updates often outperform full fine-tuning, and the main text explicitly omits random-head baselines from the cross-lingual plots 'to reduce visual clutter' (§6.1). Thus the cross-lingual comparisons in Table 2 lack the one control that would separate mechanism-targeted updates from arbitrary sparse updates. This omission is compounded by the evaluation protocol: Table 2 reports both 'Circuit' and 'NearZero' variants and marks the better of the two as the result that is bolded and used in the headline statements (e.g., ace: 0.547 via Circuit, jav: 0.636 via NearZero), but the protocol does not specify how, when, or on what data the better scope is chosen. If the scope is selected on test data, the headline numbers describe an oracle-augmented method, not the specified CT-SFT. Note Table 3 also states 'CT-SFT uses the Circuit head set at discovery depth d=2, taking the better of mechanism-preserving/editing scopes,' confirming the oracle dependence at the point of the non-forgetting claim. The paper explicitly acknowledges in §6.3 (and Appendix C.2, Table 6) that the 50-sample XNLI competence setting yields unstable or degrading transfer, and the Limitations section admits single-model-family scope, but the central cross-lingual retention and accuracy claims are not supported by a fixed, reproducible protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CT-SFT, a two-stage method: (1) discover a task-relevant circuit of attention heads in a higher-resource 'proxy' language by adapting Contextual Decomposition for Transformers (CD-T) to unstructured text via a label-balanced activation mean (Eq. 3) and a directional relevance score (Eqs. 4-6); (2) adapt to a low-resource target by fine-tuning only the selected heads plus LayerNorm. The authors evaluate on NusaX sentiment and XNLI, claiming that CT-SFT improves cross-lingual accuracy over continued full fine-tuning while preserving source-language performance, and identify an editing-preserving trade-off depending on transfer difficulty.","tokens_in":17514,"tokens_out":5085,"duration_ms":48258,"significance":"If the claims were fully supported, the paper would make a useful practical contribution: a counterfactual-free circuit-discovery procedure that enables mechanism-targeted adaptation for low-resource languages, with a strong forgetting-reduction result. The paper is also commendably transparent, reporting detailed diagnostics (faithfulness, topology, selection-ratio sweep, shared-pool stability) and an explicit limitations section. However, the headline empirical claims currently rest on an oracle scope-selection protocol and on cross-lingual comparisons that lack the controls needed to attribute gains to circuit targeting.","major_comments":[{"comment":"CT-SFT is defined as 'taking the better of the mechanism-editing and mechanism-preserving update scopes' (Table 4; also Table 3). The protocol does not specify when or on what data this better scope is chosen. If it is chosen on the test set, the reported numbers describe an oracle-augmented method rather than a fixed CT-SFT procedure, and the Abstract's claim that 'CT-SFT improves ... while updating only a small subset' is not supported. Please define a deterministic scope-selection rule (e.g., using the competence-tuning baseline A0 or validation faithfulness) and report results for the resulting fixed method, or report Circuit and NearZero separately without selecting the better.","section":"§6.2, Table 2; Appendix C.1 Table 4; Table 3"},{"comment":"Section 6.1 reports that random sparse head updates frequently outperform full fine-tuning in the within-language sanity check (Figure 2), and the text states the random-head baseline is omitted from cross-lingual plots 'to reduce visual clutter.' Without a random-head (and least-relevant-head) control in the cross-lingual setting, the gains in Table 2 cannot be attributed to the task-relevance of the selected heads rather than to sparsity/regularization. Please add these controls to the cross-lingual tables/figures, or at minimum report them in an appendix.","section":"§6.1, Figure 2; §6.2, Table 2"},{"comment":"Competence tuning, baseline mean µ, circuit-discovery inputs, and faithfulness diagnostics are all drawn from the same 50-example pool. Appendix D.3 shows only that iteration-0 head ranking is more stable under the shared pool; it does not establish that the resulting circuit is a faithful, transferable mechanism for target languages. If the balanced mean is not neutral or the selected heads reflect pool-specific noise, the cross-lingual transfer results in Tables 2 and 4 would not be mechanistically grounded. Please validate the discovery with held-out pools or report cross-lingual faithfulness.","section":"§4.1.1 (Eq. 3); Appendix D.3"},{"comment":"All results are averaged over 4 seeds, but no standard deviations, confidence intervals, or significance tests are reported. Given the small tuning sizes (n=25) and small performance differences (e.g., Table 2 bug n=25: 0.374 vs 0.376), the claim that CT-SFT 'consistently improves' over full fine-tuning is not statistically supported. Please include variance estimates or significance tests (e.g., bootstrap or paired tests across seeds).","section":"§5.4; Tables 2-4"}],"minor_comments":[{"comment":"The notation Y and Y_{-other} is used but not explicitly defined at first use; please clarify that Y is the label set and Y_other is all labels except the correct one.","section":"Eq. (4)"},{"comment":"The training recipe (epochs, learning rate, batch size) is deferred to Appendix A; please state it in the main text or add a forward reference.","section":"§5.1"},{"comment":"The underline/bold convention is visually hard to distinguish in the table; consider using symbols (e.g., daggers) or a separate column for the selected scope.","section":"Table 2 caption"},{"comment":"Several references use incomplete author lists (e.g., 'Alan Ansell and 1 others'); use standard et al. or full author lists consistently.","section":"References"},{"comment":"The XNLI summary is very brief; a key table (e.g., the 250-sample English competence setting) should be moved from the appendix to the main text, as the current text relies on a separate figure.","section":"§6.3"}],"recommendation":"major_revision","confidential_remarks":"I recommend requiring the authors to re-run the evaluation with a pre-specified scope-selection rule and to add random-head baselines to the cross-lingual experiments. If those results show no advantage over random sparse updates, the central claim should be substantially revised. The paper is well-written and the diagnostics are thorough, but the current headline results are not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version first. The paper proposes CT-SFT: discover a task-relevant attention-head circuit on a source-language checkpoint using a modified CD-T (label-balanced mean baseline, directional relevance score), then fine-tune only those heads plus LayerNorm on a low-resource target. That is a genuinely useful combination: CD-T normally needs clean counterfactuals, and the authors make it work on unstructured sentiment/NLI text. The within-language sanity check is also honest—it shows continued full fine-tuning is unstable on 25–50 examples and that random sparse updates regularize well.\n\nBut the main cross-lingual results are not yet a fair test of the method as specified. The biggest problem is the 'takes the better of Circuit and NearZero' protocol (Tables 2, 3, and Appendix Table 4). No rule is given for when to pick mechanism-editing vs mechanism-preserving scope; if that choice is made after seeing the test results, the reported numbers describe an oracle-augmented variant. That is load-bearing because the headline claim is that CT-SFT beats full fine-tuning.\n\nSecond, the cross-lingual comparisons omit random-head and least-relevant baselines, which the paper itself reports within language. Since random sparse updates often beat full fine-tuning, the current tables cannot separate 'targeted circuit' from 'any small sparse update.' The authors say they left the baselines out to reduce clutter; that is not a scientific reason.\n\nThird, there are no error bars or significance tests. Four seeds in this few-shot regime can easily hide high variance, and several numbers are close (e.g., 0.636 vs 0.623). This is fixable but necessary.\n\nThe forgetting result is the most interesting piece: CT-SFT keeps Indonesian accuracy near the competence baseline while full fine-tuning drops to 0.34–0.45. That pattern is visible and worth taking seriously, though it also inherits the oracle-scope issue.\n\nThe limitations section is candid about single model family and the 50-example mean sensitivity, and the shared-pool D.3 analysis is a reasonable acknowledgment of a real constraint. The abstract's claim about 'model families' is not supported by the experiments, but the limitation text says so.\n\nWho is this for? People working on low-resource adaptation or mechanism-guided fine-tuning. It is a solid empirical exploration with a clear flaw in evaluation protocol, not a paper to ignore. I would send it for peer review with a clear request: specify the scope-selection rule, add random/least-relevant baselines to the cross-lingual tables, and report variance. Then the central claim might hold or collapse, but at least it would be testable.\n\nMy bottom line: engage seriously, but ask for the fixed-protocol revision before believing the headline numbers.","headline":"Useful idea—counterfactual-free circuit discovery for low-resource tuning—but the headline numbers rely on oracle scope selection and missing sparse baselines, so treat them as upper bounds until the protocol is fixed.","tokens_in":18051,"tokens_out":3086,"would_cite":true,"duration_ms":31338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Updating only task-relevant attention heads—roughly 0.2–0.7% of parameters—can adapt a model to a low-resource language while preserving its source-language competence.","keywords":["low-resource adaptation","mechanistic interpretability","circuit discovery","attention heads","catastrophic forgetting","cross-lingual transfer","contextual decomposition","parameter-efficient fine-tuning"],"falsifier":"Run mean-ablation or activation-patching on target-language inputs: if removing or ablating the selected circuit heads does not drop target accuracy more than removing the same number of random heads, the circuit is not causally relevant in the target. Alternatively, match random-head selection to the circuit's layer distribution and parameter count and compare target accuracy and source-language retention; if random heads reproduce the gains, sparsity rather than mechanism targeting explains the results.","tokens_in":17027,"feed_emoji":"🧠","tokens_out":4496,"duration_ms":45700,"temperature":0.7,"pith_summary":"This paper tries to establish that a model can be adapted to a low-resource language by updating only the attention heads that carry the task in a higher-resource proxy language, rather than fine-tuning everything. It adapts circuit-discovery machinery to unstructured text, where clean counterfactuals do not exist, by defining a neutral baseline as a label-balanced mean and scoring heads by how much they move the model toward the correct label. Applied to sentiment transfer from Indonesian to four related languages and to cross-lingual natural-language inference, the method matches or beats continued full fine-tuning in target accuracy while updating under one percent of parameters. Its distinctive claim is that this targeted update prevents catastrophic forgetting: source-language accuracy stays near its pre-transfer level, whereas full fine-tuning degrades it sharply. The paper also reports an editing–preserving trade-off: harder transfers benefit from updating the discovered circuit, while easier transfers benefit from updating near-zero heads.","feed_headline":"Updating 6 attention heads beats full fine-tuning for low-resource transfer","feed_subtitle":"Mechanism-guided tuning matches or beats full-model adaptation while keeping the source language's competence intact.","key_machinery":"The central object is the circuit—a sparse set of attention heads identified by a counterfactual-free adaptation of Contextual Decomposition for Transformers (CD-T). CD-T separates each component activation into an irrelevant baseline (γ) and a relevant deviation (β); the paper replaces templated counterfactual baselines with a label-balanced activation mean, replaces the magnitude ratio with a directional relevance score projecting β onto the task direction (the difference between the correct-label and average-incorrect-label unembedding directions), and selects the top K heads by mean directional relevance at successive depths. Circuit-targeted supervised fine-tuning then applies head-leve","core_discovery":"The central claim is that circuits discovered on a source-language checkpoint transfer across languages and can serve as the training locus for low-resource adaptation. Using a 50-example label-balanced mean as the CD-T baseline and a directional relevance score that measures support for the correct label, the authors select a small set of attention heads (6, 12, or 18 in their settings). They then freeze everything except those heads and LayerNorm. Across two tasks, this CT-SFT procedure attains cross-lingual accuracy competitive with or better than continued full fine-tuning, while preserving source-language competence: Indonesian accuracy stays near 0.76 after target tuning, versus drops","pith_inferences":["A testable extension is an automated router that measures source-checkpoint accuracy on the target (the paper's A0 baseline) and chooses Circuit versus NearZero heads accordingly; this could make the method deployment-ready.","If the transferred-circuit premise is right, closely related language pairs should need almost no parameter movement and distant pairs the most; sampling a wider typological spread would stress-test the claim.","The paper implicitly defines transfer difficulty as the competence-tuned checkpoint's accuracy on the target; this definition could be validated against independent difficulty metrics such as lexical overlap or syntactic distance.","As the authors themselves caution, the discovered heads are empirical candidates for one model family and task setup, not universal mechanisms; practitioners should run minimal target-language sanity checks before deployment."],"forward_implications":["Updating only the discovered circuit heads plus LayerNorm should be preferred over continued full fine-tuning for low-resource transfer when source competence exists: it matches or exceeds target accuracy while using 0.2–0.7% of model parameters.","Source-language competence survives adaptation: models tuned with CT-SFT retain near-baseline accuracy in the source language, whereas full fine-tuning can lose 20–40 percentage points.","The update locus should be chosen by transfer difficulty: harder transfers benefit from updating the circuit heads, easier transfers from updating near-zero-relevance heads to preserve the source mechanism.","A checkpoint must first be task-competent; CT-SFT is a post-competence adaptation method, not a substitute for learning the task—a limitation the authors flag explicitly.","The same editing–preserving pattern appears on a second task (natural language inference) once competence is achieved, suggesting the mechanism-transfer view is not sentiment-specific."],"fun_headline_variants":["Tune 6 attention heads, keep source language intact","Circuit-targeted SFT: match full fine-tuning without forgetting","Six heads match full fine-tuning for low-resource transfer","Fewer updates, no forgetting: circuit-targeted tuning","Low-resource adaptation: small circuit, safe tuning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise—flagged by the authors in the Limitations section—is that the circuit discovered on the source checkpoint, using a 50-example label-balanced mean baseline drawn from the same pool as competence tuning, is faithful and transfers to the target language; if that baseline is not neutral or the heads do not carry a transferable mechanism, the accuracy and retention gains cannot be attributed to mechanism targeting.","fun_headline_variants_meta":{"raw":{"variants":["Tune 6 attention heads, keep source language intact","Circuit-targeted SFT: match full fine-tuning without forgetting","Six heads match full fine-tuning for low-resource transfer","Fewer updates, no forgetting: circuit-targeted tuning","Low-resource adaptation: small circuit, safe tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1717,"prompt_tokens":677,"completion_tokens":1040,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":421,"completion_tokens_details":{"reasoning_tokens":959}},"tokens_in":421,"tokens_out":1040,"duration_ms":9753,"temperature":1.0,"reasoning_tokens":959,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:51:54.743456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run mean-ablation or activation-patching on target-language inputs: if removing or ablating the selected circuit heads does not drop target accuracy more than removing the same number of random heads, the circuit is not causally relevant in the target. Alternatively, match random-head selection to the circuit's layer distribution and parameter count and compare target accuracy and source-language retention; if random heads reproduce the gains, sparsity rather than mechanism targeting explains the results.","supporting_citations":[],"review_version":1}